Caching in 10.1 produce inconsistent results

[EyadHammad]

hello,

i have published a number of MXD files as map services. when attempting to cache, we have different results at different trials. number of completed tiles keep changing with every new attempt. i am not confident of any result i get. sometimes i get zero tiles consisting 0% and other times i get 400%. and the shock is that sometimes i get cache failure message.

i would assume caching procedure is well defined and outcome is constant given same set of input variables. However, i do keep providing the same parameters yet i get different results.

i also noticed that when i do multiple scales at one time, usually something wrong goes on

when i perform caching at one scale at a time, tool is more stable

but, still results vary

have anyone else experienced this issue  ?

are there any detailed literature explaining what exactly is happening when a cache order is placed ?

is there any verification of QC method to double check our work ?

i noticed there is a fix errors command in the cache status wizard. does anyone know what this operation fix exactly ? and why once we run it and it completes and the job status is updated, why does it not update the number of tiles completed ?

[EyadHammad]

hey guys,

i feel your frustration..


from my caching exercise, i found that the main two major factors  having heavy impact on caching process are Hardware & caching instance configuration. do not focus only on map authoring or the data side when trying to figure out what went wrong.

My caching experience was a scary nightmare on my old environment (4 Core, 8GB) before i tried out moving ArcGIS Server to an enhanced environment ( 16 Core, 49 GB) and before i figured out that i can increase my productivity by increasing the number of instances assigned to the System/Caching geoprocessing tool.

i found out that if your CPU usage is dancing around 100% Or if your Memory usage is dancing around 100%

most probably your cache will be failing at least in several extents.

similar thing is when you lose your network connectivity even from short period of time

Now, the experience is still away from satisfaction. However, i have moved from Nightmare  to Inconvenience state.

eagerly waiting for SP1 and praying that it would resolve at least half of my concerns

[JulieKusic]

I'm also seeing incredibly inconsistent results.

I'm building 2 caches at a national scale and I'm running into the following issues.

1) The cache status window will tell me that tile generation is in progress, but no new tiles are being generated after a refresh.  The completed tiles stay the same, the percent stays the same and the in progress column is indicating it's still building.  I've waited overnight and let my site churn this out to come back in the morning and nothing is new.  The job status says the job is done, with a 100% completion and no errors but those tiles still aren't completed on the cache status.

2) the cache process seems to skip/give up on certain scales. 
I'll have scales that are partially completed and the cache process will simply move onto the next scale and never return to complete the missing tiles in previous scales.  So the cache status is telling me that they're only partially complete, but again the job is saying the cache is 100% complete and with no errors.  The services themselves have obvious missing tiles as you zoom/pan around with no data.

Does anyone have any advice or has anyone been able to successfully build a national scale cache with significant data?  The cache I'm generating is a nationwide Navteq 2011 Q3 street dataset with basemaps and I have clients waiting for this service.

My site consists of 4 64 bit Windows Server 2008 Machines with 4 cores/8 CPU's @ 2.4GHz (32 total CPU's in the site) and each machine has 18GB of RAM.  I don't think my hardware should be the issue.

I am having the same exact issues.  My cache started off with a bang and then it slowed to a crawl and completed some levels and then only did some of the tiles at other levels.  No error reports.  Dialog said cache was completed at 100%.  It obviously wasn't because half of the tiles are missing.  I am going to be doing a nationwide cache at 17 levels shortly and I'm worried that it will give very inconsistent results.  Any help would be appreciated!

[Anonymous User]

I'm also seeing incredibly inconsistent results.

I'm building 2 caches at a national scale and I'm running into the following issues.

1) The cache status window will tell me that tile generation is in progress, but no new tiles are being generated after a refresh.  The completed tiles stay the same, the percent stays the same and the in progress column is indicating it's still building.  I've waited overnight and let my site churn this out to come back in the morning and nothing is new.  The job status says the job is done, with a 100% completion and no errors but those tiles still aren't completed on the cache status.

2) the cache process seems to skip/give up on certain scales. 
I'll have scales that are partially completed and the cache process will simply move onto the next scale and never return to complete the missing tiles in previous scales.  So the cache status is telling me that they're only partially complete, but again the job is saying the cache is 100% complete and with no errors.  The services themselves have obvious missing tiles as you zoom/pan around with no data.

Does anyone have any advice or has anyone been able to successfully build a national scale cache with significant data?  The cache I'm generating is a nationwide Navteq 2011 Q3 street dataset with basemaps and I have clients waiting for this service.

My site consists of 4 64 bit Windows Server 2008 Machines with 4 cores/8 CPU's @ 2.4GHz (32 total CPU's in the site) and each machine has 18GB of RAM.  I don't think my hardware should be the issue.

It definitely could be the hardware, if you're maxing it out.  And you can't visually tell if you're maxing it out; the cpu usage could be under 50% and still be maxed out, for ArcGIS Server's purposes.  Try cutting the number of instances you're using.  The problem, i've been told, is the amount of available memory per server core, and the only way to sort of control this is with the number of instances.  Drop it down to 2, it will take longer, but it could help it actually finish, and actually create all the tiles without skipping any.  You should also limit the mxd you're using to a few scales of data, and only one scale per mxd at larger scales like 1:18k or 1:36k.