Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[RobertBuck1]

I'd humbly like to offer my 2-cents, if I may. I found that when I have a failed tile cache update job, similar (if not identical) to the problem listed on this thread I do the following:

1. Turn off the ArcGIS Server service (e.g. 'net stop "ArcGIS Server"') and then wait 2 minutes
2. Delete all '*tmp.bundle', '*.bundle.lock', and '*.freelist' files, in the arcgiscache folders for each affected map service - it seems that ArcGIS Server releases any file system holds or locks on these files after the "ArcGIS Server" service is shut down.
   1. I chose to do this manually, during normal business hours
   2. Also, even after the "ArcGIS Server" service is restarted, these files can be deleted (if no one is trying to consume the service)
3. Start the ArcGIS Server service (e.g. 'net start "ArcGIS Server"')
   1. I think at this point, the map tiles could be successfully updated (as long as no client is consuming the service). I chose to wait an extra day before updating the tiles in order to be careful and make sure that nothing at that point had broken.
4. The next 'time of no demand' (e.g. 12:30 AM), turn off the "ArcGIS Server" service and wait for 2 minutes
5. Restart the "ArcGIS Server" service and wait for 5 minutes
6. Regenerate the map tiles (e.g. using the arcpy.ManageMapServerCacheTiles_server gp tool)

My plan for preventing this issue, until I can install the 'Map Cache Consumption' patch, is (via python script ran by a scheduled task late at night):

1. Turn off the ArcGIS Server service and pause for 2 minutes
2. Restart the ArcGIS Server service and pause for 5 minutes
3. Regenerate the Map tiles (e.g. using the arcpy.ManageMapServerCacheTiles_server gp tool)

I think that, once the temp/lock files are removed and after 'bouncing' the ArcGIS Server service, the update process should run successfully - if no one is consuming the tiles of the service in question. I could be wrong - it's only been 3 or 4 successful updates using this method - but so far it looks promising. NOTE: I am running ArcGIS for Server v10.2.2, Windows 2008 R2 OS, single server deployment.

- Robb

[MichaelVolz]

Question for cache experts:

Do you think that the caching performance has increased between 10.0 and 10.2?  I ask because the same cache update in 10.2.2 is taking twice as long as the existing process that is still occurring with a cached v10.0 mapservice.

[JeffPace]

I am still having this problem at 10.3.1 using UNC paths

[DavidColey]

Really?  I don't know what to say.  My caching has been very stable at 10.3 since applying the ArcGIS 10.3 for Server Map Consumption patch and subsequently at 10.3.1.  I have not had to change to drive-letter paths, and even my maplex-engine basemap service is caching well. 

[RonnieRichards]

Jeff,

We also still have this issue in 10.3.1 but we are still using the 10.2.2 compact cache format and we hoping it was fixed but does not seem to be the case. It appears instead of fixing the problem with the compact cache, a new cache format was created in 10.3. Are you using the new 10.3.1 cache format?

[MichaelVolz]

Is there a patch for this issue at 10.3.1?  Or is the fix supposed to be built into the software?

[AnibalMmartinez]

Hola, les comento que nos sucede lo mismo que a ustedes sobre ArcGis 10.1 Sp1, Windows Server 2012 R2, direct conect contra 11G

Tenemos 3 mensajes detectados:

-ERROR: @ Error HRESULT E_FAIL has been returned from a call to a COM component.

-INFO: Failed to cache extent: -58.128823 -29.225041 -58.040981 -29.147969 at scale 250000 The index was either too large or too small.

-INFO: Field is not editable.

incluso, a veces nos responde:

INFO: Succeeded at Mon Sep 28 14:43:20 2015 (Elapsed Time: 1 minutes 3 seconds)

y  en realidad no genero los datos.

¿Alguno encontro una forma de trabajo para evitarlo.?

Saludos,

Anibal Martinez

Telecom Argentina

[DavidColey]

Hi Anibal - is your server environment still at 10.1?

[AnibalMmartinez]

Hola David, terminamos de migrar desde 9.3.1 hace 30 días. El proyecto nos demando mas de 8 meses, dado la gran cantidad de desarrollo que tenemos sobre lo openthebox. Esto recién comienza.

En este breve tiempo que seguimos investigando, hemos logrado reproducir el error en el ambiente de Desarrollo a demanda. Si se crea un cache y en simultaneo consumís la escala que se esta cacheando (ej paning), ocurre este isue.

Evidentemente es un problema de concurrencia al directorio cache con los archivos (*.bundlx , *.freelist, *.bundle y .bundle.lock).

Saludos,

Anibal Martinez

Telecom Argentina

[DavidColey]

Correct.

Caching services under load at 10.1 persist locks on the bundle exchange files, preventing bundles from moving and causing caches to fail. The only way I could maintain caches at 10.1 and 10.2 was to create a duplicate service with a new cache for each service, and then move it's bundles into the existing cache folders via Windows xcopy or something.

If at all possible move to 10.3 or better, to 10.3.1 and these problems will disappear.

Cheers-

David