Manage Map Cache Tile: Errors at 10.2.2

DavidColey · ‎05-13-2014

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

MichaelVolz · ‎05-14-2014

Are you using an exploded or compact cache?

ILIMAKENNEDY · ‎05-14-2014

Compact cache.

(I didn't mean to hijack the thread...only to get updates to the original poster's query.)

DavidColey · ‎05-15-2014

Yes thanks Ilma I understand and as soon as I find out anything I will post it. Perhaps esri will chime in here at some point? 🙂

Thanks
David

DavidColey · ‎05-15-2014

Sorry, one more bit of info. This is only occuring on my multi-machine production site. I am not encountering this error at my single-machine dev site. Of course, that makes sense as a single-machine site doesn't have to move a bundle file, right?
David

JustinHansen1 · ‎05-19-2014

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

I'm seeing the exact same behavior with one of our new development 10.2.2 ArcGIS Server sites.

I'm unable to re-cache a service unless I follow the same steps as David: delete the service, stop the ArcGIS Server service, delete the cache directory, start the ArcGIS Server service, and re-create both the service and the cache. ArcGIS Server is holding an exclusive read lock on the .bundle files even after the service is stopped or deleted. The exclusive read locks are only released after the ArcGIS Server service is stopped and started again. I'm assuming this behavior is what's causing the re-caching tools to fail. I've added the ArcGIS Server Account as a local admin on the ArcGIS Server server to eliminate any permission issues.

My site is a single machine site with both a UNC share cache directory and a local directory. The behavior is the same between the shared directory and the local directory.

I'm still hunting for a workaround.

DavidColey · ‎05-20-2014

Hi Justin- Yes I am still experiencing the same error.

As far as a workaround goes, I could script the service deletion and run as a task, use task scheduler on each machine to perform a ArcServer.exe service stop, use an OS rmdir command in task scheduler to remove the directories, start the ArcServer.exe on each machine, then script and schedule service definition (sd) files for the map services, publish the sd's, and then finally re-run the caches. Sure seems like a lot of work for something that was so simple. Hey, if you think of another way let me know!
David

MichaelVolz · ‎05-21-2014

David:

In your multi-machine production site, do you have the cache files stored on each server in the cluster?

DavidColey · ‎05-21-2014

Hi Michael-
No, the cache tiles are stored in a shared directory on a network server. A temp directory has been configured through ServerAdmin on each server with Server 10.2.2 installed, which of course the manage tiles operation should handle in terms of moving the bundles from the temp directory to the cache directory. I am certain this is per ESRI help documentation:

http://resources.arcgis.com/en/help/main/10.2/#/Local_cache_directories_on_the_server/01540000046r00...

Thanks
David

ScottMcQuarrie · ‎05-28-2014

I am also experiencing this with the same error. Any thoughts ESRI?

DavidColey · ‎05-29-2014

Interesting, glad (well, not glad) to hear it's not just us. I have a support request in with ESRI at the moment but thus far no real replies from them. Whatever I find out I will post.

In further testing, I have also discovered this behavior is happening on my stand-alone, single machine edn dev box. What I mean is that there shouldn't be any error regarding the 'moving' of bundle or bundlex files from any physical directories to the cache directory,unless it's an error moving fromt the jobs directory to the cache directory (I guess).

However, I'm beginning to suspect that this particular error is just the latest manifestation of other problems with service overwrites and caching in general. For example, we have long had difficulites in overwriting services that have a cache in place. When we tell server (through the server editor at overwrite time), that the cache will be updated manually typically the overwrite will leave behind it's 'scratch' service, visible in the rest or catalog. Often at that time, the original service will leave behing a .glock file, not allowing for further service manipulation until the site is bounced via stopping the ArcServer.exe processes.

Thanks-
David