Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[DavidColey]

Hi Michael-

I don't know what to tell you other than to delete all aspects of the 10.2.0 service and completly recreate the map service, with a new tiling scheme at 10.2.2 and then re-cache it before it is under load.  I had difficulty and odd behavior when I tried re-caching the same maps from previous releases at .2.  Sure looking forward to the 10.3 fix on this. 

[MelissaNorthey]

I'm so glad I found this thread!  I have been going CRAZY.  Go figure, it's locking itself up - ha!  Sounds about right.  It also appears as it is a problem that bleeds through to ArcGIS Online Tile Layers as well.  I just put in a support call about that one today.

[SebastienPetit]

Good to find this

Same problem here :S

[BobNutsch1]

This is a great thread, especially after wasting hours if not days on trying to figure out why I can no longer maintain caches as I have before for years using various workflows. When I first came across this problem I ended up corrupting part of an important cache.  I've spent a lot of time with Esri support on it - this is becoming a critical problem for me due to the size of the caches I have to create and maintain. 

We too are running a clustered environment, 10.2.2, compact cache, UNC path. Our sys admin has reported literally 1000s of file locks on the tiles bundles that only disappear once ArcServer is restarted. I can verify that the file locks disappear after restarting the ArcGIS Server service on the servers.

I plan to ask that a hotfix be generated.

[RonnieRichards]

This is amazing. We reported this issue in June only to get push back from support and months of back-forth about our environment, cluster and file shares and look at all the folks coming into this thread the last few weeks!

Just a fyi we installed 10.3 prerelease hoping this issue would be resolved and we ran into a major problem with the CachingTools service. It was not started/added during the in-place upgrade of our 10.2.2 instance. It basically is not running so no cache updates or tools are available. Waiting to get a call from support to see what we do. Not even sure if they test upgrade scenarios!

[DavidColey]

Thanks for all the replies everyone..  I've been really busy putiing out a jsapi app and haven't had even a second to check my threads, esp. this one.  That's really disappointing about this not being addresssed in the 10.3 prerelease.  I don't understand.  A NIM was generated for this issue over two months ago after I spent hours and hours with esri on this problem.  And it has always been my understanding that NIMs are addresssed in the next release.  ESRI, please issue some kind of statement about this issue.  Or better yet, a hotfix as Bob suggested. Please!

[BobNutsch1]

Yesterday I asked our regional office for a hotfix and to escalate the NIM.  I haven't heard back yet.  Once I hear something I'll post it here.  But meanwhile, maybe if everyone on this thread asks their office for a hotfix/escalation... 🙂

[RonnieRichards]

Well we finally got the 10.3 pr environment up and running after a failed in-place upgrade of 10.2.2. (be warned!). However I have bad news to report we are still getting cache bundling issues updating cache on specific scales in the 10.3 prerelease environment on a server with very minor load. As echoed by David this is frustrating and I will be contacting my Regional office today.

[MichaelVolz]

Ronnie:

Did you install AGS v10.3 on a server(s) that previously had AGS v10.2.2?

Or did you need to install AGS v10.3 on a new server(s), because the in-place upgrade failed?

[BobNutsch1]

Until this bug is fixed, I believe this will be my workaround workflow which so far seems to work:

1. Build/update the staging cache (all or parts) on staging
2. Shut off the staging cache service and stop the staging ArcGIS Server windows service to clear up any locks on the staging cache (if possible, I will leave the staging ArcGIS Server windows service in a stopped state then start it up later)
3. Late the night before production ArcGIS Server windows service will get restarted as normal, shut off the production cache service that is to be updated.  This is done so that once ArcGIS Server gets bounced, new files locks cannot be created.
4. The next morning the production cache service should still be off and thus the file locks are still off so now if a manual file/directory copy is to be done, rename the old production cache directories and from the staging area, copy in the new files/directories, or use Export Cache to update the old production cache files with the new cache files on staging (noting that if Export Cache is used, then the staging ArcGIS Server windows service needs to be started first).
5. Restart the production cache service and test it out
6. Restart the staging ArcGIS Server windows service if needed

Using this method I'm not able to update a production cache service on the fly from the staging service like I've done in the past which can be a very efficient approach, but at least I have a method now of updating a production cache, though in a clunky fashion.  If this bug is not fixed in a timely manner, I may end up scripting the above steps.