Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[MichaelVolz]

I do not have a solution only a sad comment about this issue.

When developing a web adf application 3 years ago, I tried to use the compact cache option for my cached services, but found many missing tiles, which of course would be confusing to the enduser and therefore unacceptable.  The only solution after a few months of testing and an ESRI support ticket was to use exploded caches, which are still in use today.  Hopefully, ESRI will come up with a solution at this time, but it sounds like an exploded cache might be your best option at this point in time based on my past experience.

Is this problem occurring in all version of ArcGIS Server v10 (10.0, 10.1, and 10.2)?  I ask because I might be looking to use weekly updated cached mapservices in my current production environment (v10.0) if dynamic mapservices fail to perform well.

[DavidColey]

Hi Michael, no.  The bundle move error (which from what I see is really is a *.bundlex file not being moved from an arcgistemp folder on a gp machine in a cluster environment to a shared cahe directory) bundleissue is brand new at 10.2.2 (to us anyway).  I've never encountered an bundlex move error prior to this release since the bundles have been in existence.....

[MichaelVolz]

David:

So you are saying that the cache update worked at v10.2.0, but when you upgraded to v10.2.2 the cache update stopped working properly?  Or did you upgrade from a lower version (e.g. v10.1) straight to v10.2.2 without ever using v10.2.0?

[DavidColey]

We've always gone in order:  10.1, 10.2, 10.2.1, 10.2.2.  Cache bundles worked at every release unitl .2  I'm working with esri now on a ProcMon for further discovery . . .

[DavidColey]

Good Afternoon all:  I have a status update on this issue.  On Friday, while working with Jon at ESRI Redlands, he was able to sucessfully re-create the behavior we have been experiencing regarding the bundle error move.  In fact, Jon reported back to me that he saw the same behavior at 10.2.1 which I found quite interesting as we never experienced a bundle error move at 10.2.1, only cache failures for a given extent which could always be reparied. 

However, the good news is that this 10.2.2 bundle move error has now been logged as a NIM:

[#NIM104348  In ArcGIS Server 10.2.2, you're unable to recache tiles while the existing tiles are being accessed. ]

Which of course means the development team will now be looking at this issue. 

In the meantime, we all can go back to caching identical non-production services either on an edn or on a production GP cluster machine, as long as that service is not being accessed, and then moving the new bundles to production using REMDIR and XCOPY bat's. 

I have since last Thursday successfully completed three full UPDATE_EXISTING_TILES operations on a non-load basemap service of considerable cartographic complexity from scales of 577000k down to 1128K, both with and without the use of an AOI polygon. 

Thanks-

David

[MattMcGuire]

Thank you, David Coley, for your persistence.

[DavidColey]

Thanks Matt- Oh sorry all I did mean RECREATE_ALL_TILES, not UPDATE_EXISTING_TILES  in my earlier post

[RonnieRichards]

We are experiencing the same issue and the cached map service and MXD have gone thru 10.0, 10.1, SP1, 10.2, 10.2.1 and now 10.2.2.. We can build the cache just fine only to having bundling issues when attempting to do updates. Had an open ticket with ESRI for several months and of course it works when building a new cache each time but how many more times can we do this?

We will attempt to use the EXPLODED format because that is the only thing that seems to be in common on this thread along with the 10.2.2 release. We too have had to resort to the stop service, delete, publish, recache using silly dos commands. These tools should be easy and just work. Having to go in behind the scenes to fix the black box AGS is getting rather old and too time consuming to be testing scenarios that ESRI should be testing themselves instead of blaming things on environment or data issues.

[DavidColey]

Ronnie-

ESRI has idendtified this as an in use or under load issue and has created a NIM.  It looks like it will be fixed at 10.3.  We have set up 3 identical cache services and secured them.  Because they are not under load, re-creating _all_tiles does work.  I simply then perform an xcopy of my bundles over the existing, under load services.  In this way I do not have to resort to exploded caches.

[MichaelVolz]

Dave:

I am having an issue with a static cached image service that I created in AGS v10.2.0, but am using in AGS v10.2.2.  When I use the image service at the REST endpoint using ArcGIS JavaScript or in a JavaScript application, there are large rectangular areas where the imagery does not show up.  Did you ever have this occur for a cached image service that was static and did not require constant changes to the cache?

I am now recreating the cache in AGS v10.2.2, instead of using the v10.2.0 cache, and I have noticed that it creates the bundles at the level with the most tiles (smallest tiles) which is the opposite of how caches have always been created in the past where it start with the level with the least tiles (largest tiles).  This you notice this type of cache creation behavior in the AGS v10.2.2 environment.