Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[WillAllender]

We are having similar issues (cannot delete a handful of .bundle files) and yes, this is new at 10.2.2.  It is very hit or miss.  Out of 50,000+ files over 16 levels of cache and 6 different services, it is usually less than 200 .bundle files that cannot be deleted.

In the past we have built cache on a staging server, then xcopy into production.  Now, I am building cache on a staging server, using an os move command on the production file out to a garbage can area on the network share, then doing an os move from staging into production.  I have a sys admin working on how to delete the files that were moved out of production.  But for the moment, the directory is a garbage dump of files that cannot be deleted.

Our architecture at a high level:
1) 9 caching servers building cache from local file GDBs on each server, sending to a single cache repository on a network share.
2) build in staging
3) move to production

[DavidColey]

Hi Will-

If you are able to during off hours, you can stop your ArcServer.exe process directly from the Services folder on each of your site machines.  This will remove the locking on the affected bundle files and you will be able to delete the remaining bundle files.  Stopping the site machines through Server Manager (whether catalog or web) won't remove the lock.  We have a 3-machine site in two clusters and I am able to remove all the bundle and bundlex files once I stop each Server.exe process.  I can then xcopy over updated cache bundles from my edn machine.  However, as I'm sure you already know we really need esri to look into this.

Thanks
David

[DavidColey]

Hi All-
Hate to keep bumping the thread, but I did a screen share session today (06/05) with esri and wouldn't you know it the 'Line 30Output failure, error string = Error moving bundle' did not occur on our stand-alone development box (Server 2008R2) during a ReCreate All Tiles operation.  The only thing that has changed since the error first occured on or about 05/09 was that we applied MS Updates for May.  I do not know at this time if that has had any affect, since I did not do anything differently.  Here is a list of updates that were applied:

KB2931368 - Security Update for Microsoft .NetFramework 4.5.1
KB2852386 - Update for Microsoft Windows
KB2953522 - Security Update for Microsoft Windows
KB2871997 - Security Update for Microsoft Windows
KB2926765 - Security Update for Microsoft Windows
KB22931356 - Security Update for Microsoft Windows

Further testing will be performed on our distributed production site and I will continue to post results here-
Thanks
David

[DavidColey]

Hello All-
More information:  Yesterday's recreate all tiles was completed via the arcCatalog tool gui for Manange Map Tiles.  When run today in python (which is what we really need to do) the bundle error re-appeared.  Back to the drawing board.
Thanks
David

[MattiasEkström]

Hi!
David (or anyone else) have you found a solution to this problem? or heard something from ESRI?
Recently we upgraded from 10.1.1 to 10.2.2 and since then we're having similar problems.
Creating a new cache works fine, but recreating or removing files doesn't. Sometimes there are errors but sometimes there are not, but still the tiles becomes corrupt in some areas (i'm guessing a certain bundle file). And those bundle files can't be removed from the file system even after deleting the service.
The error messages we're getting are often something like:
Error executing tool.: Failed to cache extent: 145186,125833 6349330,818251 147554,165721 6351256,184865 at scale 25000 The index was either too large or too small.
We have single server deployment and store the caches locally.
/Mattias

[JohnFell]

Mattias,

We are experiencing the same problem that you did when you posted this. We are at 10.2.2 and get this error sometimes and sometimes we do not. Did you get this fixed? If so, what workflow helped to resolve your problem?

[MichaelVolz]

You can install a patch to address this issue

• ArcGIS 10.2.2 for Server Map Cache Consumption Patch

found at

Patches and Service Pack - Esri Support

You'll need to log in to download the patch.

There also is a bug with ArcGIS Server v10.2.2 where bundle errors do not get deleted from a temporary directory on the ArcGIS Server server.  You might want to periodically check this location so it does not fill up storage on the server which could eventually impact the performance of AGS.  These issues have been fixed at v10.3.x if you can upgrade your software.

[MattiasEkström]

John,

I just want to confirm that the patch Michael referring to fixed the issue for us.

[JohnFell]

Michael and Mattias,

Thank you for your feedback! I have installed the ArcGIS 10.2.2 for Server Map Cache Consumption Patch​ on both ArcGIS Servers that participate in the site some time ago. Yet we are still experiencing caching errors.

The most recent error was received after trying to update the cache using a feature class as the AOI when applying the DELETE_TILES option to the ManageMapServerCacheTiles tool. See below:

ERROR: Failed to cache extent: -10798905.658617 3874446.062724 -10762339.554976 3891581.043078 at scale 4513.9887049999998 The index was either too large or too small.

Failed to execute (ManageMapServerCacheTiles).

I am not familiar with the bundle errors nor the temporary directory where they are stored. How can I find this location to determine if it is contributing to the problem?

Unfortunately I cannot upgrade to 10.3.x at this time because we are required to support a legacy ArcIMS application until we migrate it to ArcGIS server.

Thanks again!

[RebeccaStrauch__GISP]

Hmmm....I was going to respond with the oath and found a "recovered" comment (from who knows when) that I obviously never sent. I'll just send this as is.....with the added note that the AppData folder will be hidden on many machines by default, but it you type the path in (after you get to the user folder) you should be able to find it.

Interesting thread.  And although my error "the index was either too large or too small" is coming from a different process, my guess is it related to all the above.

In this case, I'm testing 10.3 pre-release because I wanted to compare the 10.2.x bundle with the new 10.3 bundle, to see if it really was smaller.  So, although I've come at it several ways, what I'm really wanting to do is to use the "Upgrade Map Server Cache Storage Format" in the Server Tools->Caching toolbox.  The other tool I tested to see if this would word was an attempt to export/convert it back to an exploded cache to see if I would get it to work from that format.  I get the error from both.

For me,  I think the issue is I'm running out of temp space on my C drive.  Even though it never gets to zero, it seems that the more space I have available when I start one of these commands, the longer it will go before it crashes.  I know from previous experiences (when caching caches with 10.0) that there are many temp files that are created in

c:\users\<AGS service account>\AppData\Local\Temp      that never get cleaned out properly.  I also notice, many files are created here that are never cleaned out when the command crashes (although I am deleting between running my processes).    My test 10.2.2 bundled cache I am trying to convert is ~ 9GB.  I have about 13 GB free on C.

by the way... the <AGS service account> it the account that the actual windows service for ArcGIS Server is running under, not what you necessarily use to log into manager.