Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[MichaelVolz]

Not a problem.  I just wanted to make sure AGS users and administrators know the exact nature of the issue, as many environments do not need to worry about this issue if it is a single-server environment.

[MattiasEkström]

We have problems with bundle files gets locked in a non clustered environment. My approach has been to switch to exploded format instead of compact until this is fixed. 

[DavidColey]

Thanks.  I believe Ronnies comment from 2 weeks ago says that the issue still exists.  Like you say, we're just dealing with it.  Long-term, I am always concerned with constantly stopping and starting the Server windows service from a stability standpoint, so hopefully esri will resolve this locking issue (soon!)

[JonathanIrwin]

Hi all

We are not seeing the moving bundle error but I thought I'd check if any of you are seeing the Invalid XML registry error, as there appears to be a lot of commonality with the issues and David noted in one of his earlier posts that esri did reproduce at version 10.2.1.

So for reference purposes:

-  we have a 10.2.1 clustered environment (windows 2012)  we have just moved in the last 6 months from 10.1.

- we use Shared storage (UNC)

- We do cache in production and not to a staging environment.

- We publish service and then cache post service creation, so we are updating an empty cache or recreating tiles in existing.

- We need to update the cache nightly, which will be scripted in python. So the load on the service should be low.

- Unfortunately it looks like we will have to include within our script an AGS Service restart for each machine in the cluster before updating the cache.

Thanks all

Jonathan

[BobNutsch1]

Thanks for the update Jonathan, 

In your workflow, depending on how quickly the cache completes and how many users are using the cache and at what time they use the cache relative to your script, instead of a restart of the AGS Service you may want to instead use a stop service - run cache - start service.  Otherwise, in the middle of a cache run, a user could come in and lock the files that haven't yet been rebuilt in the caching process.

Regards, Bob

[SarahNoakes1]

Hi,

I'm experiencing issues with my cache at 10.2.2 too. I'm getting errors copying bundles, problems deleting files and generally finding it impossible to manage/update the cache without first destroying it and recreating it from scratch.

I don't seem able to view any information about NIM104348 which is irritating - It would be so useful to be able to track the progress of a known bug like this.

Does anyone know if this issue has been fixed in 10.3?  I see from the what's new document that "A new version of the compact cache storage format (v.2) has been developed to increase performance when consuming cached map and image services. The new version has also reduced number of files created in the cache directory. This is achieved by including the tile index information (.bundlx files) inside the .bundle files."  However, the document doesn't give any indication of which NIMs have been addressed...

Cheers,

Sarah

Cornwall Council

[JerryGarcia1]

I have been having similar problems. Constantly missing tiles. Support has been very helpful but we have been wasting lots of hours.  Really hope this will be fixed soon.

[DavidColey]

HI All -  This thread is getting a bit long, and with 10.3 out it's time to move to a new thread.  Unfortunately the error is still present with services under load. Let's continue at:

Manage Map Cache Tile: Errors at 10.3

Thanks,

David

[MichaelVolz]

David:

I created what should be a static cache with imagery in a multi-server environment and some of the bundles did not appear to be created correctly even though AGS reported that the cache level was completed successfully.  This same problem also occurred in a single server environment so I'm not quite sure if it's the multi-server environment that has caused this problem.

Now when I go to ArcCatalog just to re-cache the 1 problematic scale level, I receive the following error "Output failure, error string = Error moving bundle Failed to cache extent:2413695.345611 83295.257243 2646137.159947 384008.915350 at scale 8000 Failed to execute (ManageMapServerCacheTiles).

Have you encountered this error when trying to update your caches?  If so, do you have any idea what it means?

[RonnieRichards]

Hey Michael,

This error means an ArcGIS Server process has your bundle or index locked and cannot delete it to replace it. This happens in all environments multi or single so do not spend anymore time trying to figure it out. We spent 6 months working on the issue with ESRI support who were convinced it was an enronmental issue only to find this post with other users describing the same issue in the 10.2 release.

Restart your ArcGIS Server and attempt to rerun your process and the bundles should create. Try to keep all load off the server if this is not an option, try to resort to building the cache in another service, another server and then move the cache as David and others describe above.

Thanks

Ronnie