Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[DavidColey]

Hi Rebecca - I have ye to be able successfully update cache storage format to combined bundle type at any release.  For us though it's not really that big a deal as I just go ahead and delete the old cache and create new....

David

[JohnFell]

We ultimately found that because we use (DFSR) distributed file system replication on cached images between tiles that there was some contention in the replication process causing problems. After disabling the replication process from one GIS server to the other for cached images we stopped getting errors.

[DavidColey]

Hi Mattias et al - No sorry I have not.  I have a request ticket in with a Robinson at ESRI but have not heard back from him in many days.  So for us, nothing has chnaged since June 6 sorry to say.  I'm certain this is a bug but am not expecting much back from ESRI at this time, esp with the UC coming right up.  Unfortunate, as I'd really like to get a NIM created for this.  As Mattias says, if anyone has found a solution for the bundle move error please share. 

As far as the extent error you are receiving, I have had that occur when in earlier releases when using an AOI fc.  I was able to stop the error by either keeping all my cache data and the AOI in a fdgb, OR by keeping all cache data and the AOI in our SDE.  I always received the extent errror when I mixed them-
David

[MattMcGuire]

I'm having the same problem. Namely, "esriJobMessageTypeError: Output failure, error string = Error moving bundle", and then the cache generation fails.

Our configuration is very similar to OPs. However, I have not been able to fix it using the method described in the OP.

1) Kill the service
2) Stop the site
3) Remove the entire cache directory
4) Recreate service
5) Recreate cache


Are you deleting the cached directory for the entire site? Or just the cache directory for the service? And by stopping the site, do you mean stop each machine participating in the site?

[DavidColey]

Hi Matt-
No, just the cache directory for the service.  I delete the service, then delete that services' cache folders.  You may or may not have to stop each ArcServer.exe process on each of your site machines (ie stopping the site) in order to delete the services' cache folders'.  We found that we had to in order to remove the services' cache folders containing the corrupted bundles because windows would not let us completely delete the directories otherwise. 

Robinson at ESRI contacted me yesterday asking if I was seeing the same behavior (when running the ManageMapTiles Recreate or Delete Tiles) from both python and catalog to which I responded yes I am-
David

[DavidColey]

Greetings all, Did anyone find out about this issue at UC?

[DavidColey]

A small update: The Recreate_All_Tiles option worked again last week (repeatedly using my python IDLE) on our stand-alone development server, but continues to fail with the same frequency and 'error moving bundle file' error in our distributed three-machine, 2 cluster production environment where our shares, directories and stores are stored on our 1Gig Lan connected file server.  It would be really, really great if ESRI could comment.

David

[DanielHall_Ballester]

Sorry to re-start this thread, but are there any updates on this?

I'm having near-identical issues to the OP with a very similar set up.

A 1st cache of a service works.  Any subsequent caching, whether it be the full extent of the service or a number of smaller AOI's within the service leaves behind some blank spaces and some areas where a cache only exists at some scales.

I will raise a ticket with ESRI today, but was wondering if anybody had received a response/workaround/fix to this as I'm aware that a number of you had tickets in with ESRI.

Many thanks

Dan

[MattiasEkström]

I have a ticket with ESRI Sweden, they have been able to reproduce this error but have no solution yet.

Meanwhile I've been testing to use the EXPLODED Storage Format to get rid of the bundle-files and see if some of all the png-images gets locked as well, so far they have not.
I only have a couple of cached services that are updated regularly, I will publish them with EXPLODED storage format until I hear of another solution/workaround for the COMPACT storage format.

[DavidColey]

Thanks Matt, interesing result regarding the Exploded v Compact. 

I too am working with Jon out of Redlands to get this solved.  We ran some I/O tests yesterday using IOzone and thus far have determined that throughput does not seem to be the culprit.  We will be running a ProcMon test later today and I will post the results