Manage Map Cache Tile: Errors at 10.2.2

[DavidColey]

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories.  Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

[MichaelVolz]

Ronnie:

Thank you for getting back to me so quickly on this thread and providing your advice and insight (This is the power of these forums).

I do have a follow up question based on your response.  Do you need to keep load off the entire server or just the mapservice that is being cached as there is quite a difference in the procedure to accomplish one or the other option?

Thank you for your assistance on this matter.

[DavidColey]

Michael, Ronnie - I suggest regardless of where you try to build the cache, secure the service first.  That way it cannot appear in the rest and has no chance of being accessed while the cache is running

David

[MichaelVolz]

Dave:

Did you get a notification from ESRI about a patch for this bug in v10.2.2?  Do you have the ability to test caching in the v10.2.2 environment anymore?

I did not see a patch for v10.3 though.

[DavidColey]

Michael-

No I did not and sorry I no I don't have the ability to cache in 10.2.2.  I did however find that I still receive a bundle move error when recreating all tiles if those tiles have not been deleted first.  I've adopted that as part of my workflow for 10.3:

Secure Service

Delete Tiles

Recreate all tiles (full)

Rercreat all tiles (aoi)

Unsecure Service (ie make public)

so it's 5 scripts in 3 tasks.

David

[MattMcGuire]

Note on ArcGIS 10.2.2 for Server Map Cache Consumption Patch | Samples and Utilities

This patch from March 30th is for BUG-000082467 not NIM104348. However, I gave it a try to see if it would also address this issue.

First, I tried to update an area of interest using Manage Map Server Cache Tiles on a bundled tiled service that is often under load (though the specific area that I was updating is not necessarily likely to have been in use at the time) I Got the error, "Output failure, error string = Error moving bundle"

I installed this patch on both machines in a cluster (10.2.2)

I reran the Manage Map Server Cache Tiles. Low a behold, it worked.

So, I guess more testing is in order.

[DavidColey]

That's great Matt.  Unfortunately I'm already at 10.3 so no help. Now if they would release one for 10.3 that would be great

[DavidColey]

I've just been informed by @David Crosby that the same patch will be incorporated into the 10.3.1 impending release

[RonnieRichards]

Matt,

This is great thanks for letting us know. We applied this to our environment and it seems to working for us. Amazing how cool caching is when it actually works and none of the silly work-a-rounds have to be initiated. Is ArcGIS the only software package that we all clap when it works? Whew finally that is fixed now we wait for the 10.3.1 release

Sure wish I heard this news from ESRI because we are on the bug watch list. That system needs to improve since we reported this issue last summer around the same time David reported it AND we should not have to report an issue to be aware of it. This information should be readily available and subscriptions to interested parties.

[MichaelVolz]

Ronnie:

This is an E-mail I received from an ESRI representative on 3/30/0215:

"

I’m writing because you’ve been attached to the following
bug: 

[#NIM104348 In ArcGIS Server 10.2.2, you're unable to
re-cache tiles while the existing tiles are being accessed.]

Esri has today released a patch for a similar, but different
issue:

ArcGIS
10.2.2 for Server Map Cache Consumption Patch

http://support.esri.com/en/downloads/patches-servicepacks/view/productid/66/metaid/2191

This patch was built specifically to address the following
bug:

[#BUG-000082467 ArcGIS for
Server opens a large number of files and does not release the file handles when
serving cached services.]

We believe that the patch for BUG-000082467 *MAY*
also help to address the issues reported as NIM104348.

We recommend that all users currently at ArcGIS for Server
10.2.2 apply this patch. Patches for other versions (10.2.1, 10.3) will be
released soon.

As you were added to NIM104348, I’d like to solicit your
feedback and input regarding the efficacy of this patch in terms of the cache
generation process by asking you to re-test failing caching workflows with this
patch installed.

If the results from the users asked to test this workflow are positive, we will update the patch details to also list  [#NIM104348 In ArcGIS Server 10.2.2, you're unable to re-cache tiles while the existing tiles are being accessed.]

We very much value your input, please let me
know if you have any questions at all."

I'm not sure why you did not get this alert if you were on the bug list.

[DavidColey]

Matt- I'd like to mark your reply as the correct reply to this long, long thread but unfortunately I didn't post this discussion as question.  Nevertheless, the ArcGIS
10.2.2 for Server Map Cache Consumption Patch

is the answer. 

After applying the patch, caches are now created using any tile management method and whether under load or not, cache directories are now renamed along with the service if a cached service is renamed, and service overwrites with a cache in place no longer leave behind a temp service.

Same at 10.3.

Thank you ESRI, I would consider this thread complete.

David