topic Re: Manage Map Cache Tile: Errors at 10.2.2 in ArcGIS Enterprise Questions

Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Tue, 13 May 2014 15:51:46 GMT

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

Re: Manage Map Cache Tile: Errors at 10.2.2

MichaelVolz — Tue, 13 May 2014 16:09:37 GMT

I do not have an answer for your question, but a possible workaround.

Has the mapservice that you have cached been around across multiple AGS versions (e.g. 10.0, 10.1, and 10.2)?

Do you cache the mapservice because it is too slow as a dynamic mapservice? If so, was the performance of the dynamic mapservice for this service been retested in later versions where performance of dynamic mapservices has been improved?

I recommend this approach because a dynamic mapservice (if performing at an acceptable level) is much easier to maintain as you do not need to account for updating the cache which is currently creating problems for you.

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Tue, 13 May 2014 16:23:22 GMT

Thanks Michael. The services have been through several updates, but this is the first time we received this error. We are able to create entirely new caches (which we have), but then cannot overwrite or delete those cache bundles. The tool deletes the bundle exchange file (bundlex) leaving in place the locked bundle, and then the tool cannot overwrite or delete the bundle as it has lost itsreference

Re: Manage Map Cache Tile: Errors at 10.2.2

MichaelVolz — Tue, 13 May 2014 16:40:32 GMT

Have you tried to use the same service as a dynamic mapservice instead of a cached mapservice?

I ask because my organization used a bunch of cached mapservices at v10.0, but at v10.2 with better performance we have been able to switch most services to dynamic so we no longer need to update the cache and deal with the problems that come with having to manage the update of the cache.

Does the below thread have any relevance to your current problem:

http://forums.arcgis.com/threads/104423-Caching-please-help!?highlight=manage+cache+error

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Tue, 13 May 2014 17:13:37 GMT

Of course. I've tried dynamic basemaps at each release and whether the data is local or sde, the performance suffers, at least for us. I would consider a workaround something like what I used to do at 10.0 or less: that is, caching on a dev o rstage environment and then moving the tiles to produciton via OS commands. But since 10.1, this is somewhat problematic with the advent of the Status gdb. Over time, this tends to corrupt the service.

Plus, we shouldn't have to go through these incredible scripting gymnastics just to get tools that are supposed to work; to work. Each release since 10.0 has introduced some new issue that is addressed in a subsequent release, which in turn introduces some new error . . . .
David

Re: Manage Map Cache Tile: Errors at 10.2.2

MichaelVolz — Tue, 13 May 2014 17:35:01 GMT

David:

Did your cached service exist at v10.2.0 (That is where my current production environment sits)? If so, did this problem exist at this release?

Or did you migrate from an earlier version (e.g. v10.1 sp1) directly to v10.2.2 whereby you bypassed v10.2.0? I ask because I might be creating cache mapservices soon, so I'm wondering if I will also encounter your problem when I need to update the cache.

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Tue, 13 May 2014 18:01:31 GMT

No, this is brand new at .2; I have never seen this error before in any release. No, I have not skipped any releases. We updated to 10.2.2 from 10.2.1 and in general, things have worked very well with this exception thus far.

Re: Manage Map Cache Tile: Errors at 10.2.2

ILIMAKENNEDY — Wed, 14 May 2014 17:43:37 GMT

This is discouraging, and I'm sorry that I don't have a solution. I was looking forward to installing 10.2.2 in order to fix the caching issues that I've had since migrating from 10.0 to 10.2, but with your new errors, I'm extremely hesitant to fix one thing only to introduce another.

My issues at 10.2 are with the following bugs (both fixed at 10.2.2) which makes my cached basemap appear to have missing tiles at some scales:
NIM099294 Improve performance of tiles access when the cache directory is on a shared network location (UNC).
NIM099281 A significant decrease in throughput occurs when requesting cached tiles through the ArcGIS Web Adaptor (IIS).

If you find a work around, please post.
Thanks!

Re: Manage Map Cache Tile: Errors at 10.2.2

MichaelVolz — Wed, 14 May 2014 17:55:45 GMT

Ilma:

You indicated that the below NIM is a problem you are encountering with v10.2.0 cached mapservices:

NIM099294 Improve performance of tiles access when the cache directory is on a shared network location (UNC).

Did you use this same cache at an earlier AGS version (e.g. v10.0 or v10.1) without this issue?

Re: Manage Map Cache Tile: Errors at 10.2.2

ILIMAKENNEDY — Wed, 14 May 2014 18:02:28 GMT

Michael,
Thanks for a quick response. I used the same cache at 10.0 where the tiles were stored on a NAS and accessible via share without issue. I've rebuilt the cache many times because I thought something was corrupted, but I still have a few scales where tiles disappear (even though the number of files on disk is the same each time) or some scales just take forever for the tiles to appear and the end users complain that the apps are too slow.

When I publish a service with a local copy of the cache directory on a different server, the map displays perfectly.

ilima

Re: Manage Map Cache Tile: Errors at 10.2.2

MichaelVolz — Wed, 14 May 2014 18:03:41 GMT

Are you using an exploded or compact cache?

Re: Manage Map Cache Tile: Errors at 10.2.2

ILIMAKENNEDY — Wed, 14 May 2014 19:07:38 GMT

Compact cache.

(I didn't mean to hijack the thread...only to get updates to the original poster's query.)

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Thu, 15 May 2014 16:09:59 GMT

Yes thanks Ilma I understand and as soon as I find out anything I will post it. Perhaps esri will chime in here at some point? 🙂

Thanks
David

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Thu, 15 May 2014 16:13:08 GMT

Sorry, one more bit of info. This is only occuring on my multi-machine production site. I am not encountering this error at my single-machine dev site. Of course, that makes sense as a single-machine site doesn't have to move a bundle file, right?
David

Re: Manage Map Cache Tile: Errors at 10.2.2

JustinHansen1 — Mon, 19 May 2014 18:28:02 GMT

We are experiencing brand new caching issues at 10.2.2. I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running RECREATE_ALL_TILES or DELETE_TILES operations of the Manage Map Cache Tiles (run as a tool or script) when run from a dedicated server gp cluster machine. Our clustered setup consists of a network file server storing our config, security and data-stores, as well as all server and system directories. Two servers serve as a mapCluster for map and image services, a single server in a gpCluster for caching and user-defined gp services, and a virtualized web server for the WebAdaptor.

This has occurred as of lastweek whether reading from the GIS SDE database OR from a data-store file gdb. All permissions for directory shares and folder and file security have been accounted for.

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc). During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite thebundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.

Has anyone encountered this?

Thanks-
David

I'm seeing the exact same behavior with one of our new development 10.2.2 ArcGIS Server sites.

I'm unable to re-cache a service unless I follow the same steps as David: delete the service, stop the ArcGIS Server service, delete the cache directory, start the ArcGIS Server service, and re-create both the service and the cache. ArcGIS Server is holding an exclusive read lock on the .bundle files even after the service is stopped or deleted. The exclusive read locks are only released after the ArcGIS Server service is stopped and started again. I'm assuming this behavior is what's causing the re-caching tools to fail. I've added the ArcGIS Server Account as a local admin on the ArcGIS Server server to eliminate any permission issues.

My site is a single machine site with both a UNC share cache directory and a local directory. The behavior is the same between the shared directory and the local directory.

I'm still hunting for a workaround.

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Tue, 20 May 2014 21:12:59 GMT

Hi Justin- Yes I am still experiencing the same error.

As far as a workaround goes, I could script the service deletion and run as a task, use task scheduler on each machine to perform a ArcServer.exe service stop, use an OS rmdir command in task scheduler to remove the directories, start the ArcServer.exe on each machine, then script and schedule service definition (sd) files for the map services, publish the sd's, and then finally re-run the caches. Sure seems like a lot of work for something that was so simple. Hey, if you think of another way let me know!
David

Re: Manage Map Cache Tile: Errors at 10.2.2

MichaelVolz — Wed, 21 May 2014 11:52:10 GMT

David:

In your multi-machine production site, do you have the cache files stored on each server in the cluster?

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Wed, 21 May 2014 19:37:37 GMT

Hi Michael-
No, the cache tiles are stored in a shared directory on a network server. A temp directory has been configured through ServerAdmin on each server with Server 10.2.2 installed, which of course the manage tiles operation should handle in terms of moving the bundles from the temp directory to the cache directory. I am certain this is per ESRI help documentation:

http://resources.arcgis.com/en/help/main/10.2/#/Local_cache_directories_on_the_server/01540000046r000000/

Thanks
David

Re: Manage Map Cache Tile: Errors at 10.2.2

ScottMcQuarrie — Wed, 28 May 2014 22:24:33 GMT

I am also experiencing this with the same error. Any thoughts ESRI?

Re: Manage Map Cache Tile: Errors at 10.2.2

DavidColey — Thu, 29 May 2014 15:49:05 GMT

Interesting, glad (well, not glad) to hear it's not just us. I have a support request in with ESRI at the moment but thus far no real replies from them. Whatever I find out I will post.

In further testing, I have also discovered this behavior is happening on my stand-alone, single machine edn dev box. What I mean is that there shouldn't be any error regarding the 'moving' of bundle or bundlex files from any physical directories to the cache directory,unless it's an error moving fromt the jobs directory to the cache directory (I guess).

However, I'm beginning to suspect that this particular error is just the latest manifestation of other problems with service overwrites and caching in general. For example, we have long had difficulites in overwriting services that have a cache in place. When we tell server (through the server editor at overwrite time), that the cache will be updated manually typically the overwrite will leave behind it's 'scratch' service, visible in the rest or catalog. Often at that time, the original service will leave behing a .glock file, not allowing for further service manipulation until the site is bounced via stopping the ArcServer.exe processes.

Thanks-
David