topic Re: Manage Map Cache Tile: Errors at 10.3 in ArcGIS Enterprise Questions

Manage Map Cache Tile: Errors at 10.3

DavidColey — Fri, 16 Jan 2015 18:48:02 GMT

A continuation of -

Manage Map Cache Tile: Errors at 10.2.2

Since we are moving to 10.3, I thought I'd re-post the discussion to keep it current with the 10.3 release. Unfortunately, at this time the error on bundle move is still occurring when the cache service is in use or under load. Although the 10.3 release is thus far proving to be more efficient and responsive this error still occurs. Overwriting cache services has stabilized however with the re-inclusion of the 'Keep Existing Cache' parameter at overwrite, and overall communication between the desktop and the server directories is much improved, at least on our network.

In general, (because I can allow my basemaps to be offline for a couple hours each week or each month) I have adopted a modified strategy where I stop the service, set the service to private (Admin role access), start the service, and begin caching - all via python scripts run through task scheduler.

I think I prefer this to my previous method of storing both a private service and cache and a public service and cache, then caching to private, stopping the site, and overwriting the bundles via an OS xcopy.

Hopefully esri will keep trying to solve this issue!

David

Re: Manage Map Cache Tile: Errors at 10.3

MattMcGuire — Fri, 16 Jan 2015 21:28:31 GMT

We provide 24 hour service. Practically speaking, we can get away with some brief downtime, but not at any time of day that I'd like to be awake. I definitely can't take them down long enough to rebuild them.

Thanks for starting a new thread. I suppose I should think about automating a routine:

Build in a private service
Stop public service
XCopy tiles, replacing with newer
start public service
Then if it works, schedule it for sometime after the witching hour.

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Fri, 16 Jan 2015 23:30:30 GMT

Yeah, that's what I've been doing as well but I've since been able to carve out some time where some of my internal caches don't have to be up 24-7

Re: Manage Map Cache Tile: Errors at 10.3

BobNutsch1 — Tue, 20 Jan 2015 15:26:48 GMT

We are still having this problem and still waiting to hear back from Esri. Meanwhile, we've found another "feature" of ArcGIS Server that is creating some severe stability issues.

We have the source of the problem narrowed down to a couple of things with one of the biggest suspects is OpLocks. We store our cache and our configs on a network file share via DFS. The DFS in turn is using a storage system that has OpLocks turned on. I'm becoming very suspicious that OpLocks is creating our stability issues and perhaps the caching problem (too many cooks in the kitchen...ArcServer wants a file locks, the storage system wants file locks).

While working with Esri on our stability issue and another issue, it became apparent that Esri is working to making ArcServer work/work better with DFS/OpLocks. Apparently DFS, SAN, NAS, etc. all can use OpLocks.

So my question to you folks who are struggling with the cache bug, what is your data storage system for your cache tiles? Does that storage system use OpLocks?

Regards, Bob

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Tue, 20 Jan 2015 17:24:50 GMT

Bob- good to know, wish I could say I was surprised. I can tell you right now that I moved our server directories off of our SAN and onto a local raided array back at 10.2 for both performance and stability reasons.

Interestingly, even though the ArcServer directories, stores are now local to the file server, another mapped disk array (our GIS drive for mapshop, support, etc) became dangerously overloaded and drastically killed ArcServer performance. So apparently even though the offending disks are part of a SAN array, and not local to the server, the server didn't make that distinction....

David

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Thu, 12 Feb 2015 23:18:00 GMT

Well sad to say that in our clustered production envrionment the 'Update Storage Format' appears to be corrupting the existing 10.2.2 bundles when upgrading to 10.3 format. This did not occur on my stand-alone dev machine. In prodcution, the tool runs correctly, but then to tiles are visible. Upon running a recreate all tiles, the bundle move errors occur even with no load on the service.

As such, I am deleting and the recreating new caches. Once I have some new caches in place, I'll re-try a recreate_all and post back what I find....

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Wed, 01 Apr 2015 18:38:06 GMT

I have once again found that I still receive a bundle move error when recreating all tiles. So what I have done is move to a workflow that

Task 1:

stopsService.bat

securesService.py

startsService.bat

Task 1 removes load and ensures the service can't be accessed

Task 2:

DeletesTiles.py

Task 2 ideally would minimize the amount of time a new bundle file is in memory or in a temp loaction or in tranaction during the Recreate tiles tasks below.

Task 3:

RecreateFullTiles.py

Task 4:

RecreateAioTiles.py

UnSecureService.py

Interestingly, the workflow performs without error when I execute the scheduled task manually my desktop. The workflow fails when the trigger condition executes task because the DeleteTiles.py script fails with at least one:

Failed to cache extent: -9157755.541216 3087885.815688 -9079507.910821 3130848.734828 at scale 9027.9774109999998 The index was either too large or too small.

error.

Hard to figure. Something about a trigger condition runnning the task makes the task behave differently.

David

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Mon, 27 Apr 2015 20:25:17 GMT

Hello all-

The ArcGIS
10.3 for Server Map Cache Consumption Patch

is the answer.

After applying the patch, caches are now created using any tile management method and whether under load or not, cache directories are now renamed along with the service if a cached service is renamed, and service overwrites with a cache in place no longer leave behind a temp service.

Thank you ESRI, I would consider this thread complete.

David

Re: Manage Map Cache Tile: Errors at 10.3

PhilipSlater — Tue, 05 May 2015 17:07:40 GMT

Thanks for the link but is seems to be linking to the 10.2.2 patch. So here is the link for the 10.3 patch.

ArcGIS 10.3 for Server Map Cache Consumption Patch | Samples and Utilities

Thanks.

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Tue, 05 May 2015 20:37:06 GMT

thanks sorry about that all

Re: Manage Map Cache Tile: Errors at 10.3

Anonymous User — Mon, 04 Apr 2016 21:18:09 GMT

Great stuff, David.

I just used your method of stopping the ArcGISServer.exe service and was able to delete the locked bundle files.

We're still on ArcGIS 10.1SP1, will be upgrading to 10.3.1 in the near future!

Re: Manage Map Cache Tile: Errors at 10.3

MichaelVolz — Tue, 05 Apr 2016 13:18:02 GMT

David:

Would you know if ArcGIS for Server v10.3.1 comes already with the ArcGIS 10.3 for Server Map Cache Consumption Patch? I upgraded straight to v10.3.1 from v10.2.2 so I'm wondering if the issues associated with this patch were handled with the upgrade.

I'm planning on changing cache scales on existing tiled mapservices and I'm just wondering if I will have issues under load as I had no problems in my test environment, but no one is really hitting that service in test like production is being hit.

Your feedback is greatly appreciated.

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Wed, 06 Apr 2016 21:35:39 GMT

sounds good. I'm moving to 10.4 shortly in my dev environment and will report any difficulties,etc

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Wed, 06 Apr 2016 21:38:19 GMT

Hi Michael, no I did not and have not encountered any issues at 10.3 or 10.31 and I've been running weekly cache updates since sometime last year. The consumption patch was included at 10.3 and at 10.3.1.

Re: Manage Map Cache Tile: Errors at 10.3

ChrisRobinson — Thu, 02 Jun 2016 13:04:51 GMT

All,

I had been plagued by the "Failed to cache extent:" error for the better part of a year now while on ArcGIS for server 10.2.x. Most every time I would try and rebuild it would throw the error, most of the times at different scales and different extents. I too had come to the work around for creating a batch file that stops the ArcGIS Server service, then delete all cache tiles while keeping the scale level folders in tact, starting the ArcGIS Server service and then rebuilding the cache. It always seemed to work on a full rebuild when there were no existing tiles. I had even put a ticket in with ESRI to have them look to which we never found a concrete solution so the ticket was eventually closed. I chalked it up to be that it was something with my data that was causing it.

I happened to stumble across this thread yesterday (was lead here by the 10.2.2 thread discussing OPLocks) and applied the patch to a 10.3 server and have been able to rebuild all my caches so far without the "Failed to cache extent:" error displaying.

Thank you all for contributing to these threads as they do really help others to work through similar issues.

Chris

Re: Manage Map Cache Tile: Errors at 10.3

DavidColey — Thu, 02 Jun 2016 14:55:27 GMT

Sounds good. I am also happy to report no caching errors regardless of scenario (new, rebuild, AOI use) at 10.4. I expect that will continue at 10.4.1