Manage Map Cache Tile: Errors at 10.3

[DavidColey]

A continuation of -

Manage Map Cache Tile: Errors at 10.2.2

Since we are moving to 10.3, I thought I'd re-post the discussion to keep it current with the 10.3 release.  Unfortunately, at this time the error on bundle move is still occurring when the cache service is in use or under load.  Although the 10.3 release is thus far proving to be more efficient and responsive this error still occurs.  Overwriting cache services has stabilized however with the re-inclusion of the 'Keep Existing Cache' parameter at overwrite, and overall communication between the desktop and the server directories is much improved, at least on our network.

In general, (because I can allow my basemaps to be offline for a couple hours each week or each month) I have adopted a modified strategy where I stop the service, set the service to private (Admin role access), start the service, and begin caching - all via python scripts run through task scheduler.

I think I prefer this to my previous method of storing both a private service and cache and a public service and cache, then caching to private, stopping the site, and overwriting the bundles via an OS xcopy.

Hopefully esri will keep trying to solve this issue!

David

[MattMcGuire]

We provide 24 hour service. Practically speaking, we can get away with some brief downtime, but not at any time of day that I'd like to be awake. I definitely can't take them down long enough to rebuild them.

Thanks for starting a new thread.  I suppose I should think about automating a routine:

1. Build in a private service
2. Stop public service
3. XCopy tiles, replacing with newer
4. start public service
5. Then if it works, schedule it for sometime after the witching hour.

[DavidColey]

Yeah, that's what I've been doing as well but I've since been able to carve out some time where some of my internal caches don't have to be up 24-7

[BobNutsch1]

We are still having this problem and still waiting to hear back from Esri.  Meanwhile, we've found another "feature" of ArcGIS Server that is creating some severe stability issues. 

We have the source of the problem narrowed down to a couple of things with one of the biggest suspects is OpLocks.  We store our cache and our configs on a network file share via DFS.  The DFS in turn is using a storage system that has OpLocks turned on.  I'm becoming very suspicious that OpLocks is creating our stability issues and perhaps the caching problem (too many cooks in the kitchen...ArcServer wants a file locks, the storage system wants file locks).

While working with Esri on our stability issue and another issue, it became apparent that Esri is working to making ArcServer work/work better with DFS/OpLocks. Apparently DFS, SAN, NAS, etc. all can use OpLocks.

So my question to you folks who are struggling with the cache bug, what is your data storage system for your cache tiles?  Does that storage system use OpLocks?

Regards, Bob

[DavidColey]

Bob- good to know, wish I could say I was surprised. I can tell you right now that I moved our server directories off of our SAN and onto a local raided array back at 10.2 for both performance and stability reasons.

Interestingly, even though the ArcServer directories, stores are now local to the file server, another mapped disk array (our GIS drive for mapshop, support, etc) became dangerously overloaded and drastically killed ArcServer performance.  So apparently even though the offending disks are part of a SAN array, and not local to the server, the server didn't make that distinction....

David

[DavidColey]

Well sad to say that in our clustered production envrionment the 'Update Storage Format' appears to be corrupting the existing 10.2.2 bundles when upgrading to 10.3 format.  This did not occur on my stand-alone dev machine.  In prodcution, the tool runs correctly, but then to tiles are visible.  Upon running a recreate all tiles, the bundle move errors occur even with no load on the service. 

As such, I am deleting and the recreating new caches.  Once I have some new caches in place, I'll re-try a recreate_all and post back what I find....

[DavidColey]

I have once again found that I still receive a bundle move error when recreating all tiles.  So what I have done is move to a workflow that

Task 1:

stopsService.bat

securesService.py

startsService.bat

Task 1 removes load and ensures the service can't be accessed

Task 2:

DeletesTiles.py

Task 2 ideally would minimize the amount of time a new bundle file is in memory or in a temp loaction or in tranaction during the Recreate tiles tasks below.

Task 3:

RecreateFullTiles.py

Task 4:

RecreateAioTiles.py

UnSecureService.py

Interestingly, the workflow performs without error when I execute the scheduled task manually my desktop.  The workflow fails when the trigger condition executes task because the DeleteTiles.py script fails with at least one:

Failed to cache extent: -9157755.541216 3087885.815688 -9079507.910821 3130848.734828 at scale 9027.9774109999998 The index was either too large or too small.

error.

Hard to figure.  Something about a trigger condition runnning the task makes the task behave differently.

David

[DavidColey]

Hello all-

The ArcGIS
10.3 for Server Map Cache Consumption Patch

is the answer.

After applying the patch, caches are now created using any tile management method and whether under load or not, cache directories are now renamed along with the service if a cached service is renamed, and service overwrites with a cache in place no longer leave behind a temp service.

Thank you ESRI, I would consider this thread complete.

David

[PhilipSlater]

Thanks for the link but is seems to be linking to the 10.2.2 patch.  So here is the link for the 10.3 patch.

ArcGIS 10.3 for Server Map Cache Consumption Patch | Samples and Utilities

Thanks.

[DavidColey]

thanks sorry about that all