Unable to cancel stalled cache processing job

[Anonymous User]

Original User: stevel

I'm using ArcGIS Server 10.1 and have created a cached service.

For some reason the cache creation failed, so I have 2 cache creation job services running. I am unable to cancel the stalled process.

See the screenshot below, where I choose Cancel Job there is no effect - the display briefly displays "Updating Status" then returns to show the 5.2% job still listed as Processing.

[ATTACH=CONFIG]18448[/ATTACH]

Any ideas for how I can kill that process? I've tried deleting and recreating the map service, rebooting the server, etc.

A possibly related problem is that the service keeps changing itself from Update Cache Manually to Update Cache Automatically. How can I prevent this, too?

[ATTACH=CONFIG]18449[/ATTACH]

Thanks,
Steve

[JustinHunter]

I'm having the same issue after we experienced a power outage in the middle of my caching job. I have a ticket into support but in the meanwhile I was hoping someone may have experienced this issue.

I tried to RECREATE_EMPTY_TILES but that succeeded after 2 hours when I had about another 4-5 days of caching remaining.

Thanks

[Anonymous User]

Original User: simpsonr

You might try restarting the caching service under "system" folder.

[Anonymous User]

Original User: timmerspk

You'll find the job cancels but the Status.gdb does not get updated. 

Generally regard the Status.gdb as a bit of a guide. 

I don't rely on it too much.  Jobs crash and your status will be still updating.

Just restart the caching tool and restart your job.  The status will fix itself (mostly).

[JustinHunter]

What about all the bundles that are still previously locked? Do I delete the entire bundle or just the lock file?

I've run into this problem a lot and I'm considering rolling back to ArcGIS 10, 10.1 doesn't seem stable enough when it comes to caching big jobs (10+ days).

edit- i'd also like to know what RECREATE_EMPTY_TILES does. It seems like it goes through every bundle previous just to make sure it's all there. Surely it's faster.... right?

[Anonymous User]

Original User: timmerspk

I've actually found 10.1 a little more stable.

Supposedly sp1 is getting a better way to view the status  (some sort of dashboard??)

The lock files have never been an issue for us.

We're also building 200GB-700GB caches.

Status.gdb has good for finding exactly which layer/scale is causing the problems... usually broken topology.

I've also had much more luck moving network based data locally into fgdbs... our flakey network means your 10+day caching stalls.

[GarimaTiwari · Accepted solution]

Steve,

The chances are none of the cache jobs on your server are in-progress. This issue occurs when the cache service instances have crashed due to some error and have failed to update their cache status. We have tried to fix such failure scenarios in 10.1 SP1.

To check if cache job is indeed in progress, check how many instances of caching gp service are being used.
a)If no instances are in use then the chances are that the cache job has failed and the instances crashed before being able to update their status.
b) If two or more instances of the caching gp service are being used then there is a possibility that one or both of the cache job is/are in progress.

To stop any cache job is in progress, stop the caching gp service.

To fix the cache status, run "ManageMapServerCacheStatus" tool with update mode set to "REBUILD_CACHE_STATUS". This will provide you the current cache status of your service. You can open up the status.gdb file to determine which map extents have Partial cache, complete cache or missing cache.

Using the information available in cache status & status.gdb, submit a new cache job to update the cache of the service. If the new job fails then please contact ESRI customer support, we will try to investigate what is causing the problem.

>> A possibly related problem is that the service keeps changing itself from Update Cache Manually to Update Cache Automatically. How can I prevent this, too?
This is not related to the issue described above. This setting comes into play only when one updates the min ???max scale of the cache service.

-Garima

[Anonymous User]

Original User: crombezk

I have the same exact problem on ArcGIS Server 10.2.2.

Creating a cache failed on an image service over the weekend and reported this error:

Drawing failure, error string = Failed to create an image cache tile for OFFLINE_IMAGERY at R280B0C28950.Failure to access the DBMS server [08S01:[Microsoft][SQL Server Native Client 10.0]TCP Provider: An existing connection was forcibly closed by the remote host. ] [CCGISR.CCGIS.AMD_OFFLINE_IMAGERY_CAT]

1:500 996646.614583 1263213.281250 1010864.496528 1277431.163194

This morning I selected to recreate empty tiles and it only created some of them.
There are still several missing tiles that will not draw.

The job that failed still reads status "processing" and will not cancel.
(same as stevel... says "Updating Status" but then goes right back to processing.

Every time I change the update mode to manual it just resets itself to automatic.

I tried running the caching tools in the system folder Garima suggested, but they just keep failing.

ERROR 001428: Failed to retrieve configuration for service
ArcGIS Server requires a service type (ParentType) and a server object extension type (Type)
Failed to execute (Manage Map Cache Status)

the solution for Error 001428 is "Verify that the cached map/image service is running on the server"

YES, my service is running.

[DavidColey]

I too am experiencing brand new caching issues at 10.2.2.  I am unable to either DELETE_TILES or RECREATE_ALL_TILES from an existing cache. The error we keep receiving is:

Line 30Output failure, error string = Error moving bundle Failed to cache extent: -9223211.962860 3087885.815688 -9079003.170544 3271103.291998 at scale 577790.55428899999.

The error occures when running a RECREATE_ALL_TILES or operation of the Manage Map Cache Tiles (run as a tool or script) from our dedicated server gp cluster machine. 

This has occurred as of yesterday whether reading from the GIS SDE database OR from a data-store file gdb. 

When the error occurs, the entire cache becomes corrupt, making any other caching operations in-operable (delete cache, delete tiles, etc).  During the recreate, the bundlx file is being removed but the bundle file is not, thus not allowing the tool to overwrite the bundle from the admin-defined D:/arcgistemp folder(s) on the GP server.

Thus far, the only solution has been to kill the service, stop the site, and then remove the entire cache directory, restart the site, recreate both the service and the cache.  Not ideal.

Thanks-
David

[SimonSchütte_ct]

It is possible to stop jobs from the ArcGIS Server Administrator Directory: Cancel a job | ArcGIS REST APIs | ArcGIS Developers