Geoprocessing service error

[MarkNorbury]

I've set up a geoprocessing service for a client which is an ETL tool. Most of the time it works, but occasionally it fails with an error similar to the following. On each occasion I'm running it for the same area of interest, same map layer to extract from and same format (shape file), ie. all the input parameters are the same each time, yet sometimes it works, sometimes it fails.

Error executing tool. MyExtract Job ID: jc9b53901a9dd43d489210ecb9fbe7533 : Traceback (most recent call last): File "\agisdev\arcgisserver\directories\arcgissystem\arcgisinput\GeoMy\ExtractDataTask.GPServer\extracted\v101\geoMy\ExtractData.py", line 522, in zipFolder = clipAndConvert(layers, areaOfInterest, featureFormat, rasterFormat, coordinateSystem) File "\agisdev\arcgisserver\directories\arcgissystem\arcgisinput\GeoMy\ExtractDataTask.GPServer\extracted\v101\geoMy\ExtractData.py", line 368, in clipAndConvert numFeatures = clipFeatures(lyr, aoi, featureFormat, zipFolderPath, scratchFolderPath, convertFeaturesDuringClip, dctColsMaster, dctRowsMaster, dctColsExport, wsExport) File "\agisdev\arcgisserver\directories\arcgissystem\arcgisinput\GeoMy\ExtractDataTask.GPServer\extracted\v101\geoMy\ExtractData.py", line 246, in clipFeatures with arcpy.da.SearchCursor(scratchpath, ["OID@"]) as cursor: RuntimeError: cannot open 'C:\Users\agis-srv\AppData\Local\Temp\geoMy\extractdatatask_gpserver\jc9b53901a9dd43d489210ecb9fbe7533\scratch\scratchfolder\ctLand_Dispositions.shp' Failed to execute (My Extract). Failed to execute (ExtractDataTask). Failed to execute (MyExtract).

The error seems to be that it can't find this file:
 'C:\Users\agis-srv\AppData\Local\Temp\geoMy\extractdatatask_gpserver\jc9b53901a9dd43d489210ecb9fbe7533\scratch\scratchfolder\ctLand_Dispositions.shp'

Why is it creating a scratch folder here? I've looked for this file in \agisdev\arcgisserver\directories\arcgisjobs, where I'd expect it to be but the job folder, jc9b53901a9dd43d489210ecb9fbe7533, doesn't exist here at all, not in any of the subfolders.

What's going on? Is it perhaps running out of memory occasionally and having to use the server's temp folder as its scratch workspace?

[JonathanQuinn]

Is this a multi-machine environment?  If so, GP services write to a local jobs directory for performance.  Do you see the job folder under the local jobs directory?

[MarkNorbury]

Hi Jonathan

Thanks for your reply.  No, this is a single-machine environment.

[JonathanQuinn]

Sorry, a better question, is the jobs directory on the local machine or a shared path?

[MarkNorbury]

The jobs directory is defined in the geoprocessing service as: \\agisdev\arcgisserver\directories\arcgisjobs.  This is actually c:\arcgisserver\directories\arcgisjobs on the local machine, because agisdev is its name and the arcgisserver folder is a share.

[JonathanQuinn]

The behavior you see is expected, then, in that the Server will write to the local jobs directory instead of the shared jobs directory.  This is mainly for performance. Did you check that directory for the jobID folder? Is this the OOTB Extract Data Task, or is it a customized version? Perhaps you can add more messages throughout the process, especially when it's creating the ctLand_Dispositions.shp and see if you can determine when it fails to create the data.

[MarkNorbury]

It creates the ctLand_Dispositions.shp shape file temporarily in the 'C:\Users\agis-srv\AppData\Local\Temp\geomy\extractdatatask_gpserver\jc9b53901a9dd43d489210ecb9fbe7533\scratch\scratchfolder\ folder then deletes it again after it has copied it to the jobs folder under C:\arcgisserver\directories\arcgisjobs\geomy\extractdatatask_gpserver.

It's as if sometimes ArcGIS gets out of sync with itself.  It creates the jobs folder and files in c:\users\agis-serv\AppData\Local\Temp then tries to retrieve from here, but has deleted them before it has had time to.

It strange how it works sometime, but on other occasions it doesn't.

[JonathanQuinn]

There is a cleanup policy for the arcgisjobs directory, but I'm not sure how the local jobs directory fits into that.  I think it's 10 minutes from the last time the contents were last modified so I don't see how the files would be deleted while the job is running.  Can you run ProcMon with a filter on the local jobs directory when running the gp service and see if there's another process that deletes files from the directory?

[MarkNorbury]

By the way, agis-srv is the name of the ArcGIS Server service account, ie. the account which runs the ArcGIS Server Windows service.  agisdev is the name of the server that ArcGIS Server is running on.