Using the Spatial Analysis Service Extract Data feature

[SteveBurdette]

We're trying to take advantage of the new extract data option in the ArcGIS JavaScript API, and are testing it out using the OurServer/ArcGIS/rest/services/System/SpatialAnalysisTools/GPServer/ExtractData/submitJob REST API method. 

We shared our feature class as a service (checking the Extract option for the feature service), and are trying to do a simple shapefile extract to test it out.  We have started the SpatialAnalysisTools service, and are testing with the following inputLayers data:

[ { 'url' : 'OurServer/ArcGIS/rest/services/ServiceName/FeatureServer/0', 'filter': '1 = 1' } ]

Whenever we submit a job it errors out with a "Failed to execute" error.  The message code is "AO_100026" and the extra log text in the system error logs is "Missing organization data store." 

We have version 10.4.1 of ArcGIS server installed. 

Our feature class works fine with the other REST web api methods, and the database data store has been validated.  Any ideas why we are getting the missing organization data store error?

Thanks,

Steve

[DavidWendelken]

This is probably a bit late to help the original poster, but it might help someone else.  (It certainly would have helped me!)

I was trying to export data using the Analysis Widget in Web AppBuilder.  I got the same message code but our text just said "Extract data failed."  

We have a federated portal and web server.

I staged the app to be ready to submit a request to extract the data and then I pressed F12 on the IE browser to get the debug helper window. (Other browsers have similar features, adjust as appropriate.) 

I turned on the monitor to report on network traffic from and to the browser.
I looked for the Post command that requested the extract and saw that the extract job had been submitted successfully.  I then looked for the Get command and checked its response.   The response message text was equally uninformative (but in other circumstances might not be).  However, it also had a JOB ID field.  That info was golden, as you'll see in a bit.

I then went to our esri webserver to the gis administrator page.   I picked the SITE tab and then the DIRECTORIES tab.   This gave me a list of directories to look into.

In my case, the directory for JOBS turned out to be useful.  It appears that all the tools that accept jobs have their own directory.  In this case, it was the SYSTEM/SpatialAnalysisTools_GPServer file directory that was of interest.

I noticed a bunch of file folders inside that directory with gibberish alpha-numeric names.   It turns out the file folders are named for the JOB ID!  

Inside that jobs folder was a SCRATCH folder.   I found a MESSAGES.XML file that turned out to be a more detailed log of the extract job.   It pointed us to the python script that was failing, to the line number of the error, and gave us a more useful error message than "ExtractData Failed." 

We're still tracking down what the exact problem is, but now we've got a clue.   (I'll post back if/when we find the solution.)

[DavidWendelken]

In our case, the gis analysts had altered the layer names in the portal "service" to be different from the layer names in the underlying web service.   Once we made the layer names match in both places, it worked like a charm.

[SteveBurdette1]

David, 

Thanks for your replies and information.  My messages.xml file on the server for my failed job looks like this...I'm not seeing much to go by here.  Do you see anything that might give a clue as to what's going on?  

<GPMessages xsi:type='typens:GPMessages' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xs='http://www.w3.org/2001/XMLSchema' xmlns:typens='http://www.esri.com/schemas/ArcGIS/10.5'><GPMessages xsi:type='typens:ArrayOfGPMessage'><GPMessage xsi:type='typens:GPMessage'><MessageType>1</MessageType><MessageCode>0</MessageCode><MessageDesc>Executing (ExtractData): ExtractData {&quot;url&quot;:&quot;http://myserver.domain:6080/arcgis/rest/services/Folder/Feature1/FeatureServer/0","filter":"City = &apos;City1&apos;&quot;} # false ShapeFile # #</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>2</MessageType><MessageCode>0</MessageCode><MessageDesc>Start Time: Fri Sep 15 11:05:48 2017</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>0</MessageType><MessageCode>0</MessageCode><MessageDesc>Running script ExtractData...</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>0</MessageType><MessageCode>0</MessageCode><MessageDesc>File &quot;C:\Program Files\ArcGIS\Server\ArcToolBox\Services\scripts\extractdata.py&quot;, line 275, in &lt;module&gt;
hostedgp = agolgp.HostedGP(5, 4)</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>0</MessageType><MessageCode>0</MessageCode><MessageDesc>File &quot;c:\program files\arcgis\server\arcpy\hostedgp\__init__.py&quot;, line 46, in __init__
raise GPCloudExec(&quot;HostedGP&quot;, unicode(err).encode(&apos;utf-8&apos;))</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>0</MessageType><MessageCode>0</MessageCode><MessageDesc>GPCloudExec</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>0</MessageType><MessageCode>0</MessageCode><MessageDesc>AddExceptionError</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>100</MessageType><MessageCode>1</MessageCode><MessageDesc>{&quot;messageCode&quot;: &quot;AO_100026&quot;, &quot;message&quot;: &quot;ExtractData failed.&quot;}</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>0</MessageType><MessageCode>0</MessageCode><MessageDesc>Completed script ExtractData...</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>100</MessageType><MessageCode>-2147467259</MessageCode><MessageDesc>Failed to execute (ExtractData).</MessageDesc></GPMessage><GPMessage xsi:type='typens:GPMessage'><MessageType>3</MessageType><MessageCode>0</MessageCode><MessageDesc>Failed at Fri Sep 15 11:05:49 2017 (Elapsed Time: 0.46 seconds)</MessageDesc></GPMessage></GPMessages></GPMessages>

Thanks,

Steve

[JonathanQuinn]

You can check the job status directly through the REST endpoint:

http://<server>.<domain>.com/<context>/rest/services/System/SpatialAnalysisTools/GPServer/ExtractDat... 

Once you submit the job through the REST endpoint, get the job ID and substitute it for the job ID in the URL above.

That will give you the messages in a clearer format.

For anyone else troubleshooting an issue like this, you can run Fiddler as the user running the ArcGIS Server Windows service so you can see the outbound requests that Server is making when running the tool.  You'll see the request to query the URL using the 1=1 where clause.

I did find a strange issue where if the service was running on a federated Server and shared with Everyone, Servers internal request for the service included a token parameter, but the token value was None:

https://<server>.<domain>.com/server/rest/services/<service>/FeatureServer?token=None&f=json

[DavidWendelken]

This is the only useful part I noticed:

c:\program files\arcgis\server\arcpy\hostedgp\__init__.py

&quot;, line 46, in __init__
raise GPCloudExec(&quot;HostedGP&quot;, unicode(err).encode(&apos;utf-8&apos;))

I don't have an xml editor on the machine I'm reading this on, so it would be easy to miss something else. 

It looks like the error may be line 46 of the above file.  Perhaps you can  look there and see if it becomes more obvious.  If not, perhaps you can add some diagnostic code to it to display additional values for you to check out.   There are some MessageCode and MessageType values scattered about in the log file, its possible those values may be useful while investigating the python script.

Best of luck!

[AndresCastillo]

you can also test the extract data directly within the web map (not the app) as it uses the same portal tool.

Here are the errors I encounter from setting SpatialAnalysisTools>Parameters>Properties>Message Level to info in ArcGIS Server Manager, and observing fiddler/Chrome Dev Tools Network Traffic:

description=File "C:\Program Files\ArcGIS\Server\ArcToolBox\Services\scripts\extractdata.py", line 277, in <module>
    if not hostedgp.CheckPrivilege(PRIVILEGE_CREATE_ITEM):

description=File "c:\program files\arcgis\server\arcpy\hostedgp\__init__.py", line 273, in CheckPrivilege
    raise GPCloudExec("CheckPrivilege",unicode(err).encode('utf-8'))

description=GPCloudExec

description={"messageCode": "AO_100026", "message": "ExtractData failed."}


System/SpatialAnalysisTools.GPServer
	Portal request self failed with error Service Unavailable. (status code 503).
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This is a federated and hosting ArcGIS Server.

According to the error above, the gp service may have a permission issue that it cannot access to ArcGIS Feature Service data.

The fix may be to grant the ArcGIS Server service account permissions to the data found in the Datastore and Hosting ArcGIS Server directories.

https://enterprise.arcgis.com/en/server/latest/manage-data/windows/making-your-data-accessible-to-ar...