JSONToFeatures_converson - ArcGIS Server GPService fails

[PF1]

I love the new JSONToFeature_conversion tool that was just released at 10.2

Works great when running it from ArcGIS Desktop, but not from ArcGIS Server...  Here is a simple script and output.  I would rather pass in the RAW JSON data rather than store it to a file so the script accepts basic text (string param type) and creates the JSON file, stores the text and runs the tool. 

import os import arcpy from datetime import datetime   ###################################### ######  Getting params   ###### ###################################### input_json=arcpy.GetParameterAsText(0)     ###################################### ######  Store Data   ###### ###################################### dt="DATA_EXTRACT_"+datetime.today().strftime("%y%m%d_%H%M%S") json_file=os.path.join(arcpy.env.scratchFolder,"%s.json"%dt) fc=os.path.join(arcpy.env.scratchGDB,dt)    ###CONVERT JSON TO FC### arcpy.AddMessage("Converting JSON to FC...") f=open(json_file,'w') f.write(input_json) f.close() res=arcpy.JSONToFeatures_conversion(json_file,fc)    ###Storing result...###  arcpy.SetParameter(1,arcpy.FeatureSet(fc))

Here is a real simple JSON input:

{"features": [{"geometry": {"y": 35.748388281000075, "x": -114.85761096699997}, "attributes": {"SITENAME": "MYSITE", "OBJECTID": 1}}], "fieldAliases": {"SITENAME": "SITENAME", "OBJECTID": "OBJECTID"}, "fields": [{"alias": "OBJECTID", "type": "esriFieldTypeOID", "name": "OBJECTID"}, {"alias": "SITENAME", "length": 70, "type": "esriFieldTypeString", "name": "SITENAME"}], "displayFieldName": "SITENAME", "spatialReference": {"wkid": 4269, "latestWkid": 4269}, "geometryType": "esriGeometryPoint"}

If you run this from desktop then here is the result:

Executing: JsonToFc "{"features": [{"geometry": {"y": 35.748388281000075, "x": -114.85761096699997}, "attributes": {"SITENAME": "MYSITE", "OBJECTID": 1}}], "fieldAliases": {"SITENAME": "SITENAME", "OBJECTID": "OBJECTID"}, "fields": [{"alias": "OBJECTID", "type": "esriFieldTypeOID", "name": "OBJECTID"}, {"alias": "SITENAME", "length": 70, "type": "esriFieldTypeString", "name": "SITENAME"}], "displayFieldName": "SITENAME", "spatialReference": {"wkid": 4269, "latestWkid": 4269}, "geometryType": "esriGeometryPoint"}" Start Time: Mon Aug 05 09:51:35 2013 Running script JsonToFc... Converting JSON to FC... Completed script JsonToFc... Succeeded at Mon Aug 05 09:51:44 2013 (Elapsed Time: 9.49 seconds)

but publish to ArcServer succeeds and when trying to run it fails to create the feature class...

{  "error": {   "code": 400,   "message": "Unable to complete operation.",   "details": [    "Error executing tool."   ]  } }

And here is the content in the server logs:

Level Time Message Source Machine Code Time Elapsed User Name Process Thread Method Name
SEVERE Aug 5, 2013, 9:53:11 AM Error executing tool. ExtractDataTools/JsonToFc.GPServer <HOSTNAME>.<DOMAINNAME> 20010  anonymous 5212 7528 GPServerSync.CheckMessages
SEVERE Aug 5, 2013, 9:53:11 AM  Traceback (most recent call last): File "\\<FILESERVER>\<SHARE>$\directories\arcgissystem\arcgisinput\ExtractDataTools\JsonToFc.GPServer\extracted\v101\zipandship\JsonToFc.py", line 44, in res=arcpy.JSONToFeatures_conversion(json_file,fc) File "c:\program files\arcgis\server\arcpy\arcpy\conversion.py", line 430, in JSONToFeatures raise e ExecuteError: ERROR 000206: Cannot create FeatureClass 'C:\Users\<AGS SERVICE ACCOUNT>\AppData\Local\Temp\extractdatatools\jsontofc_gpserver\j631f8a7a49c34355a03361601e636c75\scratch\scratch.gdb\DATA_EXTRACT_130805_095310'. Failed to execute (JSONToFeatures). Failed to execute (JsonToFc). Failed to execute (JSON To Feature Class).

Any help is greatly appreciated.. I've tinkered with this for a few days with no success.  Tried changing the path of the FGDB to a custom location (not the arcpy environment workspaces) with no luck.  Made sure the service account has at least modify permissions to the output locations.  I can run these just from desktop. 

ArcGIS Desktop 10.2
ArcGIS Server 10.2
Portal for ArcGIS v 2.2 (ArcServer is federated with the portal)

Thanks!

[JamesCrandall]

Cannot create FeatureClass 'C:\Users\<AGS SERVICE ACCOUNT>\AppData\Local\Temp\extractdatatools\jsontofc_gpserver\j631f8a7a49c34355a03361601e636c75\scratch\scratch.gdb\DATA_EXTRACT_130805_095310'

Just a complete shot in the dark here and I have absolutely zero experience with this method, but...  Have you attempted to write the FC to in_memory instead of the Scratch.gdb?  Really not sure if this is going to anything at all for you, but maybe worth a try.

[PF1]

Cannot create FeatureClass 'C:\Users\<AGS SERVICE ACCOUNT>\AppData\Local\Temp\extractdatatools\jsontofc_gpserver\j631f8a7a49c34355a03361601e636c75\scratch\scratch.gdb\DATA_EXTRACT_130805_095310'

Just a complete shot in the dark here and I have absolutely zero experience with this method, but...  Have you attempted to write the FC to in_memory instead of the Scratch.gdb?  Really not sure if this is going to anything at all for you, but maybe worth a try.

Great suggestion.  I swapped the output to 'in_memory\<FC_NAME>' with no luck...

import os
import arcpy
from datetime import datetime


######################################
######  Getting params   ######
######################################
input_json=arcpy.GetParameterAsText(0)




######################################
######  Store Data   ######
######################################
dt="DATA_EXTRACT_"+datetime.today().strftime("%y%m%d_%H%M%S")
json_file=os.path.join(arcpy.env.scratchFolder,"%s.json"%dt)
#fc=os.path.join(arcpy.env.scratchGDB,dt)
fc="in_memory\%s"%dt



###CONVERT JSON TO FC###
arcpy.AddMessage("Converting JSON to FC...")
f=open(json_file,'w')
f.write(input_json)
f.close()
res=arcpy.JSONToFeatures_conversion(json_file,fc)



###Storing result...###

arcpy.SetParameter(1,arcpy.FeatureSet(fc))


###cleanup...###
#arcpy.Delete_management(fc)
#arcpy.Delete_management(json_file)

And here is the result in the server log.  Once again it succeeded from ArcGIS Desktop...

Traceback (most recent call last): File "\\<HOST>\<SHARE>$\directories\arcgissystem\arcgisinput\ExtractDataTools\JsonToFc.GPServer\extracted\v101\zipandship\JsonToFc.py", line 45, in res=arcpy.JSONToFeatures_conversion(json_file,fc) File "c:\program files\arcgis\server\arcpy\arcpy\conversion.py", line 430, in JSONToFeatures raise e ExecuteError: ERROR 000206: Cannot create FeatureClass 'in_memory\DATA_EXTRACT_130806_075502'. Failed to execute (JSONToFeatures). Failed to execute (JsonToFc). Failed to execute (JSON To Feature Class).

I have spent a little more time with some testing.  This appears that it does not work when running in 64-bit mode.  I've run the script tool from a cmd.exe process in the C:\Python27\ArcGIS10.2 directory.. Runs just fine.  I change my directory to C:\Python27\ArcGISx6410.2 and it fails with the same error... 


If I get some time I will report this 'bug' with Esri.  In the meantime... does anyone know if there is an easy way to have the process run in 32-bit mode from the ArcGIS Server service account?  I might try doing a 'subprocess' call to the 32-bit python executable as a temporary work around.  I will post back if that works.  Thanks for the help!

[PF1 · Accepted solution]

OK.  I have a very hacky work around that will suffice for now.  We've also submitted an incident with ESRI support. 

See code below:

 import os import arcpy from datetime import datetime   ###################################### ######  Getting params   ###### ###################################### input_json=arcpy.GetParameterAsText(0)    ###################################### ######  Prepare workspaces ###### ###################################### dt="DATA_EXTRACT_"+datetime.today().strftime("%y%m%d_%H%M%S") ws=arcpy.env.scratchFolder json_file=os.path.join(ws,"%s.json"%dt) res=arcpy.CreateFileGDB_management(ws,dt) fgdb=res.getOutput(0) fc=os.path.join(fgdb,dt)    ###################################### ######  Store JSON to file ###### ###################################### arcpy.AddMessage("Converting JSON to FC...") f=open(json_file,'w') f.write(input_json) f.close() #DOES NOT WORK FROM 64-bit... try:          res=arcpy.JSONToFeatures_conversion(json_file,fc) except:         arcpy.AddWarning("Could not run JSONToFeature_conversion nativly... Attempting to run in 32-bit mode...")         ##############################################         ######  Subprocess call to 32-bit ######         ##############################################                  #setup call and execute...         import subprocess         python=r'C:/Python27/ArcGIS10.2/python.exe'         cmd="""%s -c "         import sys         sys.path=[]         sys.path.append(r'C:\Windows\system32\python27.zip')         sys.path.append(r'C:\Python27\ArcGIS10.2\DLLs')         sys.path.append(r'C:\Python27\ArcGIS10.2\lib')         sys.path.append(r'C:\Python27\ArcGIS10.2\lib\plat-win')         sys.path.append(r'C:\Python27\ArcGIS10.2\lib\lib-tk')         sys.path.append(r'C:\Python27\ArcGIS10.2')         sys.path.append(r'C:\Python27\ArcGIS10.2\lib\site-packages')         sys.path.append(r'C:\Program Files (x86)\ArcGIS\Desktop10.2\\bin')         sys.path.append(r'C:\Program Files (x86)\ArcGIS\Desktop10.2\\arcpy')         sys.path.append(r'C:\Program Files (x86)\ArcGIS\Desktop10.2\ArcToolbox\Scripts')         import arcpy;         res=arcpy.JSONToFeatures_conversion(%s,%s)         print res.status         print res.getMessages()         "         """%(python,"'"+json_file.replace("\\","/")+"'","'"+fc.replace("\\","/")+"'")         arcpy.AddMessage("SUBPROCESS COMMAND: \n%s"%cmd)         out=""         try:                  p = subprocess.Popen(cmd, stdout=subprocess.PIPE)                 out, err = p.communicate()                 if (err!=None):                         arcpy.AddError("Execution failed for some reason...\n%s"%out)                         sys.exit()         except:                 arcpy.AddError("Exception thrown... Not caught... ")                 sys.exit()                  #get output and check status         sout=out.split("\r\n")         if len(sout) >1:                 res_status=sout[0]                 res_messages="\n".join(sout[1:])                 if res_status != '4':                         arcpy.AddError("JSONToFeatures_conversion failed...\n%s"%out)                         arcpy.AddError("CMD: %s"%cmd)                 else:                         arcpy.AddMessage("JSONToFeatures_converson succeeded!  \n%s"%res_messages)         else:                 sys.exit()  

This tries to run it as if it were to work and when an exception gets thrown it runs it in a subprocess using the 32-bit python installation.  The path was still mucked with 64-bit locations so I just re-built the path with the 32-bit variables...

Very hacky but this tool works from desktop (32-bit mode) and published to server (64-bit mode). 

If I remember to I will update if and when there is a resolution with ESRI support.

Result from ArcGIS Desktop:

Messages
Executing: JsonToFcSubprocess "{"features": [{"geometry": {"y": 35.748388281000075, "x": -114.85761096699997}, "attributes": {"SITENAME": "MYSITE", "OBJECTID": 1}}], "fieldAliases": {"SITENAME": "SITENAME", "OBJECTID": "OBJECTID"}, "fields": [{"alias": "OBJECTID", "type": "esriFieldTypeOID", "name": "OBJECTID"}, {"alias": "SITENAME", "length": 70, "type": "esriFieldTypeString", "name": "SITENAME"}], "displayFieldName": "SITENAME", "spatialReference": {"wkid": 4269, "latestWkid": 4269}, "geometryType": "esriGeometryPoint"}"
Start Time: Tue Aug 06 14:13:57 2013
Running script JsonToFcSubprocess...
Converting JSON to FC...
Completed script JsonToFcSubprocess...
Succeeded at Tue Aug 06 14:14:01 2013 (Elapsed Time: 4.46 seconds)

Result from ArcGIS Server REST endpoint:
http://<SERVER>.<DOMAIN>/arcgis/rest/services/ExtractDataTools/JsonToFcSubprocess/GPServer/JSON%20To%20Feature%20Class%20subprocess/execute?Input_JSON=%7B%0D%0A+%22features%22%3A+%5B%7B%0D%0A++%22geometry%22%3A+%7B%0D%0A+++%22y%22%3A+35.748388281000075%2C%0D%0A+++%22x%22%3A+-114.85761096699997%0D%0A++%7D%2C%0D%0A++%22attributes%22%3A+%7B%0D%0A+++%22SITENAME%22%3A+%22MYSITE%22%2C%0D%0A+++%22OBJECTID%22%3A+1%0D%0A++%7D%0D%0A+%7D%5D%2C%0D%0A+%22fieldAliases%22%3A+%7B%0D%0A++%22SITENAME%22%3A+%22SITENAME%22%2C%0D%0A++%22OBJECTID%22%3A+%22OBJECTID%22%0D%0A+%7D%2C%0D%0A+%22fields%22%3A+%5B%0D%0A++%7B%0D%0A+++%22alias%22%3A+%22OBJECTID%22%2C%0D%0A+++%22type%22%3A+%22esriFieldTypeOID%22%2C%0D%0A+++%22name%22%3A+%22OBJECTID%22%0D%0A++%7D%2C%0D%0A++%7B%0D%0A+++%22alias%22%3A+%22SITENAME%22%2C%0D%0A+++%22length%22%3A+70%2C%0D%0A+++%22type%22%3A+%22esriFieldTypeString%22%2C%0D%0A+++%22name%22%3A+%22SITENAME%22%0D%0A++%7D%0D%0A+%5D%2C%0D%0A+%22displayFieldName%22%3A+%22SITENAME%22%2C%0D%0A+%22spatialReference%22%3A+%7B%0D%0A++%22wkid%22%3A+4269%2C%0D%0A++%22latestWkid%22%3A+4269%0D%0A+%7D%2C%0D%0A+%22geometryType%22%3A+%22esriGeometryPoint%22%0D%0A%7D&env%3AoutSR=&env%3AprocessSR=&returnZ=false&returnM=false&f=pjson

{
"results": [
 
],
"messages": [
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Executing (JSON To Feature Class subprocess): JsonToFcSubprocess {\"features\":[{\"geometry\":{\"y\":35.748388281000075,\"x\":-114.85761096699997},\"attributes\":{\"SITENAME\":\"MYSITE\",\"OBJECTID\":1}}],\"fieldAliases\":{\"SITENAME\":\"SITENAME\",\"OBJECTID\":\"OBJECTID\"},\"fields\":[{\"alias\":\"OBJECTID\",\"type\":\"esriFieldTypeOID\",\"name\":\"OBJECTID\"},{\"alias\":\"SITENAME\",\"length\":70,\"type\":\"esriFieldTypeString\",\"name\":\"SITENAME\"}],\"displayFieldName\":\"SITENAME\",\"spatialReference\":{\"wkid\":4269,\"latestWkid\":4269},\"geometryType\":\"esriGeometryPoint\"}"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Start Time: Tue Aug 06 14:18:53 2013"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Executing (JsonToFcSubprocess): JsonToFcSubprocess {\"features\":[{\"geometry\":{\"y\":35.748388281000075,\"x\":-114.85761096699997},\"attributes\":{\"SITENAME\":\"MYSITE\",\"OBJECTID\":1}}],\"fieldAliases\":{\"SITENAME\":\"SITENAME\",\"OBJECTID\":\"OBJECTID\"},\"fields\":[{\"alias\":\"OBJECTID\",\"type\":\"esriFieldTypeOID\",\"name\":\"OBJECTID\"},{\"alias\":\"SITENAME\",\"length\":70,\"type\":\"esriFieldTypeString\",\"name\":\"SITENAME\"}],\"displayFieldName\":\"SITENAME\",\"spatialReference\":{\"wkid\":4269,\"latestWkid\":4269},\"geometryType\":\"esriGeometryPoint\"}"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Start Time: Tue Aug 06 14:18:53 2013"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Running script JsonToFcSubprocess..."
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Converting JSON to FC..."
  },
  {
   "type": "esriJobMessageTypeWarning",
   "description": "Could not run JSONToFeature_conversion nativly... Attempting to run in 32-bit mode..."
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "SUBPROCESS COMMAND: \n\\\\<SERVER>\\<SHARE>$\\directories\\arcgissystem\\arcgisinput\\ExtractDataTools\\JsonToFcSubprocess.GPServer\\extracted\\v101\\..\\cd\\arcgis10_2\\python.exe -c \"\n        import sys\n        sys.path=[]\n        sys.path.append(r'C:\\Windows\\system32\\python27.zip')\n        sys.path.append(r'C:\\Python27\\ArcGIS10.2\\DLLs')\n        sys.path.append(r'C:\\Python27\\ArcGIS10.2\\lib')\n        sys.path.append(r'C:\\Python27\\ArcGIS10.2\\lib\\plat-win')\n        sys.path.append(r'C:\\Python27\\ArcGIS10.2\\lib\\lib-tk')\n        sys.path.append(r'C:\\Python27\\ArcGIS10.2')\n        sys.path.append(r'C:\\Python27\\ArcGIS10.2\\lib\\site-packages')\n        sys.path.append(r'C:\\Program Files (x86)\\ArcGIS\\Desktop10.2\\bin')\n        sys.path.append(r'C:\\Program Files (x86)\\ArcGIS\\Desktop10.2\\arcpy')\n        sys.path.append(r'C:\\Program Files (x86)\\ArcGIS\\Desktop10.2\\ArcToolbox\\Scripts')\n        import arcpy;\n        res=arcpy.JSONToFeatures_conversion('C:/Users/<SERVICE ACCOUNT>/AppData/Local/Temp/extractdatatools/jsontofcsubprocess_gpserver/j68e8efd3ab8c4a4aaec0045ab6debe1a/scratch/DATA_EXTRACT_130806_141853.json','C:/Users/<SERVICE ACCOUNT>/AppData/Local/Temp/extractdatatools/jsontofcsubprocess_gpserver/j68e8efd3ab8c4a4aaec0045ab6debe1a/scratch/DATA_EXTRACT_130806_141853.gdb/DATA_EXTRACT_130806_141853')\n        print res.status\n        print res.getMessages()\n        \"\n        "
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Completed script JsonToFcSubprocess..."
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Succeeded at Tue Aug 06 14:18:55 2013 (Elapsed Time: 1.84 seconds)"
  },
  {
   "type": "esriJobMessageTypeInformative",
   "description": "Succeeded at Tue Aug 06 14:18:55 2013 (Elapsed Time: 1.84 seconds)"
  }
]
}

[KevinHibma]

YIKES.
... having to call subprocess in 32-bit from a 64bit GP service should not be a workaround! (though that was crafty)
Can you post the incident number so I can read any of the details there.
I'll try to find out whats going on with this and see if I can offer up a better solution.

From the notes above, nothing stands out as "incorrect".

[PF1]

YIKES.
... having to call subprocess in 32-bit from a 64bit GP service should not be a workaround! (though that was crafty)
Can you post the incident number so I can read any of the details there.
I'll try to find out whats going on with this and see if I can offer up a better solution.

From the notes above, nothing stands out as "incorrect".

Incident #1180602

[KevinHibma]

Pat,

I copy/pasted your first script, plugged it into a script tool and setup input:string, output:featureclass and was able to run your tool without any problems (as well published/ran on AGS).

I suspect it has something to do with your environment. Looking a little closer at the log messages, you posted:
SEVERE Aug 5, 2013, 9:53:11 AM  Traceback (most recent call last): File "\\<FILESERVER>\<SHARE>$\directories\arcgissystem\arcgisinput\ExtractDataTools\JsonToFc.GPServer\extracted\v101\zipandship\JsonToFc.py", line 44, in res=arcpy.JSONToFeatures_conversion(json_file,fc) File "c:\program files\arcgis\server\arcpy\arcpy\conversion.py", line 430, in JSONToFeatures raise e ExecuteError: ERROR 000206: Cannot create FeatureClass 'C:\Users\<AGS SERVICE ACCOUNT>\AppData\Local\Temp\extractdatatools\jsontofc_gpserver\j631f8a7a49c34355a03361601e636c75\scratch\scratch.gdb\DATA_EXTRACT_130805_095310'. Failed to execute (JSONToFeatures). Failed to execute (JsonToFc). Failed to execute (JSON To Feature Class).

My guess is you're in a multi-machine deployment as you're writing to a UNC path?
With GP Services, when using a UNC path, by default all processing is done local to the machine being executed on and the end result gets moved to the JOBS/OUTPUT directory as needed.
To me it sounds like Server doesn't have the ability to write to the temp directory (this temp directory would be the temp directory for the user account server is running under - as you indicate in the log messages). So I'm a little confused why Server wouldnt have the ability to write there.
My suggestion would be to log on to the machine using that account and make sure you can actually create files in that location. Maybe there is permissions issue on the folder? You of course can update this location as well by changing the temp directory for that user.
This help topic (bottom of the page) has more information: http://resources.arcgis.com/en/help/main/10.2/#/Advanced_properties/0057000000m1000000/

[PF1]

Hi Kevin,

Thanks for the feedback.  My original script was written to use a custom output location on a file share outside if arcserver.  I made sure the service account had modify permissions to that share and folder permissions.  I switched it to the scratch workspace for simplicity when posting here. 

I found out that we are still running the pre-release version on that box and have not yet deployed the final.  I will try to republish this once we get the final 10.2 version installed. And report back when the results.  Maybe it was/is just an issue with the pre release version.  It might be a few weeks before we get the general release version installed in a sandbox/test mode.   Thanks!

[DonaldBrown]

Keep in mind that when you publish a GeoProcessing Service, ESRI changes your code. Your intermediary output to a specific UNC location will get flagged and changed to the scratchFolder location. You should go look at the 'extracted' version of your code after you publish the Service to see what is really going on.