topic Re: How to edit a feature class used in a Feature Service via Python? in Python Questions

How to edit a feature class used in a Feature Service via Python?

NickJacob — Fri, 23 Aug 2013 13:29:50 GMT

Hello,

I'm experimenting around a bit with ArcGIS for Server and I have a feature class that I'd like to edit with a python script. The catch is that this feature class is the basis for a Feature Service I have published. Does anyone know what the best approach is for editing these features via arcpy? I initially thought I could simpy overwrite the feature class, but I fear server will place a lock on it. Is it possible to delete or add new features through a POST operation, or do I have the wrong idea?

I essentially have a feature class of polygons used in a feature service and I want to execute a python script every couple of minutes with Windows Task Scheduler that updates it.

Any advice is definitely welcome!

- Nick

Re: How to edit a feature class used in a Feature Service via Python?

JamesGustine — Fri, 23 Aug 2013 18:16:19 GMT

Do you need to perform a process of some kind on the data?

If your doing a wholesale update, delete/append will work around locks

arcpy.management.DeleteRows()
arcpy.management.Append()

Or perhaps arcpy.da.InsertCursor just update a few rows?

You can also open an edit session using arcpy and perform many a gp tasks.

We would need to know more about what your trying to accomplish.

Re: How to edit a feature class used in a Feature Service via Python?

PF1 — Tue, 27 Aug 2013 20:56:51 GMT

You should be able to 'truncate' (delete the rows) and append even if ArcGIS Server has it published as jgustine mentioned.

I have created a few python tools that will update a feature service using the POST method by interacting with the ArcGIS Server REST API. We used the The python script uses the httplib and creates either an HTTPSConnection or HTTPConnection object (depending if it is HTTP or HTTPS access). My code essentially does something like this:

Inspect the URL. Determine if HTTP or HTTPS to build the correct connection object

Inspect the security configuration. If token based (isTokenBasedSecurity = True)obtain a token by prompting for username/password. If IWA then attempt the following and assume the user running the tool has access

Convert the records from the Feature Class to a JSON object using the arcpy.FeatureSet tool and accessing the JSON property.

Obtain all of the primary keys in the feature set object (iterating through the JSON object).

Query the feature service to determine if these are adds or updates

if its not in the feature service create a feature object for the 'adds' parameter for each record not found in the service.

If it is in the feature service create a feature object for the 'updates' parameter for each record found in the service.

Execute an HTTP post method to the applyEdits operation with the adds and updates specified as parameters

Inspect the result, store it as an IN_MEMORY table and return it to the user

It actually performs very fast. Here were a few things/problems that I ran into while developing these tools:

The arcpy.FeatureSet.JSON property would return strings in unicode. The HTTP Post did not like that. Had to convert all string types to str before creating the Feature Object

Had some issues with arcpy.FeatureSet missing/truncating the geometries as posted here

If token based authentication then we needed to prompt the user for credentials. We used the getpass library in a sub-process as discussed here

There is a max size limit imposed on the POST method. I think it is something like 2MB. Cannot seem to find information about it off the top of my head but you might run in this if there is a lot of changes/updates. Might need to split the job into small chunks. Process like 10% at a time and run the POST 10 times...

So a basic workflow would look like this:

Usesr use ArcMap to extract some data from a Feature Service to a local Feature Class

Update records desired

Insert new records as desired

Run a python script tool to sync the features to the feature service using the applyEdits operation (POST)

Hope this helps. I might be able to show you a few code samples if you really wanted them but I've got all the logic rolled up into a robust set of classes to interact with ArcGIS Server.

Re: How to edit a feature class used in a Feature Service via Python?

NickJacob — Mon, 02 Sep 2013 00:04:49 GMT

Thank you both for your replies! I learned quite a bit from both responses. So far I've used a combination of arcpy's Truncate Table and Append tools with some good results.

Re: How to edit a feature class used in a Feature Service via Python?

Anonymous User — Mon, 04 Nov 2013 21:44:54 GMT

I am also interested in adding data to AGOL through Python. I have scripts to download data but require assistance in the upload of new data. I am having problems executing an HTTP post method to the applyEdits with ADDS or addFeatures operation though REQUEST or URLIB2. Is it possible for pfoppe to post some of his code to assist, it would be well appreciated.

Cheers rossco

Re: How to edit a feature class used in a Feature Service via Python?

PF1 — Sat, 11 Dec 2021 11:50:27 GMT

Hi Rossco,

I've attached a snippet of the UML class diagram of my implementation for reference, and a little snippet of code that might help you (although you will not be able to run this since you do not have all the underlying classes...). Hopefully just seeing the parameters on the request will help.

Most of the times that I run into a problem it is because there is a unicode string somewhere in the json structure... and I have not yet figured out a way to get a POST to work when there is a unicode in there, thats why I built a few of my own objects (AgsFeature, AgsFeatures, AgsGeometry, ...) to remove those unicode characters...

This runs against a sample server that Esri has available for editing capabilities... I created a new record with a 'rotation' equal to 42 (since 42 is the answer to life)


from datetime import datetime
from ArcServer.ArcGisServer import AgsLayer
from ArcServer.ArcGisServer import AgsFeatureServiceLayer
from ArcServer.ArcGisServer import AgsFeature
from ArcServer.ArcGisServer import AgsSpatialReference



#Build a layer object to query
print "\n\n****QUERYING SERVICE***"
ags_fs_url=r'http://sampleserver6.arcgisonline.com/arcgis/rest/services/Wildfire/FeatureServer/0'
lyrObj=AgsLayer.fromUrl(ags_fs_url)

#Find the record for NFL Broncos...
where="rotation=42"
result=lyrObj.query(where,"*",geometry=True)
print "Query Result: %s"%str(result)
print "Response JSON: \n%s"%str(lyrObj.json())

#Try to add the date to the description
print "\n\n****TRYING TO ADD THE DATE WITH UNICODE CHARACTERS***"
jdata=lyrObj.json()
feature=jdata["features"][0]
dt=datetime.strftime(datetime.now(),"%Y%m%d_%H:%M:%S")
old_desc=feature['attributes']['description']
new_desc="Go Broncos! @ %s"%dt
print "Setting description from '%s' to '%s'"%(old_desc,new_desc)
feature['attributes']['description']=new_desc

fslyrObj=AgsFeatureServiceLayer.fromUrl(ags_fs_url)
fslyrObj.params=dict()
fslyrObj.params['updates']=[feature]
fslyrObj.params['f']='json'
pth="/applyEdits"


print "Executing 'applyEdits' via POST..."
print "Host: %s"%str(fslyrObj.host)
print "Path: %s"%str(fslyrObj.path+pth)
print "Method: %s"%str(fslyrObj.method)
print "port: %s"%str(fslyrObj.port)
print "https: %s"%str(fslyrObj.https)
print "headers: %s"%str(fslyrObj.headers)
print "params: %s"%str(fslyrObj.params)

result=fslyrObj.execute(pth)
print "Query Result: %s"%str(result)
print "Query Messages: %s"%str(fslyrObj.messages())
print "Query Data: %s"%str(fslyrObj.data())


#Lets try without unicode...
print "\n\n****TRYING TO ADD THE DATE WITHOUT UNICODE CHARACTERS***"
srObj=AgsSpatialReference.fromFeatureSetSr(jdata['spatialReference'])
featureObj=AgsFeature.fromFsFeature(feature,jdata['spatialReference'])


del fslyrObj
fslyrObj=AgsFeatureServiceLayer.fromUrl(ags_fs_url)
fslyrObj.params=dict()
fslyrObj.params['updates']=featureObj.json()
fslyrObj.params['f']='json'
pth="/applyEdits"

print "Executing 'applyEdits' via POST..."
print "Host: %s"%str(fslyrObj.host)
print "Path: %s"%str(fslyrObj.path+pth)
print "Method: %s"%str(fslyrObj.method)
print "port: %s"%str(fslyrObj.port)
print "https: %s"%str(fslyrObj.https)
print "headers: %s"%str(fslyrObj.headers)
print "params: %s"%str(fslyrObj.params)

result=fslyrObj.execute(pth)
print "Query Result: %s"%str(result)
print "Query Messages: %s"%str(fslyrObj.messages())
print "Query Data: %s"%str(fslyrObj.data())

And... Output:

****QUERYING SERVICE***
Query Result: True
Response JSON:
{u'features': [{u'geometry': {u'y': 4828807.519031979, u'x': -11690778.563894253}, u'attributes': {u'description': u'Go Broncos! @ 20131105_08:52:52', u'objectid': 1609, u'eventtype': 12, u'created_user': u'', u'created_date': 1383664073000L, u'rotation': 42, u'last_edited_date': 1383666772000L, u'eventdate': 0, u'last_edited_user': u''}}], u'fields': [{u'alias': u'OBJECTID', u'type': u'esriFieldTypeOID', u'name': u'objectid'}, {u'alias': u'Rotation', u'type': u'esriFieldTypeSmallInteger', u'name': u'rotation'}, {u'alias': u'Description', u'length': 75, u'type': u'esriFieldTypeString', u'name': u'description'}, {u'alias': u'Date', u'length': 36, u'type': u'esriFieldTypeDate', u'name': u'eventdate'}, {u'alias': u'Type', u'type': u'esriFieldTypeInteger', u'name': u'eventtype'}, {u'alias': u'created_user', u'length': 255, u'type': u'esriFieldTypeString', u'name': u'created_user'}, {u'alias': u'created_date', u'length': 36, u'type': u'esriFieldTypeDate', u'name': u'created_date'}, {u'alias': u'last_edited_user', u'length': 255, u'type': u'esriFieldTypeString', u'name': u'last_edited_user'}, {u'alias': u'last_edited_date', u'length': 36, u'type': u'esriFieldTypeDate', u'name': u'last_edited_date'}], u'spatialReference': {u'wkid': 102100, u'latestWkid': 3857}, u'geometryType': u'esriGeometryPoint', u'objectIdFieldName': u'objectid', u'globalIdFieldName': u''}

****TRYING TO ADD THE DATE WITH UNICODE CHARACTERS***
Setting description from 'Go Broncos! @ 20131105_08:52:52' to 'Go Broncos! @ 20131105_09:01:36'
Executing 'applyEdits' via POST...
Host: sampleserver6.arcgisonline.com
Path: /arcgis/rest/services/Wildfire/FeatureServer/0/applyEdits
Method: POST
port: 80
https: False
headers: {'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'text/plain'}
params: {'updates': [{u'geometry': {u'y': 4828807.519031979, u'x': -11690778.563894253}, u'attributes': {u'description': 'Go Broncos! @ 20131105_09:01:36', u'objectid': 1609, u'eventtype': 12, u'created_user': u'', u'created_date': 1383664073000L, u'rotation': 42, u'last_edited_date': 1383666772000L, u'eventdate': 0, u'last_edited_user': u''}}], 'f': 'json'}
Query Result: False
Query Messages: ['Response indicated there was an error, please check the obj.data()...']
Query Data: {"error":{"code":400,"message":"Unable to complete operation.","details":[]}}

****TRYING TO ADD THE DATE WITHOUT UNICODE CHARACTERS***
Executing 'applyEdits' via POST...
Host: sampleserver6.arcgisonline.com
Path: /arcgis/rest/services/Wildfire/FeatureServer/0/applyEdits
Method: POST
port: 80
https: False
headers: {'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'text/plain'}
params: {'updates': [{'geometry': {'y': '4828807.51903', 'x': '-11690778.5639', 'spatialReference': {'wkid': 102100, 'latestWkid': 3857}}, 'attributes': {'DESCRIPTION': 'Go Broncos! @ 20131105_09:01:36', 'OBJECTID': 1609, 'EVENTTYPE': 12, 'CREATED_USER': '', 'CREATED_DATE': 1383664073000L, 'ROTATION': 42, 'LAST_EDITED_DATE': 1383666772000L, 'EVENTDATE': 0, 'LAST_EDITED_USER': ''}}], 'f': 'json'}
Query Result: True
Query Messages: []
Query Data: {"addResults":[],"updateResults":[{"objectId":1609,"success":true}],"deleteResults":[]}

If you post your parameters on your HTTP Post (and the description of the service) it might help... Best of luck!

Re: How to edit a feature class used in a Feature Service via Python?

PF1 — Sat, 11 Dec 2021 11:50:30 GMT

Ok... Ok... I created a better sample that you should be able to run (without all my custom classes, error checking, validation, etc). This runs on my environment with just the base python installation for ArcGIS 10, 10.1 or 10.2.

import httplib
import urllib
import json
from urlparse import urlparse
from datetime import datetime



def parseUrl(url):
 result=dict()
 up=urlparse(url)
 host=up.netloc
 #assume port 80 at first
 port=80
 #result["host"]=host
 result["path"]=up.path
 result["https"]=False
 if (up.scheme.upper()=='HTTPS'):
  result["https"]=True
  #switch to port 443 if HTTPS by default
  port=443
 #accept non-default port if provided...
 if (up.netloc.find(":") != -1):
  port=int(host[host.find(":")+1:])
  host=host[0:host.find(":")]
 result["host"]=host
 result["port"]=port
 return result


ags_fs_url=r'http://sampleserver6.arcgisonline.com/arcgis/rest/services/Wildfire/FeatureServer/0'
urlp=parseUrl(ags_fs_url)
headers={"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}


print "\n\n****QUERYING SERVICE***"

conn=httplib.HTTPConnection(urlp['host'],urlp['port'])
params=dict()
params['f']='json'
params['where']="rotation=42"
params['outFields']="objectid,description,rotation"
params['returnGeometry']=False
conn.request("POST",urlp['path']+"/query",urllib.urlencode(params),headers)
resp=conn.getresponse()
data=resp.read()
jdata=json.loads(data)
print "Response JSON: \n%s"%str(jdata)


#Try to add the date to the description
print "\n\n****TRYING TO ADD THE DATE WITH UNICODE CHARACTERS***"
feature=jdata["features"][0]
dt=datetime.strftime(datetime.now(),"%Y%m%d_%H:%M:%S")
old_desc=feature['attributes']['description']
new_desc="Go Broncos! @ %s"%dt
print "Setting description from '%s' to '%s'"%(old_desc,new_desc)
feature['attributes']['description']=new_desc

conn=httplib.HTTPConnection(urlp['host'],urlp['port'])
params=dict()
params['f']='json'
params['updates']=[feature]
conn.request("POST",urlp['path']+"/applyEdits",urllib.urlencode(params),headers)
resp=conn.getresponse()
data=resp.read()
jdata=json.loads(data)
print "Request Parameters: \n%s"%str(params)
print "Response JSON: \n%s"%str(jdata)

#Lets try without unicode...
print "\n\n****TRYING TO ADD THE DATE WITHOUT UNICODE CHARACTERS***"
new_feature=dict()
for k in feature.keys():
 new_feature[str(k)]=dict()

for k in feature['attributes'].keys():
 if type(feature['attributes']) == unicode:
  new_feature['attributes'][str(k)]=str(feature['attributes'])
 else:
  new_feature['attributes'][str(k)]=feature['attributes']

conn=httplib.HTTPConnection(urlp['host'],urlp['port'])
params=dict()
params['f']='json'
params['updates']=[new_feature]
conn.request("POST",urlp['path']+"/applyEdits",urllib.urlencode(params),headers)
resp=conn.getresponse()
data=resp.read()
jdata=json.loads(data)
print "Request Parameters: \n%s"%str(params)
print "Response JSON: \n%s"%str(jdata)

And the output:

****QUERYING SERVICE***
Response JSON:
{u'fields': [{u'alias': u'OBJECTID', u'type': u'esriFieldTypeOID', u'name': u'objectid'}, {u'alias': u'Rotation', u'type': u'esriFieldTypeSmallInteger', u'name': u'rotation'}, {u'alias': u'Description', u'length': 75, u'type': u'esriFieldTypeString', u'name': u'description'}], u'globalIdFieldName': u'', u'objectIdFieldName': u'objectid', u'features': [{u'attributes': {u'rotation': 42, u'description': u'Go Broncos! @ 20131105_12:33:55', u'objectid': 1609}}]}

****TRYING TO ADD THE DATE WITH UNICODE CHARACTERS***
Setting description from 'Go Broncos! @ 20131105_12:33:55' to 'Go Broncos! @ 20131105_12:38:14'
Request Parameters:
{'updates': [{u'attributes': {u'rotation': 42, u'description': 'Go Broncos! @ 20131105_12:38:14', u'objectid': 1609}}], 'f': 'json'}
Response JSON:
{u'error': {u'message': u'Unable to complete operation.', u'code': 400, u'details': []}}

****TRYING TO ADD THE DATE WITHOUT UNICODE CHARACTERS***
Request Parameters:
{'updates': [{'attributes': {'rotation': 42, 'description': 'Go Broncos! @ 20131105_12:38:14', 'objectid': 1609}}], 'f': 'json'}
Response JSON:
{u'addResults': [], u'deleteResults': [], u'updateResults': [{u'success': True, u'objectId': 1609}]}

BTW... GO Broncos!! 🙂