I have been working on a python script to query a feature service hosted on ArcGIS Online. It may give you an idea of what is involved. It uses the REST API. Basically the steps are: 1. Log in to get a token; 2. Format a query to select recent updates; 3. Loop through the returned JSON data and copy to your database. Here is a code sample that will help take you through the process of getting the data.
import arcpy, urllib, urllib2, json, sys, time, datetime, collections
from datetime import datetime, timedelta
# how far back do you want to go? Remember AGO uses GMT / UTC.
H = 220
date_H_hours_ago = datetime.now() - timedelta(hours=H)
# Credentials and feature service information
# URL, referrer and tokenurl may vary based on ArcGIS Online or your server setup
username = '<username>'
password = '<password>'
URL = "https://<server.arcgis.com/AbxQRxx3etc>/arcgis/rest/services/<feature service>/FeatureServer/<layer>/query"
referer = "http://www.arcgis.com/"
tokenurl = "https://www.arcgis.com/sharing/rest/generateToken"
# obtain a token
query_dict = { 'username': username, 'password': password, 'referer': referer }
query_string = urllib.urlencode(query_dict)
token = json.loads(urllib.urlopen(tokenurl + "?f=json", query_string).read())
if "token" not in token:
print(token['error'])
sys.exit(1)
query_dict = {
"where" : "EditDate >= DATE '"+date_H_hours_ago.strftime("%Y-%m-%d %H:%M:%S")+"'",
"outFields" : "OBJECTID, EditDate",
"returnGeometry" : "true",
"f": "json", "token": token['token'] }
# to select all fields use: "outFields" : "*",
# to select individual fields use comma delimited list: "outFields" : "OBJECTID, EditDate",
# a date certain: "where" : "EditDate >= DATE '2016-05-29 18:30:00'",
# this morning at 2 am: "where" : "EditDate >= DATE '"+(datetime.now()).strftime("%Y-%m-%d 02:00:00")+"'",
# some time ago: "where" : "EditDate > DATE '"+date_H_hours_ago.strftime("%Y-%m-%d %H:%M:%S")+"'",
# if you do not want geometry: "returnGeometry" : "false",
# results in json format using POST method
jsonResponse = urllib.urlopen(URL, urllib.urlencode(query_dict))
features = json.loads(jsonResponse.read(),
object_pairs_hook=collections.OrderedDict)[u'features']
# print json.dumps(features, indent=4, sort_keys=False) # formatted json
print 'ObjectID\tEditDate'
for feature in features:
# AGO uses GMT/UTC, you may wish to convert to local time
editTime = time.strftime('%c', time.localtime(feature['attributes']['EditDate']/1000))
print str(feature['attributes']['OBJECTID']) + '\t' + editTime
# if you wanted geometry; AGO returns web mercator, reproject if necessary
print "x: " + str(feature['geometry']['x'])+ "\ty: " + str(feature['geometry']['y'])
# WGS 1984 : (4326) Lat/Lon
# WGS 1984 Web Mercator (auxiliary sphere) : (102100) or (3857)
ptGeometry = arcpy.PointGeometry(arcpy.Point(feature['geometry']['x'],feature['geometry']['y']),
arcpy.SpatialReference(3857)).projectAs(arcpy.SpatialReference(4326))
print "lon: " + str(ptGeometry.firstPoint.X) +"\tlat: " + str(ptGeometry.firstPoint.Y)
# add code to insert into another geodatabaseYou will probably need to adjust the script to distinguish between new entries and updated entries by using the CreateDate and EditDate fields (edit tracking) when inserting the data in your db.
To set up the script to run at specified times, see the blog: Scheduling a Python script or model to run at a prescribed time | ArcGIS Blog. You may also find useful code at GitHub - Esri/ArcREST, and consider asking your question in the REST API or Python sections.
If Collector is used in off-line mode, the Create/EditDate field is set when the feature is collected and not when it is synced. Features collected off-line and not synced before the script's scheduled time could be missed if only edit/create times are used.
