Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

[#1]

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

[AmandaOpp]

Hi there Doug,

Thanks for the information!  I never thought of doing that.  I will give it a try.  Are you using the script for Survey123 Data with attachments?  Can you set it up to automate? Thank you for the tip!

[DougBrowning]

Yep easy to script just call the toolbox.

Replaced my path with the word PATH

Yes it handles attachments just fine.

import arcpy

syncToolbox = r"PATH\SyncFeatureService.tbx"

arcpy.ImportToolbox(syncToolbox)

# sde destination - where the HFS data will be copied TO
sdeDest = r"PATH\WorkingGIS.sde"

# all FCs from the HFS will get this prefix in the SQL DB
sdeTablePrefix = "s123"

# URL to the HFS for the survey to sync
urlHFS = "http://services2.arcgis.com/PATH/arcgis/rest/services/service_STUFF/FeatureServer"

userName = "UN HERE"
passWord = "PASS HERE"

# Run the toolbox
arcpy.SyncFeatureService(sdeDest, sdeTablePrefix, urlHFS, "US/Pacific", "https://www.arcgis.com", userName, passWord)

print arcpy.GetMessages()

# Clean up memory otherwise it will not run twice in a row
arcpy.Delete_management("in_memory")

print arcpy.GetMessages()

[DougBrowning]

I did notice that there are 3,425 records but the script tries to do 4000 to 5000 for some reason.  It is a table.

[yusrulsr]

Hi, Jack Skinner

I'm new for using ArcMap 10.1, Could you guide me how to install "Download Service" (Your Attachment) in ArcMap 10.1 ? Really hope your help

Regards,

Yusrul

[rawansaleh1]

Hi All,

i produce a manual for this tool, hope this will help.

you can find it in the following link:

File sharing and storage made simple 

Best,

Rawan

[yusrulsr]

Hi Rawan Saleh,

Thanks for sharing the manual book of download service, but when i try

connect folder to download service that has been extracted, the script of

download service was not appeared on download service toolbox.

depending of that, can you help me to solve this problem??

Best Regards,

Amri

[rawansaleh1]

try to download the tool from the below link:

File sharing and storage made simple 

best,

Rawan

[yusrulsr]

Hi Rawan Saleh,

It was still can not appear the script of download service and i attach

screen shot of the problem

Please help me to fix this problem. thanks

Regards,

Amri

[deleted-user-t3dSviijg-m9]

Jake Skinner‌

Thank you for this tool. I have successfully gotten it to work with AGOL Hosted Feature layers that are shared publicly, however, whenever I try to use the tool on a Feature Service that is either shared to a group, or not shared at all, using the credentials of the owner (which is also an admin)... it keeps spitting out this error.

I've run it on 10.5 and 10.3 desktop with the same issue. 

Can I share it to a group and add your AGOL account to see if you can re-produce the error on your end? It is a hosted feature layer that was created from an existing ArcGIS Server feature service. I used the feature service as a template, but it is 100% a hosted feature layer. 

Any help would be appreciated! 

Thanks

[JakeSkinner]

naalexandrou‌ it looks like you're not adding the layer index (/0) to the URL.  For example 'http://services3.arcgis.com/AWDwYUpli8WqpWxQ/arcgis/rest/services/Public_Tree_Request/FeatureServer/0'

[deleted-user-t3dSviijg-m9]

I forgot to when I ran it quickly to grab a screen shot for my post.... I've been adding the layer index. 

[deleted-user-t3dSviijg-m9]

[JakeSkinner]

naalexandrou‌ feel free to invite my user account (jskinner_CountySandbox) to a Group and I can take a look.

[deleted-user-t3dSviijg-m9]

jskinner-esristaff‌ ... okay, I just did. Thank you for you help

[JakeSkinner]

Nick Alexandrou‌ I was not able to reproduce this.  Here is how I had the tool setup:

Make sure you have the latest version of this tool downloaded.

[deleted-user-t3dSviijg-m9]

That's so strange.... I have Version 18 downloaded. I'm wondering if there is something corrupt in my Python install. I've recently moved to 10.5 and maybe something got fudged?

[OBVYamaska]

Hi,

I'm very grateful for this tool, and it works great.

That being said, is there something I'm missing, or it can only download a Shapefile version of the feature service, and not a FGDB?

I'm getting the error (error exporting service to FGDB), but still really need to download a FGDB. I'm losing info (domain for precoded menus) while importing a Shapefile...

Does anybody now what causes the initial error on Online?

Thanks!

[JakeSkinner]

naalexandrou‌ I would try running the tool from another machine to see if you are running into the same issue.

[JakeSkinner]

OBVYamaska‌ this tool can download to a File Geodatabase, however it will not maintain the domains.  These will need to be recreated.

[OBVYamaska]

Hi Jake,

How can this be done? I don't see any option about the type of output file...

[deleted-user-t3dSviijg-m9]

jskinner-esristaff‌ I'm going to test that next. I will give an update as soon as I have one. When you ran it.... you just ran it right out of the box, no need to change anything in the python file for Token generation URL or anything right? Thanks again

[JakeSkinner]

OBVYamaska‌ for the Output Feature Class parameter just browse to a File Geodatabase and enter a name.  It will then create the new feature class.

[JakeSkinner]

naalexandrou‌ that is correct, you will not need to modify the code at all.

[deleted-user-t3dSviijg-m9]

Jake Skinner‌ so I've been able to reproduce this on all three machines I have available to me. I'm going to wipe one completely clean of ArcMap and Python, and do a full install. 

If this continues... do you think it could be something like an organizational setting in ArcGIS Online that is being applied to the 2 different usernames I am testing it with that are members of my organization? Something that prohibits token generation if accessed under certain conditions? I'll try this with a personal account and see if it works. 

[deleted-user-t3dSviijg-m9]

I created a person developer account, and added that account to the same group you are a member of. I ran the tool using the credentials of the newly created account which isn't a member of my organization, and it ran perfectly.

So then, I suppose there is a rule somewhere in our AGOL settings that is prohibiting token creation? This  is bizarre 

[deleted-user-t3dSviijg-m9]

jskinner-esristaff‌ Quick question: The username is getting stored as a string? Or something else since it is using arcpy.GetParamterAsText.... all the usernames I was trying with and failing have a "." in their names... and I was wondering if maybe that was what was throwing it off? Both your username and the Developer account username I created do not have a "." 

Just trying to figure out where the credential conflict is coming up with those usernames, because it works on the same machine with an account from outside our AGOL organization.

[JakeSkinner]

Nick Alexandrou‌ run the below script to see if you are able to generate a token successfully:

import urllib, urllib2, json

username = "agol"
password = "gis12345"

tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
req = urllib2.Request(tokenURL, urllib.urlencode(params))
response = urllib2.urlopen(req)
data = json.load(response)
token = data['token']
print(token)‍‍‍‍‍‍‍‍‍‍‍‍

[deleted-user-t3dSviijg-m9]

Jake Skinner‌ - the output

Not really sure what to make of this? I was able to successfully run your script on a secure, and public service using the developer account I created, so I don't think it would be a corrupt python install?

C:\Python27\ArcGIS10.5\python.exe S:/GIS/Scripts/untitled1/testing.py
Traceback (most recent call last):
  File "S:/GIS/Scripts/untitled1/testing.py", line 11, in <module>
    token = data['token']
KeyError: 'token'

Process finished with exit code 1
‍‍‍‍‍‍‍

[JakeSkinner]

naalexandrou‌ the script failed.  It should print the token (i.e. kjuulsWWk12jIOIUEW238923).  Something with your credentials is not working correctly.  I would recommend contacting Tech Support to see if they can figure out why your credentials are failing to generate a token.

[deleted-user-t3dSviijg-m9]

jskinner-esristaff

I'm an idiot, I didn't change the username and password. 

It works when I run it with my Developer account. But fails when I use my organizational account. 

[deleted-user-t3dSviijg-m9]

Jake Skinner‌

I've opened a case with Tech Support. They couldn't resolve the issue with the first pass, and are now going through our fiddler logs. I've created them an account within our organization for them to perform some testing. 

I'll let you know what the resolution is when we find it, in case anyone else has this issue that you run into. 

Thanks again for you help, and for the tool. 

[DougBrowning]

In case some of you can not get this to work i just got a new way to work great.  I had been using the ArcRest Create Replica way but I found a bug in tables (it will not add records to any HFS=Hosted Feature Service tables).  So I asked and the ArcRest guy turned me onto another way.

This way is equivalent to doing a Export Data from the Item page in AGO  (the one with all the settings and such).  So you may want to do the Export manually first to see if this works on your data.

First you need the Item ID of the HFS - get this from the top of the Item page in AGO.  You want the last part after id=

URL looks like http://xxx.maps.arcgis.com/home/item.html?id=46954aa9bfb24c53bba320f363af8550 

Then I slightly modified the ArcRest sample script called  create_replica_portal_item.py  (in other words some of this is not my code)

ArcREST/create_replica_portal_item.py at master · Esri/ArcREST · GitHub 

LOOK for the caps on what to change  Note the post messed up all the indents - i tried to fix but may be off

import arcrest
from arcresthelper import securityhandlerhelper
from arcresthelper import common

import os, time

def trace():

"""
trace finds the line, the filename
and error message and returns it
to the user
"""
import traceback, inspect, sys
tb = sys.exc_info()[2]
tbinfo = traceback.format_tb(tb)[0]
filename = inspect.getfile(inspect.currentframe())
# script name + line number
line = tbinfo.split(", ")[1]
# Get Python syntax error
#
synerror = traceback.format_exc().splitlines()[-1]
return line, filename, synerror

if __name__ == "__main__":

proxy_port = None
proxy_url = None

securityinfo = {}
securityinfo['security_type'] = 'Portal'#LDAP, NTLM, OAuth, Portal, PKI
securityinfo['username'] = "PUT USERNAME HERE"    #<UserName>
securityinfo['password'] = "PUT PASSWORD HERE"    #<Password>
securityinfo['org_url'] = "http://www.arcgis.com"
securityinfo['proxy_url'] = proxy_url
securityinfo['proxy_port'] = proxy_port
securityinfo['referer_url'] = None
securityinfo['token_url'] = None
securityinfo['certificatefile'] = None
securityinfo['keyfile'] = None
securityinfo['client_id'] = None
securityinfo['secret_id'] = None

# where to save the zip files
savePath = r"PUT OUTPUT DIR HERE"

itemId = "PUT ITEM ID HERE"
#sample   itemId = "5a9ecdf30f53466bad14f5dbbedb1014"  

try:

shh = securityhandlerhelper.securityhandlerhelper(securityinfo=securityinfo)
if shh.valid == False:
print shh.message
else:
admin = arcrest.manageorg.Administration(securityHandler=shh.securityhandler)

item = admin.content.getItem(itemId)
user = admin.content.users.user(username=item.owner)

# use to set jsut some layers. NOT using this
## exportParameters = {"layers":[
## {
## "id": 0,
## "where": "OBJECTID = 1"
## }]
## }

# this version waits for the download so no pause needed
res = user.exportItem(title=HFSname,
itemId=itemId,
exportFormat="File Geodatabase",
exportParameters=None,
wait=True)

exportItemId = res.id
exportItem = admin.content.getItem(exportItemId)
itemDataPath = exportItem.itemData(f=None, savePath=savePath)
 # Delete from AGO list to remove clutter
exportItem.userItem.deleteItem()

# rename file so it makes sense
#taken out for ease

except (common.ArcRestHelperError),e:

print "error in function: %s" % e[0]['function']
print "error on line: %s" % e[0]['line']
print "error in file name: %s" % e[0]['filename']
print "with error message: %s" % e[0]['synerror']
if 'arcpyError' in e[0]:
print "with arcpy message: %s" % e[0]['arcpyError']

except:
line, filename, synerror = trace()
print "error on line: %s" % line
print "error in file name: %s" % filename
print "with error message: %s" % synerror

Let me know how it goes.

[PatrickMcKinney1]

Jake Skinner‌, Thanks again for developing this tool.  I am working on using your code for a stand-alone script just to extract features from services.  I have tested successfully on unsecured ArcGIS Server (10.3.1) services.  However, when I tested with a secured service, I am getting an error.  It looks like the request is successful and a token is generated, but the error message I'm getting is 'objectIds'.  Maybe the  data['objectIds'].sort() key should be changed to match what is in my service?  I also had to update the token generation URL from the original script.  Below are some messages I logged while attempting to make the request:

Generating token for service

Parameters: {'username': 'SomeUser', 'password': 'CantShare', 'f': 'pjson', 'client': 'requestip'}

Request: <urllib2.Request instance at 0x122ED710>

Data: {u'token': u'FnGoWx6kQOBa_P-X9E6lve_p0MfoOWI4uHrMeihIASnBIf-Np7ZXar8aw57dzUac', u'expires': 1497456260540L}

Token Type is : <type 'unicode'>

Token: FnGoWx6kQOBa_P-X9E6lve_p0MfoOWI4uHrMeihIASnBIf-Np7ZXar8aw57dzUac

New Parameters: {'token': u'FnGoWx6kQOBa_P-X9E6lve_p0MfoOWI4uHrMeihIASnBIf-Np7ZXar8aw57dzUac', 'returnIdsOnly': 'true', 'where': '1=1', 'f': 'json'}

New Data: {u'drawingInfo': {u'labelingInfo': None, u'renderer': {u'symbol': {u'contentType': u'image/png', u'imageData': u'iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IB2cksfwAAAAlwSFlzAAAOxAAADsQBlSsOGwAAAFtJREFUOI1jYaAQsDAwMDDU+jNcMNdk+ESKxpPXGfiaNzIYsDAwMDC4GDAI2mkx6JNiAC87w6PmjVAXUAJGDRg1YBAZcPUhAw8nO2karz5k4IEbkLWAQZhcFwAAU4UOOJ0k87cAAAAASUVORK5CYII=', u'url': u'f3b6e42d90b9ace7811063e0f4b7f028', u'height': 12, u'width': 12, u'angle': 0, u'yoffset': 0, u'type': u'esriPMS', u'xoffset': 0}, u'type': u'simple', u'description': u'', u'label': u''}, u'transparency': 0}, u'extent': {u'xmin': 2010428.1978965253, u'ymin': 249956.09646843374, u'ymax': 358567.8215017766, u'xmax': 2220380.042427689, u'spatialReference': {u'wkid': 102729, u'latestWkid': 2272}}, u'canModifyLayer': False, u'advancedQueryCapabilities': {u'supportsQueryWithDistance': True, u'supportsDistinct': True, u'supportsReturningQueryExtent': True, u'supportsStatistics': True, u'supportsPagination': True, u'supportsOrderBy': True, u'supportsTrueCurve': True, u'useStandardizedQueries': True}, u'hasLabels': False, u'subLayers': [], u'supportsAdvancedQueries': True, u'id': 0, u'relationships': [], u'parentLayer': None, u'capabilities': u'Map,Query,Data', u'currentVersion': 10.31, u'geometryType': u'esriGeometryPoint', u'ownershipBasedAccessControlForFeatures': {u'allowOthersToQuery': True}, u'type': u'Feature Layer', u'useStandardizedQueries': True, u'supportedQueryFormats': u'JSON, AMF', u'maxRecordCount': 1000, u'description': u'', u'defaultVisibility': True, u'typeIdField': u'FType', u'minScale': 750000, u'types': [{u'domains': {u'FCode': {u'type': u'inherited'}}, u'id': 730, u'name': u'Education'}], u'displayField': u'Name', u'isDataVersioned': True, u'name': u'Schools', u'supportsStatistics': True, u'hasAttachments': False, u'fields': [{u'alias': u'OBJECTID', u'domain': None, u'type': u'esriFieldTypeOID', u'name': u'OBJECTID'}, {u'alias': u'Name', u'length': 255, u'type': u'esriFieldTypeString', u'name': u'Name', u'domain': None}, {u'alias': u'Owner', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'Owner', u'domain': None}, {u'alias': u'Address1', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'Address1', u'domain': None}, {u'alias': u'City', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'City', u'domain': None}, {u'alias': u'State', u'length': 2, u'type': u'esriFieldTypeString', u'name': u'State', u'domain': None}, {u'alias': u'Zip', u'length': 10, u'type': u'esriFieldTypeString', u'name': u'Zip', u'domain': None}, {u'alias': u'PocName', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'PocName', u'domain': None}, {u'alias': u'PocOrganization', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'PocOrganization', u'domain': None}, {u'alias': u'PocTel1', u'length': 25, u'type': u'esriFieldTypeString', u'name': u'PocTel1', u'domain': None}, {u'alias': u'PocEmail', u'length': 50, u'type': u'esriFieldTypeString', u'name': u'PocEmail', u'domain': None}, {u'alias': u'FType', u'domain': None, u'type': u'esriFieldTypeInteger', u'name': u'FType'}, {u'alias': u'FCode', u'domain': {u'codedValues': [{u'code': 73000, u'name': u'Education Facility'}, {u'code': 73002, u'name': u'School'}, {u'code': 73004, u'name': u'School: Middle School'}, {u'code': 73003, u'name': u'School: Elementary'}, {u'code': 73005, u'name': u'School: High School'}, {u'code': 73006, u'name': u'College / University'}, {u'code': 73007, u'name': u'School: Amish'}, {u'code': 73008, u'name': u'School: Mennonite'}], u'type': u'codedValue', u'name': u'Education FCode'}, u'type': u'esriFieldTypeInteger', u'name': u'FCode'}, {u'alias': u'SHAPE', u'domain': None, u'type': u'esriFieldTypeGeometry', u'name': u'SHAPE'}], u'maxScale': 0, u'copyrightText': u'', u'canScaleSymbols': False, u'htmlPopupType': u'esriServerHTMLPopupTypeAsHTMLText'}

Error: 'objectIds'

URL is incorrect. Or, Service is secure, please enter username and password.

[PatrickMcKinney1]

jskinner-esristaff‌, Please disregard, I pulled my guns too soon.  I left off the '/query' at the end of my baseURL.

[DiklaZeidman]

hi,

Thank you for this tool!

is there a way that I can also get the Domains that are in the layer I am downloading?

my layer is a feature layer from arcgis server, and it contains fields with domains.

[JakeSkinner]

SPNI‌ currently this functionality does not exist.

[DougBrowning]

My example on Jun 8 does copy over domains so give that a shot.

[EricAdolfson]

Would you be willing to post an example of the stand alone script that you completed based on Jake's code? I am trying to do the same thing. Thanks.

[PatrickMcKinney1]

EA1‌, I am planning on posting the code to GitHub, after making a few more tweaks. I'll post the link to that repo on this thread.  It may be a couple weeks.  

[PatrickMcKinney1]

Eric Adolfson‌, I was able to put the script up on Github.  This is an active project at work, so the script may change a bit.  My next step is to test with services that are down, and add in logic to e-mail that the service is down.

I'm using this script for the update process for a multi-county regional dataset.

[RELAdmin]

Hello,

I'm wondering if there is a simple way I can modify the code to only download features that fit a certain criteria/query? I want to limit the features I download to only features that have a certain attribute. I feel like there would need to be another line of code to filter the data before the download begins, but I'm unsure how to go about this. If anybody could help that would be awesome!

Thanks.

[JakeSkinner]

Philip Ohlinger‌ find the following line of code:

params = {'where': '1=1', 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}

You will want to update the 'where' statement.  Ex:

params = {'where': 'ID > 50', 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}

[RELAdmin]

How do you format a text value inside the where statement? I'm trying to filter all features where COUNTY = "ADAMS"

Seems my formatting is wrong, I am getting an error now that the URL is incorrect.

[RELAdmin]

Oh think I got it! Pesky quotes and spaces. Thank you so much for the help!

[NickHarvey]

Hi All - We're able to export features from a secure AGS FS in ArcMap (10.3.1) and ArcGIS Pro (2.0.0) - But only from the map (Contents, Layer right-click, Data, Export Features),..We haven't been able to get either version of the tool to download from the secure service....Either version works as soon as we turn off the AGS security.  The credentials are working in both desktop apps, no errors so far (can provide them if you have time to test)......Before I dig further, any ideas?...   

Thanks

-Nick

[PatrickMcKinney1]

I ran into a similar problem.  Line 57 of the tool's script defines the token generation URL the following way: tokenURL = 'http://' + server + '/arcgis/admin/generateToken'. 

Our ArcGIS Server was configured differently, so our token generation URL is arcgiswebadaptor/tokens/GenerateToken.

Not sure if this is why you're having the issue, but worth checking into.

[NickHarvey]

Bulls Eye Patrick.  Thank you so much.....I referred to this help article after getting your tip :  Acquiring ArcGIS tokens—Documentation (10.3 and 10.3.1) | ArcGIS for Server.

I've edited our copies and have them working now...I am getting a taller ObjectID count in my script output messaging than I expected? (looking into it).  But the attributes & geometry appear to be coming over faithfully. 

Thanks again!

-Nick

[ISP_graynic]

My hosted feature class has and OJBECTID_1 field instead of just OBJECTID.  What do I need to change in the script to have it have it recognize OBJECT_1 as the OBJECTID field? 

Thanks!

[JakeSkinner]

graynic‌ the script should pick up which field is the OBJECTID by looking at a parameter 'objectIdFieldName'.  If you are having trouble feel free sharing the service with an ArcGIS Online Group and invite my account (jskinner_CountySandbox).  I can take a look to see what's going on.

[ISP_graynic]

I think there might be both an OBJECTID and OBJECTID_1.  It downloads the first couple hundred of records that had an OBJECTID originally, but after the service was updated and the OBJECTID_1 was added, several more records were added which aren't being downloaded.  jskinner-esristaff I sent you an invite to a group to take a look at the service.  Thank you!