Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:


Thanks Jake for making this. It is a big help.

I have it working with what I need but I have encountered a snag in that I need to encrypt the username and password. I have tried storing the username and password in a separate file and just referencing the parameters in my code (which uses your code). The referencing works but it does not hide the actual value of the parameters. I have tried using base64 encoding but now arcgisonline does not recognize my credentials. Do you know how I can encrypt the password so that the actual value cannot be seen but arcgisonline can still recognize it?

graynic‌ I was unable to reproduce this.  Below is how I had the tool setup:

Are you outputting the data to a geodatabase?

jskinner-esristaff  Yes, I would like to output it to a geodatabse.  The end goal is to have it exported as a spreadsheet for Excel on a regular basis.  

graynic‌ can you provide a screen shot of how you have the tool setup?

Jake Skinner‌ Here is how I set it up.  Were you able to get it to run?  If so, how many records showed up in your layer after it was exported?

graynic‌ yes, it ran and returned 953 records.  Are you receiving an error when you run the tool, or is it just not returning the correct amount of records?

jskinner-esristaff  Now I am getting the error below


@Nicholas_Gray check to make sure you don't have 'Downloading tabular data' checked.  This error usually indicates you are trying to a load a table into a feature class.

jskinner-esristaff‌ I don't have that checked but am still getting that message.  I'm running it with the same settings you sent me in the screenshot.  

Thanks for helping!

Tiffany Selvidge‌ can you provide an example of using the base64 encoding.  I'm not an expert with this, but I just tried generating an AGOL token using this and it worked.

import base64, urllib, urllib2, json

username = "jskinner_CountySandbox"
password = 'agol1234'.encode('base64')

tokenURL = ''
params = {'f': 'pjson', 'username': username, 'password': password.decode('base64'), 'referer': '', 'expiration': str(21600)}
req = urllib2.Request(tokenURL, urllib.urlencode(params))
response = urllib2.urlopen(req)
data = json.load(response)
token = data['token']


Thanks for your response.

I was not including 'password.decode('base64')'. I am now able to download the service. I am wondering though that since decode will cause the password to be read, is the base64 encoding any use? Or is it just me when I run the code and print the line that is able to see the plain text? I have tried running Fiddler to see what others might see but the code will not run while Fiddler is open.

I think I answered my own question regarding reading the password. All anyone has to do is grab the code and run the line to acquire the password. That leads to my next question which is whether any encoding encrypting method can securely work since the password has to be sent to arcgisonline and agol has to be able to read the password to login. The only solution I can think of is that agol would have to be the one decrypting the password otherwise I am be encrypting and decrypting in the same exact code. Since I do not think agol decrypting is at all an option, for now I will stick with the base64 coding as at least the plain text cannot be read without having to run the line.

Thanks again for writing the script, Jake. It is exactly what was needed. I am downloading 14 layers and running them through another script and updating them into Portal on a daily basis. Manually downloading each day and then running the script would not have been pleasant. Being able to automate the entire task is much much better.

jskinner-esristaff‌ I was able to get the tool to run in Pro instead of ArcMap.  Thank you for all your help!


When I run the tool in ArcMap it works fine. I used copy as python snippet to bring it into a new python window for creating a script file to run it from. When running from a python script, I get an error (Output name cannot contain dots). I did some digging around in and found that if I removed the ".1" from "esriDataSourcesGDB.SdeWorkspaceFactory.1", it works fine.

I made this change because this code: ##desc = arcpy.Describe(os.path.dirname(wrkspc))
##print(desc.workspaceFactoryProgID) returns "esriDataSourcesGDB.SdeWorkspaceFactory".

Can you explain to me what the difference is between the workspaceFactoryProgID with the ".1" and the one without the ".1"? Is there something in my code that runs the DownloadService that needs to change so that it works with the ".1"?




I have been experimenting to modify the script in order to download only data edited within the past 2 days. I have python script that checks for all data within the past 2 days but because all data is downloaded at the start regardless of time, a bottleneck occurs with the initial download taking 2-4 minutes. This bottleneck is preventing the scripts from running during the day as the total time for all scripts is over an hour (16 layers). 

I have tried: 

x = iteration
y = minOID

where = 'EditDate+>%3D+CURRENT_TIMESTAMP+-1' +  OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)


x = iteration
y = minOID

where = 'EditDate+like+%27%252017-10-02%25%27&' +  OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)

but the modifications are ignored and all data is downloaded from the past month. I am placing the modifications within the 'if count < 1000:' and 'else:
newIteration = (math.ceil(iteration/1000.0) * 1000)' sections.

Do you happen to know of a method to achieve what I am trying to do?

jskinner-esristaff‌, can explain or provide a link that describes the logic in the PrintException(error) function?  It appears it gets the name of the python script being run.  I'm updating a custom tool that is powered by 4 scripts (modules), and want to include which script the error is originating from.  

I also typically include an EnvironmentError and Exception error except statements.  Does this function handle both situations, or would I need to refactor it to be able to handle both exception types?



tselvidge‌ I updated the tool to now include a where clause parameter that will allow you to query the service.

Patrick McKinney‌ I forgot where I snagged this portion of code from.  It should print the file and line of code that the error is originated from.  Kind of losing me with the Environment and Exception error statements question.  I typically use this function like the following:

  perform some action
  PrintException("Error performing action")


Does it download the subtypes too?


This tool currently does not download subtypes.

Thank you very much! I have downloaded and started using it. So far it is saving me an average of 3min for each layer; instead of taking ~1 1/2 hours for 16 areas it is now only taking ~16min. With the change in time I can now run the code for all the areas during the day instead of just nightly updates.

@Jake Skinner

Thanks for the script!

I noticed the globalid and OBJECTID are changed in the downloaded copy. How do I make this work when I want to download data with related tables that use the globalid field as the key?

John.Broecher‌ if you are working with a hosted feature service, the easiest approach will be to enable attachments and add an attachment to the feature.  When downloading the service, choose the option to Get Attachments.  This will create a new field called "GlobalID_Str" in the feature class that contains the original GlobalIDs.

Tried enabling attachments on the parent feature service. It gives me a permission denied error now. It was successfully downloading just the parent table when I unchecked get attachments.

Executing: DownloadService true false # true false John.Broecher ***** "Database Connections\espr1sql14 prod04 GFSurveys GFSurveys_Admin.sde\GFSurveys.GFSURVEYS_ADMIN.testWeedSpray" true #
Start Time: Wed Nov 29 07:54:06 2017
Running script DownloadService...

Generating Token

Copying features with ObjectIDs from 10 to 659

Retrieving Attachments

Failed script DownloadService...

Traceback (most recent call last):
File "M:\GFP\Wildlife\Region3\Broecher\ArcGIS Tools\Download Service 11-27-2017\Download", line 22, in <module>
arcpy.GetParameterAsText(8), arcpy.GetParameterAsText(9), arcpy.GetParameterAsText(10))
File "M:\GFP\Wildlife\Region3\Broecher\ArcGIS Tools\Download Service 11-27-2017\", line 534, in downloadservice2x
urllib.urlretrieve(replicaUrl, cwd + os.sep + 'myLayer.json')
File "C:\Python27\ArcGIS10.5\Lib\", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "C:\Python27\ArcGIS10.5\Lib\", line 249, in retrieve
tfp = open(filename, 'wb')
IOError: [Errno 13] Permission denied: '\\myLayer.json'

Failed to execute (DownloadService).
Failed at Wed Nov 29 07:54:32 2017 (Elapsed Time: 25.83 seconds)

John.Broecher‌ can you share the service to a Group and invite my user account (jskinner_CountySandbox) to the Group.  I can take a look at the service.  Also, can you check to make sure 'Sync' privileges have been enabled on the service?

I can't seem to share that service with you since its not in my content. I shared another one that gets the same error. This one has 3 related tables, and one of the related tables has attachments. I am trying to figure out how to download dozens of feature services that are made in Survey123 and published from desktop for collector. A lot have related tables and attachments. I have been using a script that uses the downloadSurvey123 library and it works OK but it can only see services published from S123, and sometimes it can't see a S123 because for an unknown reason the sharing in AGO doesn't carry over to S123.

I think I was assuming a related table to be treated the same as an attachment. It doesn't seem to work that way. I can get each related table to download only if I use its index in the URL and select tabular data. Still there is a global ID problem. I enabled attachments but you are saying I actually have to have an attachment in every table to make this work? How would I do that without trashing the surveys? Is there an entirely different approach that you would recommend for me? The related table with index 3 actually does have an attachment but it gives the permission denied error still.

Executing: DownloadService true false # false true John.Broecher ***** "Database Connections\espr1sql14 prod04 GFSurveys GFSurveys_Admin.sde\GFSurveys.GFSURVEYS_ADMIN.testMOCREEL3" true #
Start Time: Thu Nov 30 08:29:25 2017
Running script DownloadService...

Generating Token

Copying features with ObjectIDs from 0 to 2

Retrieving geodatabase domains
Adding domain cvd_Species to geodatabase
The value being added to the CodedValueDomain already exists
Applying domain cvd_Species to field Species

Retrieving geodatabase domains
Adding domain cvd_TagType to geodatabase
The value being added to the CodedValueDomain already exists
Applying domain cvd_TagType to field TagType

Retrieving geodatabase domains
Adding domain cvd_TagColor to geodatabase
The value being added to the CodedValueDomain already exists
Applying domain cvd_TagColor to field TagColor

Retrieving Attachments

Failed script DownloadService...

Traceback (most recent call last):
File "M:\GFP\Wildlife\Region3\Broecher\ArcGIS Tools\Download Service 11-27-2017\Download", line 22, in <module>
arcpy.GetParameterAsText(8), arcpy.GetParameterAsText(9), arcpy.GetParameterAsText(10))
File "M:\GFP\Wildlife\Region3\Broecher\ArcGIS Tools\Download Service 11-27-2017\", line 534, in downloadservice2x
urllib.urlretrieve(replicaUrl, cwd + os.sep + 'myLayer.json')
File "C:\Python27\ArcGIS10.5\Lib\", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "C:\Python27\ArcGIS10.5\Lib\", line 249, in retrieve
tfp = open(filename, 'wb')
IOError: [Errno 13] Permission denied: '\\myLayer.json'

Failed to execute (DownloadService).
Failed at Thu Nov 30 08:32:12 2017 (Elapsed Time: 2 minutes 46 seconds)

My post is buried back from Jul 3 but have you tried this method?  It has worked perfectly for me since July. 

This way is equivalent to doing a Export Data from the Item page in AGO  (the one with all the settings and such).  

**So you may want to do the Export manually first to see if this works on your data.


First you need the Item ID of the HFS - get this from the top of the Item page in AGO.  You want the last part after id=

URL looks like 


Then I slightly modified the ArcRest sample script called  (in other words some of this is not my code)

ArcREST/ at master · Esri/ArcREST · GitHub 


LOOK for the caps on what to change  Note the post messed up all the indents - i tried to fix but may be off


import arcrest
from arcresthelper import securityhandlerhelper
from arcresthelper import common

import os, time


def trace():

trace finds the line, the filename
and error message and returns it
to the user
import traceback, inspect, sys
tb = sys.exc_info()[2]
tbinfo = traceback.format_tb(tb)[0]
filename = inspect.getfile(inspect.currentframe())
# script name + line number
line = tbinfo.split(", ")[1]
# Get Python syntax error
synerror = traceback.format_exc().splitlines()[-1]
return line, filename, synerror



if __name__ == "__main__":

proxy_port = None
proxy_url = None

securityinfo = {}
securityinfo['security_type'] = 'Portal'#LDAP, NTLM, OAuth, Portal, PKI
securityinfo['username'] = "PUT USERNAME HERE"    #<UserName>
securityinfo['password'] = "PUT PASSWORD HERE"    #<Password>
securityinfo['org_url'] = ""
securityinfo['proxy_url'] = proxy_url
securityinfo['proxy_port'] = proxy_port
securityinfo['referer_url'] = None
securityinfo['token_url'] = None
securityinfo['certificatefile'] = None
securityinfo['keyfile'] = None
securityinfo['client_id'] = None
securityinfo['secret_id'] = None



# where to save the zip files


#sample   itemId = "5a9ecdf30f53466bad14f5dbbedb1014"  



shh = securityhandlerhelper.securityhandlerhelper(securityinfo=securityinfo)
if shh.valid == False:
print shh.message
admin = arcrest.manageorg.Administration(securityHandler=shh.securityhandler)

item = admin.content.getItem(itemId)
user = admin.content.users.user(username=item.owner)

# use to set jsut some layers. NOT using this
## exportParameters = {"layers":[
## {
## "id": 0,
## "where": "OBJECTID = 1"
## }]
## }

# this version waits for the download so no pause needed
res = user.exportItem(title=HFSname,
exportFormat="File Geodatabase",

exportItemId =
exportItem = admin.content.getItem(exportItemId)
itemDataPath = exportItem.itemData(f=None, savePath=savePath)
 # Delete from AGO list to remove clutter

# rename file so it makes sense
#taken out for ease



except (common.ArcRestHelperError),e:

print "error in function: %s" % e[0]['function']
print "error on line: %s" % e[0]['line']
print "error in file name: %s" % e[0]['filename']
print "with error message: %s" % e[0]['synerror']
if 'arcpyError' in e[0]:
print "with arcpy message: %s" % e[0]['arcpyError']


line, filename, synerror = trace()
print "error on line: %s" % line
print "error in file name: %s" % filename
print "with error message: %s" % synerror




Let me know how it goes.

Greetings Jake,
I have been trying to download my hosted feature service and continually get the attached error. The features are downloading but the attachments are not. Please advise.

troydawson‌ can you share the service you are trying to download to a Group and invite my user account (jskinner_CountySandbox)?

John Broecher‌ even though you receive an error when trying to download a related table, you should still see all of the field populated correctly.  Here are the attributes from the table with an index of 3:

Here is the downloaded result (the tool did error):

You'll see that the field "GlobalID_Str" matches that of the 'parentglobalid' field.  You can use the GlobalID_Str field to recreate the relationship class.

Just sent the request to add you to the group with the feature class. There are several teams deployed currently updating this layer offline via collectors. I need to get a backup of this data downloaded right away. Looking forward to hearing from you.

Thanks in advance

troydawson‌ I just made a minor change to the tool.  Download this new version and it should work.

Greetings Jake,

Amazing stuff there...That worked; however I seem to be having another issue. For every feature there are 2 attachments but based on the file size, only one attachment is not being downloaded properly (see attached). I would need to have both attachments downloaded. Please advise.second attachment not fully downloading/ dont work

troydawson‌ are you able to view the attachment in ArcMap when you identify the feature?

yes, both attachments are listed in Arcmap; however only one can be viewed. The other attachment gives an error when you try to open (see below).

Troy Dawson‌ not sure what is going on with this service.  I was able to reproduce your issue.  I did not receive an error, but the attachment simply states it could not be displayed.  I've shared a service (PropDamage) to the Service Download group for you to test.  This service should work successfully.

As a workaround you can always export the feature service to a File Geodatabase.  This will be stored in your Organization that you can then download.

Hi Jake,

I am not sure if your tool will help me with the following service, could you, please, assist me in obtaining a local dataset of the service?

Data Type: ArcGIS Map Service
Connection: Internet
Name: Alluvioni/Aree_pericolosita_idraulica
Operations Allowed: Map (Display), Query (Identify), Data (Find)
Map Service Type: Not Cached

I might be doing something wrong.



Gijs van den Dool‌ looks like you are missing 'rest' in the Server URL.  Try each URL for each layer in this service:

Thanks Jake,

That worked partial, the tool seems to stop after a while. The first layer hangs on: Copying features with ObjectIDs from 20800.0 to 20850.0

(I changed the records returned to 50 because that timed out earlier)

Any idea why that is happening?

If I display the polygon layer from my gdb I see that a tile (large area) is missing... could it be that the service timed out and that I have to restart (modifying your code) from the last ID? I think I noticed this before, but am wondering why this is happening. Your insight will be helpful.

I added two parameters to your code:

intCountIt = 50.0
intStartRun = 749000

and changed lines [83-84] to 

iteration = intStartRun #int(data['objectIds'][-1])
minOID = intStartRun #int(data['objectIds'][0]) - 1 

and line 94 to >> if count < intCountIt:

and lines [128-129]

newIteration = (math.ceil(iteration/intCountIt) * intCountIt)
x = minOID + intCountIt

But would not like to run subsets of data blocks to get the full layer... too many things can go wrong.

Gijs1973‌ I can't say for sure what is going on with the service.  I received the same error as you did.  It seems there is an issue with a record(s) that have OBJECTIDs between 20,000 and 21,000.  When trying to query the REST service using a where query of OBJECTID > 20000 AND OBJECTID <= 21000 it fails.  I cannot say for sure why this occurs.

Thanks Jake,

I will try tomorrow again (based in Paris, France) to get the missing

records. Unfortunately, the data provider is not responding to emails. I

keep you posted,


Does anybody know why I would be getting this error?

Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 1254, in do_open
h.request(req.get_method(), req.selector,, headers)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\", line 1107, in request
self._send_request(method, url, body, headers)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\", line 1152, in _send_request
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\", line 1103, in endheaders
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\", line 934, in _send_output
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\", line 877, in send
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\", line 1261, in connect
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\", line 385, in wrap_socket
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\", line 760, in __init__
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\", line 996, in do_handshake
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\", line 641, in do_handshake
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\pohlinger\Desktop\WWI Download\Download Service_New\Download", line 16, in <module>
arcpy.GetParameterAsText(7), arcpy.GetParameterAsText(8), arcpy.GetParameterAsText(9))
File "C:\Users\pohlinger\Desktop\WWI Download\Download Service_New\", line 225, in downloadservice3x
response = urllib.request.urlopen(req)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 163, in urlopen
return, data, timeout)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 466, in open
response = self._open(req, data)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 484, in _open
'_open', req)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 444, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 1297, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\", line 1256, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)>
Failed to execute (DownloadService).

I should have also noted, this is the feature service I am trying to download: 

I see there a SSL: CERTIFICATE_VERIFY_FAILED error appears a few times. Not sure if this is the culprit, but it could be.  What happens if you try to download it using HTTP instead of HTTPS?

Philip Ohlinger‌ not sure why you are receiving a certificate error, when it's a valid certificate.  I think it's the port (6443) in the URL that is causing this to fail.  I figured out how to fix this for the ArcGIS Desktop version of this tool.  You can comment out line 195 and add the following:

    response = urllib2.urlopen(req)
    gcontext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
    response = urllib2.urlopen(req, context=gcontext)‍‍‍‍‍

For ArcGIS Pro you will need to replace line 227 with the following:

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
response = urllib.request.urlopen(req, context=ctx)

You will also need to import the ssl module at the top of either script.  Ex:

import arcpy, urllib, urllib2, json, os, math, sys, linecache, ssl


I changed your code slightly, setting the start at the failing object id and increments of 5 objects (it's slow) and again the program stopped at range of object ids. I have narrowed it down to a small range of ID's where the website returns: 

{  "error": {   "code": 500,   "message": "Error performing query operation",   "details": []  } }

(object id 16792)

Perhaps you could include a error handling for this exception; I don't know why there is an error in the query. 


Does this tool work on any ArcGIS Online hosted feature service regardless of it's shared properties? I am having success but only on a feature service that is shared publicly. If I point the tool to something shared only inside the org or only within a group in the org it doesn't work. In all cases I am providing the appropriate ArcGIS Online credentials. I receive a similar error as folks above:

I am using a slightly older version of the tool (about 1 year) as I have not yet created a script tool from the latest downloadable version.

I am assuming the script tool is not already created as a part of the latest download correct? I see the toolbox, however it is empty. Thanks.


An update, I was partially successful running the tool however one of my service layers simply won't export while another nested layer in the same service does. Essentially layer ID #0 runs error free but nothing is exported while layer ID #1, again from the same service, exports as expected with no issues.

For layer ID #0 (the one that doesn't export) the text "...copying features with object id's _ through _ ..." is missing from messages:

Also concerning, I pointed the tool to a feature layer that I know has attachments the tool exported the layer fine but had a warning "no attachments present". It output the feature class itself with not attachment table.

That all being said, I would like to try the latest version of the tool but when I download it and view via Catalog the toolbox is empty. I am running 10.3. I don't see any mention of the toolbox version this is created in. Could there be an issue where running 10.3 can't see say a 10.5 toolbox? Thanks.

jschuckert_InterwestGIS‌, yes I believe the reason you cannot see the script within the toolbox is because this was created using ArcGIS Desktop 10.5.1.  Are you able to upgrade to 10.5.1?

Version history
Revision #:
1 of 1
Last update:
‎07-17-2015 06:12 AM
Updated by: