Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

73700
501
07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

Attachments
Comments

Sorry, I meant like the backside of the tool. 

dhuneyc‌ what version of ArcGIS Desktop are you using?  The zip file should come with a toolbox with all of these parameters already filled in.

Thanks Jake, I want sure if there was anything different to be done across

a citrix farm,,the module will have to be installed on every server correct?

I am using 10.3. When I unzipped it all I got was the script and a toolbox with no tool.

dhuneyc‌ try re-downloading the tool again.  I saved the toolbox as a 10.3 version.

Thanks for this very useful service.  When extracting an AGOL service I noticed that when I used a service other than the 0 index, the attachments were still being downloaded for only the 0 index layer, so in the python script (line 502) i changed 'layers' : '0' to 'layers' : '1' and then it worked for the layer index 1 layer:

if getAttachments == 'true':
# Create Replica to retrieve attachments
arcpy.AddMessage("\nRetrieving Attachments\n")
crUrl = baseURL[0:-7] + 'createReplica'
crValues = {'f' : 'json',
'layers' : '1',

Hi,

using ArcgIS 10.2 desktop and python 2.7.3 installed under C:\Python27\ArcGIS10.2 (as part of ArcGIS  desktop installation) and when try to run getting

C:\Python27\ArcGIS10.2

**** Messages when running the tool*****

Executing: Script true true https://services6.arcgis.com/FAgEZRLybpkpIX4O/ArcGIS/rest/services/BornTestV4/FeatureServer/0 true # # # C:\....\Test\Test.gdb\test # #
Start Time: Wed Apr 11 09:29:00 2018
Running script Script...

Traceback (most recent call last):
File "C:\Durmus\Projects\MobilityReportCard\2018\DownloadServicewithAttachments.py", line 1, in <module>
import arcpy, requests, json, os, math, sys, linecache, shutil
ImportError: No module named requests

Failed to execute (Script).
Failed at Wed Apr 11 09:29:00 2018 (Elapsed Time: 0.05 seconds)

*****

I installed requests and all its dependencies (Dependent packages) using the same process described at: Install requests in python - YouTube  

and tested  using  in the same python environment -2.7.3 using Using the Requests Library in Python  (samples from this website and some web URL) and it appears to be working just fine (see below)

However, as seen above, the download python script is error out indicating requests module missing etc. (not sure whether this is the real issue) 

and my script properties dialog box below

and parameter specifications are below (screenshots)

Note above Output feature class is specified as feature class (even though table indicated to be selected and can be downloaded using the tool) and associated required/optional settings are done for all parameters.

and trying to download the following service https://services6.arcgis.com/FAgEZRLybpkpIX4O/ArcGIS/rest/services/BornTestV4/FeatureServer/0 into the point feature class (pre-created in file geodatabase)

and getting requests module missing error message right at the start?

Filed with ESRI support, support suggested to check with Script author directly (and indicated will internally route this issue with ESRI author) 

Any suggestions/resolutions?

And does this tool download all features regardless of the feature limitations that may be set up in the service (1000 by default) and can download data from the services that are not REST-based (OpenGIS Standards such as WMS etc.) 

in C:\Python27\[ArcGISx6410.2 or  ArcGIS10.2]\Lib\site-packages have you the folder 'requests' ?

Yes this tool download all features/rows (if your where is empty)

No, only from Rest Esri.

agency22‌ thank you for finding that!  I went ahead and updated the code so that it will pull the correct attachments.

I'm a bit confused. Can this tool be run from ArcGIS Online? Can python scripts be run from ArcGIS Online? 

No, this tool cannot be run from ArcGIS Online.  The purpose of this tool is to provide the ability to download a services data so that you have a local copy (i.e. a feature class in a File Geodatabase).

Thanks for the quick reply. 

It sure would be nice to have the ability to run Python scripts from ArcGIS Online. 

You have this ability in Web AppBuilder.  You can add python scripts to a toolbox in ArcMap/Pro, execute it, and then publish as a Geoprocessing Service to ArcGIS Server.  Web AppBuilder has a Geoprocessing widget that allows you to execute Geoprocessing Services.

I am having the same issue with 10.3, but it works in my Pro instance and with my coworkers 10.5.

Trying to understand. For Web AppBuilder, I need to have ArcGIS Server (which we don't have) to run Python code. 

Here's what I would like to do in ArcGIS Online: Client A uploads a csv file or a table. Client A then pushes a button to run Python code which does all the heavy lifting, including geocoding from any coordinate system and any projection (not necessarily lon, lat), data cleaning, error checking, etc. No HTML. No Javascript. And no ArcGIS server. Is that doable?

Most of this functionality is available with ArcGIS Online:

CSV, TXT, and GPX files—ArcGIS Online Help | ArcGIS 

Got it resolved working with ESRI analyst, issue turned out to be the version of the tool that is compatible with 10.2 (Good to have some compatibility document on this together with the tool to help users out).

Another question, in the case of multiple layers combined in a single service (feature or map service with multiple layers) is there any plan (or currently existing version) to download all service related layers (instead of specifying layer links individually)

error 000840: the value is not a string hidden

jhead@lakeworth‌ can you post a screen shot of how you have the tool setup?

Can someone here help my with the following error? I am beyond excited to have found this Download Service, and can't wait to get it implemented into some new workflows.( Full Disclosure: I'm not particularly Py savy)

Error Adding Field: FILE: Z:\00706-0002\2018.04.16.Smoke Testing\DownloadServicewithAttachments.py, LINE: 575

"arcpy.AddField_management(fc, "GlobalID_Str", "TEXT")": Failed to execute. Parameters are not valid.

ERROR 000313: The length of Field Name must not be larger than 10

Failed to execute (AddField).

ArcGIS Desktop 10.5.1.7333 (Standard License)

Many thanks,

   Jonathan

Cannon‌ are you running this as a GP tool, or are you editing the script?  If the former, can you send a screen shot of how you have the tool setup?

Jake Skinner, I hope this is what you were asking for. Thanks for any help!!!

Cannon‌ change your output to a feature class instead of a shapefile.  Since you enabled the attachment option this needs to be a feature class.  It will create an attachments table, which is a component of only feature classes.

That did it! Thanks for the quick help.  I guess I should look into this sort of topic at the SE Users Conference next month.

Thank again,

Jonathan

Hi Jake Skinner

I am getting an error while adding the toolbox. The older version works fine (2015) when I add the tbx file automatically the script added. But the recent one attached in this document needs manual adding of the py script. I have tried adding the script and When I run the tool I get the error like this below;

Error: This tool has no parameters

I am using ArcGIS 10.3 version. 

I appreciate any help to sort this issue.

Thanks 

Vj

vijayasekaran‌ try re-downloading the tool.  I saved the toolbox as a 10.1/10.2 version, so you should be able to open it successfully in 10.3.

When I download this on windows 10 and extract the zip file the linkage between the script and the toolbox id broken.

is there another copy that has the linkages built in?

Also i am using arcmap 10.3.

Shawn

I found the fix on the previous page...

Jake, 

This is a great tool. Is it possible to adapt it so it can run as a scheduled task so that the same feature can be downloaded on a regular basis without having to run the toolbox from ArcGIS Pro. 

Also i increase the number of records it returns, 1000 is not enough. 

Thanks

Sheree

Tweed Shire Admin‌ yes you can run this as a scheduled task.  You will need to update the Variables in the script.  For example, change all the arcpy.GetParameterAsText to what you would enter in the tool.  For the check box variables (i.e. hostedFeatureService) you will either specify 'true' or 'false'. 

#Variables
hostedFeatureService = arcpy.GetParameterAsText(0)
agsService = arcpy.GetParameterAsText(1)
baseURL = arcpy.GetParameterAsText(2) + "/query"
whereClause = arcpy.GetParameterAsText(3)
agsFeatures = arcpy.GetParameterAsText(4)
agsTable = arcpy.GetParameterAsText(5)
username = arcpy.GetParameterAsText(6)
password = arcpy.GetParameterAsText(7)
outputFC = arcpy.GetParameterAsText(8)
getAttachments = arcpy.GetParameterAsText(9)
cwd = arcpy.GetParameterAsText(10)

Also, this tool will download all the features in the feature service.  It's not limited to 1,000.

Excellent.

Thanks Jake I will give that a go.

Thanks Sheree

Hi Jake,

I have changed those variables.

When I run I get this error

I would like have the attachment folders use a different field for the name (other than GlobalID_Str) is that easily changed?

jason heyman‌ the folder name of the attachments is used to join the attachments back to the feature class using the GlobalID_Str field.  There will be a lot more coding if you wanted to change this.  What is your use case to change this name?  Maybe there will be another solution.

Jake - We keep an archive of our images organised by address (folder is named by address field) and it would be nice to be able to adjust this as an option when downloading instead of changing the folder names later in a batch job.

Has anyone tried to use the where clause in this tool to query for a date range?  I can query via the rest api, and I can query the downloaded file geodatabase.  However, no matter what syntax I use in the Where clause dialog box, I get URL erro (URL is incorrect.  Or, Service is secure, please enter username and password)

Thanks

Yes, our requirement was to only download the features after a saved date (which gets updated and saved in a config file each time this whole process is run so that the consecutive runs only pickup data from the last execution). 

Be sure to check your date formatting and also verify your token is being added.  Here's an example of what we've implemented:

#"CreationDate" is the date column to query on
def checkForUpdatedFS(taburl, uname, pword, inputDate):

    if taburl != '':
            try:
                tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
                params = {'f': 'pjson', 'username': uname, 'password': pword, 'referer': 'http://www.arcgis.com'}
                reqp = urllib2.Request(tokenURL, urllib.urlencode(params))
                responsep = urllib2.urlopen(reqp)
                datap = json.load(responsep)
                token = datap['token']
            except:
                token = ''

    params = urllib.urlencode({'f': 'pjson', 'where': "CreationDate>'{}'".format(inputDate), 'outFields': '*', 'token': token, 'returnGeometry': 'true'})
    req = urllib2.Request(taburl + "/query", params)

    gresp = urllib2.urlopen(req)
    gdata = json.load(gresp)

    rCount = (gdata['features'])
    return len(rCount)


dFormat = '%Y-%m-%d %H:%M:%S'
writeDateVal = datetime.strptime(maxDateConfig, dFormat)
writeDateVal = writeDateVal + timedelta(seconds=1) #datetime.timedelta(0, 1) #add 1 second
writeDateValFinal = datetime.strftime(writeDateVal, dFormat) #wr

#test to see if we have any surveys to process using the last submitted date value
recChk = checkForUpdatedFS(serviceName, inputUsername, inputPswd , maxDateConfig)
if recChk>0:

    print recChk
    print 'There are {} surveys after {} to process.'.format(recChk, maxDateConfig)


Scott DeNeice‌ can you post a screen shot of how you have the tool setup?  Here is what I was able to get to work:

Nice tool. It works fine for me when executed the usual way (double click on tool, fill in parameter values in form, etc). However I've been trying to call it from a Python script, and I get errors. Something about parameters that correspond to check boxes on the form. 

Here's the caller snippet...

arcpy.ImportToolbox(tbx_path)

# setup parameters
hostedFeatureService = 'true' #True
agsService = 'false' #False
baseURL = "http://services1.arcgis.com/0U8EQ1FrumPeIqDb/ArcGIS/rest/services/OCFLAtlas_CompPlanCases_View/Featu..."
whereClause = None
agsFeatures = 'true' #True
agsTable = 'false' #False
username = None 
password = None 
outputFC = "\\t3600\D_drive\projects\Orange3\dev_APP\notification_dev\REST scripts from Dean\test_download\OCFLAtlas_CompPlanCases_View"
getAttachments = 'false' #False
cwd = None

# make the call

arcpy.DownloadService(
      hostedFeatureService,
      agsService,
      baseURL,
      whereClause,
      agsTable,
      username,
      password,
      outputFC,
      getAttachments,
      cwd
)

For the Boolean input, I have tried python True/False, and I have tried string 'true'/'false'.

Here's the error when executed in Idle...

Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:19:30) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
RESTART: \\t3600\D_drive\projects\Orange3\dev_APP\notification_dev\REST scripts from Dean\Download Service toolbox tool\test_download.py
ERROR: unable to download CompPlanCases due to arcpy error,
Please check 'Downloading Feature Data' or 'Downloading Tabular Data'

Failed to execute (DownloadService).

Any ideas?

I am using 10.6 on Win10.

the .tbx file is in a plain windows folder, not in a toolbox. 

Service works, as I can download the data without the where clause.  Also, tested against the rest service, I can query in the rest service as well.  I have figured a work around but it is not as clean as this would be.  (What I really want is a between, VISIT_DATE >= date '1/1/2018' AND VISIT_DATE <= date '3/31/2018', for example).

I have also tried VISIT_DATE > timestamp '2018-01-01 00:00:00' to no avail.

Thanks for your help

Scott DeNeice‌ can you share the service to a Group and then invite my AGOL account (jskinner_CountySandbox)?  I can then test this out.

Unfortunately, this is a third party service that I am working with on behalf of branch office of my company.  I do not even know the contact for the third party to ask.

FYI the feature class this comes from has over 600 columns (wayyyyy to big). There are, however only around 1600 features so, the process takes only a minute or two.

My work around is to download the full feature, then do a make table view where I add the where clause as a parameter. Cludgy, but it works. 

I am using this tool as part of a model (in ArcPro 2.1) to download an AGOL feature to then output to a formatted (there are several different reports from this one feature) Excel spreadsheet, and the client wants to have the ability to produce annual, bi-annual or quarterly reports (possibly even monthly reports).

Thanks anyway.

Jake,

I am getting the following error when trying to download my attachments -

Start Time: Friday, June 8, 2018 11:41:19 AM
Running script DownloadService...
Generating Token
Copying features with ObjectIDs from 28663 to 29663
Copying features with ObjectIDs from 29663 to 30663
Copying features with ObjectIDs from 30663 to 31663
Copying features with ObjectIDs from 31663 to 32663
Copying features with ObjectIDs from 32663 to 33663
Copying features with ObjectIDs from 33663 to 34663
Copying features with ObjectIDs from 34663 to 35663
Error occurred: FILE: C:\Users\jheyman\Desktop\DownloadServicewithAttachments.py, LINE: 283
"fs.load(fsURL)": RecordSetObject: Cannot load a table into a FeatureSet
Completed script Download Service...
Failed to execute (DownloadService).
Failed at Friday, June 8, 2018 11:41:59 AM (Elapsed Time: 39.50 seconds)

jason heyman‌ can you share the service to an AGOL Group and invite my user account (jskinner_CountySandbox)?

steve allenstein‌ here is how I had the parameters set up:

I would recommend updating your outputFC variable, and place an r in front of the path so python reads it as a literal string:

outputFC = r"\\t3600\D_drive\projects\Orange3\dev_APP\notification_dev\REST scripts from Dean\test_download\OCFLAtlas_CompPlanCases_View"

https://community.esri.com/migrated-users/8586

Have you seen this error before? I think it's a proxy error....maybe.  It prints the site if I add the following code but still errors out after, maybe I do not have the proxy code set correctly?

s = requests.Session()
s.proxies = {"http":"proxy1.corp:3128","https":"proxy1.corp:3128"}
r = s.get(baseURL)
print(r.text)

Line 190:

# Return largest ObjectID
if whereClause == '':
    whereClause = '1=1'
params = {'where': whereClause, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
response = requests.post(baseURL, data = params, verify = False)
data = response.json()

Variables:

hostedFeatureService = 'true'
agsService = 'false'
baseURL = 'https://gis.energy.gov.ab.ca/arcgis/rest/services/Geoview/Mineral_Agreements_Ext_PROD/MapServer/52/q...'
whereClause = ''
agsFeatures = 'true'
agsTable = 'false'
username = ''
password = ''
outputFC = r'C:\Users\chcarter\Documents\ArcGIS\Default.gdb\Test'
getAttachments = 'false'
cwd = ''

Any information would be greatly appreciated,

Chase

Sorry if someone has already answered this, but is there a way to change the script to be able to overwrite an existing output? I am trying to come up with a workflow where we download a specific hosted feature layer after edits are made, perform some analysis locally, and re-upload another feature layer.

This tool is great btw. Very helpful.

Have you tried to overwrite a layer you download and it failed? If so, can you provide the error code or image of it. Line 152 is env.overwriteOutput = 1, which should allow feature classes to be ovewritten.  You could also try changing 1 to "True," but I don't think that should make a different. But it also wouldn't hurt to try.

Ok I was able to get it to run and overwrite features by running the script through ArcMap and also hardcoding the variables and running the python script outside of ArcMap.  I was hoping to run it once, copy the python code snippet and run the code that way, but that is when I get the error.

ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000725: Output Feature Class/Table: Dataset E:\Projects\\nashville_app.gdb\grids already exists.
Failed to execute (DownloadService).

Chase Carter‌ I believe it may be a proxy issue.  I just executed the tool successfully with this service, not going through any proxy.  If you try another network (i.e. from you home) does it work successfully?

Version history
Revision #:
1 of 1
Last update:
‎07-17-2015 06:12 AM
Updated by:
 
Contributors