Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

69578
501
07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

Attachments
Comments

Jake,

Thank you for the reply. I was finally able to get on a 10.5 machine and run the latest tool with no issues. I tried pointing the tool at several ArcGIS Online layers, some shared publicly some not and some having attachment data others not, in all cases it worked very well. Thank you for developing and sharing such a useful tool!   

Steven Castongia‌ it looks like you are using the 2x version for ArcGIS Desktop.  You can copy the DownloadServicewithAttachments_2x.py to the same directory as your other script.  You can then use the following statement in your current script:

import DownloadServicewithAttachments_2x
DownloadServicewithAttachments_2x.downloadservice2x(arcpy.GetParameterAsText(0), arcpy.GetParameterAsText(1), arcpy.GetParameterAsText(2),
        arcpy.GetParameterAsText(3), arcpy.GetParameterAsText(4), arcpy.GetParameterAsText(5), arcpy.GetParameterAsText(6), arcpy.GetParameterAsText(7),
        arcpy.GetParameterAsText(8), arcpy.GetParameterAsText(9), arcpy.GetParameterAsText(10))

You will replace all of the arcpy.GetParameterAsText variables.  Ex:

import DownloadServicewithAttachments_2x
DownloadServicewithAttachments_2x.downloadservice2x('true', '', 'https://services.arcgis.com/9N1857LB1QVyzq54/arcgis/rest/services/ITSAsBuilts/FeatureServer/0',
        '', 'true', '', 'scastongiaCity', 'password',
        r'S:\Public\CityGIS\ArcSDE\connections\development\cdot_dev.sde\ITSAsBuiltsCamera', '', '')

What Jake said. I guess you would like to pass the services as a parameter,

be careful with the string formatting. I would make the tool a function,

which you call from your script.

Good luck, the tool is very adaptable, you will work it out.

Gijs

Thanks, Jake!  I have been trying to use your example and that fixed that error message.  The problem I am having now is that it is not executing anything.  It returns "exit code" but it didn't actually export the hosted feature service. 

Steven

Steven,

By when do you have this fixed... I could have a look tomorrow (if you can

pass me your code)... I'm not at a desk right now.

G

Hi Gijs,

I have no set timeframe on getting this done but it is something I want to implement so that I can copy our hosted feature service data out of the cloud.  Below is my script.  The first print statement "made it this far" will display while running the script but the last print statement "cameras copied" will not, which tells me that the tool is not executing correctly.  Thanks for your help!

import arcpy
import os
import urllib2
import urllib
import json
import math
import sys
import linecache
import DownloadServicewithAttachments_2x


arcpy.env.overwriteOutput = 1 # overwrite output is true

print "made it this far"
#ITSAsBuilts/Camera
DownloadServicewithAttachments_2x.downloadservice2x("true", "", "https://services.arcgis.com/9N1857LB1QVyzq54/arcgis/rest/services/ITSAsBuilts/FeatureServer/0", "", "true", "", "AGOLUsername", "password", r'd:\temp\temp.gdb\ITSAsBuilts_Camera', "", "")

print "cameras copied"

Steven,

I will have a look tomorrow, if Jake doesn't beat me for a solution, have a

nice evening.

Gijs

Steven,

I tried your script and it executes until line 200 in the *_2x.py script; I placed a break on the exit points.

python error

I don't have access to your service, but when I use this line:

DownloadServicewithAttachments_2x.downloadservice2x("true", "", "http://www.geoservices.isprambiente.it/arcgis/rest/services/Alluvioni/Aree_pericolosita_idraulica/Ma...", "", "true", "", "", "", r'C:\Default.gdb\test_script', "", "")

I get results.

Please check your access credentials; you could try first to run the tool from ArcGIS, linking the py script to the toolbox, to test your connection, and the geodatabase must exists, otherwise the tool can't write the features out.

Hope this helps,

Gijs

Hi Gijs,

Thank you for your help! I think you are correct, there is a problem during the credential verification that is preventing the data from being copied. That particular hosted feature layer is only shared with a group and only that group can edit it so that is probably the issue. I swapped it out with a hosted feature layer that is shared with the public and it ran correctly.

Thanks again for your help! I greatly appreciate it!

Steven Castongia, GISP

GIS Supervisor

scastongia@charlottenc.gov<mailto:scastongia@charlottenc.gov>

704-336-3816

Hi Steven,

You're welcome, it was a quick test, and your script helped me to run my

extraction outside of the ArcGIS main interface

Best regards,

Gijs

With the latest version of Pro, 2.0.1, fails with 

Running script DownloadService...
Generating Token
Copying features with ObjectIDs from 0 to 345
Failed script DownloadService...
Traceback (most recent call last):
File "C:\PY_SCRIPTS\ARCHIVE\Download Service.py", line 16, in <module>
arcpy.GetParameterAsText(8), arcpy.GetParameterAsText(9), arcpy.GetParameterAsText(10))
File "C:\PY_SCRIPTS\ARCHIVE\DownloadServicewithAttachments_3x.py", line 537, in downloadservice3x
addDomainsHosted(outputFC)
File "C:\PY_SCRIPTS\ARCHIVE\DownloadServicewithAttachments_3x.py", line 51, in addDomainsHosted
data = json.loads(json_acceptable_string)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1056 column 71 (char 31056)
Failed to execute (DownloadService).
Failed at Wednesday, December 27, 2017 9:44:19 AM (Elapsed Time: 26.00 seconds)

Works fine in Arc 10.5.1

tpcolson‌ try another service to see if you receive the same error.  I was able to get this to work successfully:

I was able to get this to work for all of my hosted feature services.  Thank you for everyone's help!  This is a great tool!

Steven

Same error on multiple different computers, tried over two dozen hosted feature services. Fyi I have installed the https://developers.arcgis.com/python/guide/install-and-set-up/#Step:-2-Install-the-arcgis-package...... that somehow breaking the rest of the python functionality that came with Pro?

Running script DownloadService...

Generating Token

Copying features with ObjectIDs from 0 to 26098

Failed script DownloadService...

Traceback (most recent call last): File "C:\PY_SCRIPTS\ARCHIVE\Download Service.py", line 16, in <module> arcpy.GetParameterAsText(8), arcpy.GetParameterAsText(9), arcpy.GetParameterAsText(10)) File "C:\PY_SCRIPTS\ARCHIVE\DownloadServicewithAttachments_3x.py", line 537, in downloadservice3x addDomainsHosted(outputFC) File "C:\PY_SCRIPTS\ARCHIVE\DownloadServicewithAttachments_3x.py", line 51, in addDomainsHosted data = json.loads(json_acceptable_string) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py", line 319, in loads return _default_decoder.decode(s) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx)json.decoder.JSONDecodeError: Expecting ',' delimiter: line 892 column 71 (char 24998)

Failed to execute (DownloadService).

More detail, from Pro, when I select "Copy Python Command", note the first few letters (extra dots)....

arcpy..DownloadService(True, False,

Hi Dave

How did you fix your issue?

/Sanne

Hi jskinner-esristaff, love the tool. While it was working perfectly for me for the past few months I've come across an undefined error. I've attached some screenshots below.

 

1) The tool always seems to open to a highlighted issue but does not provide any information when hovering over it with the cursor. 

2) When trying to execute the tool with the parameters set an error is displayed, but again, no information is given. 

Any insight into what might be the problem would be great. 

Thanks!

-Aaron 

AaronFrancis‌ try re-downloading the tool and see if the same error occurs.  One other thing is to try this on a different server.

Also, it appears you are trying to download an ArcGIS Server service.  Try checking this option rather than the Hosted Feature Service option.

Thanks for getting back to me Jake. I actually did initially try both of those potential solutions but didn't have luck. It doesn't seem to matter whether I'm attempting to access a Hosted Feature Service or an ArcGIS Server Service, when I initially open the tool I receive a blank error symbol, and receive a blank error message whenever trying to execute the tool, regardless of what my parameters are. 

I did find this article indicating that blank error messages might be due to the ArcToolbox.dat file, but in this case they're talking about the system tools whereas my problem is associated with a custom tool. 

-Aaron 

AaronFrancis what version of ArcGIS Desktop are you using?  The toolbox was created with 10.5  This could be the issue.  Are you able to upgrade to 10.5 or 10.6?

I'm currently using 10.5. Do you think upgrading to 10.6 could possibly resolve the situation if there is an issue with my 10.5 configuration?

This tool should work with no issues at 10.5, but it wouldn't hurt to upgrade to 10.6.

Hi Jake, I finally had time to upgrade to ArcGIS 10.6 and I guess that's what fixed it. It was definitely something with the configuration of my ArcToolbox, but upgrading to 10.6 brought everything back the standard configuration, not sure what I had done to mess it up but this tool was not the only one that wasn't working properly.

Thanks for the quick replies!

- Aaron

Jake,

Is there way to download attachment, but keep it in FGDB, not totally download attachment in a folder?

yuimyt‌ the attachments will be stored in the File Geodatabase within the ATTACH table.  There is a field of type BLOB that stores the attachments.  The attachments are first written to disk before they are loaded into the table.  After the tool completes, you can delete the folder you specified to store the attachments.

Hi Jake,

Your tool works fine for me when I use Where Clause with text fields.  However when I use Where Clause with date field: EditDate=Current_Date, it gives me error message: URL is incorrect.  Or, Service is secure, please enter username and password.  Any idea how I could fix this ? 

My layer is a feature service layer from AGOL.  EditDate and Editor are 2 fields that automatically added when you turn on "Keep track of who created and last updated features".

Thank you so much for your help.

helen chu‌ check to make sure the query is working in the REST services directory.  For example, you will most likely need something like:

EditDate > date '2018-02-11'

You are right.  My query (EditDate = Current_Date) did not work in rest service directory "Unable to perform query.  Please check your parameters".

If I put EditDate ='02-14-2018' it works.  What I want is today date not a fixed date.  I've tried from now, date to current_date.  None works.  Any idea? Thanks.

Here are the supported SQL functions:

Supported SQL functions in ArcGIS Server—ArcGIS Server Administration (Windows) | ArcGIS Enterprise 

Try CURRENT_DATE.  It may be case sensitive.

Thanks Jake.  It's CURRENT_TIMESTAMP.  My brilliant co-worker figured it out! 

Is there any reason why converting the queryURL that references a hosted feature service on AGOL would populate the FeatureSet() with GlobalID values as {00000000-0000-0000-0000-000000000000}?

Sorry to cross post but I'm not making any progress from my post in python section https://community.esri.com/thread/209848-arcpyfeatureset-from-feature-service

Very nice tool, thank you!

I'm trying to maintain the original ObjectID or GlobalID from the hosted feature service, but it seems to create new ones.

Hosted to SDE or FGDB: Different GloablID and different ObjectID

Hosted to Shapefile: No GlobalID and different ObjectID.

Before I get into it, is this the expected behavior?

dgoldb2@luc.edu‌ yes this is expected.  The tool will make a copy of the data.  Anytime this occurs new OBJECTIDs and GlobalIDs are generated.  One thing you can do to maintain the GlobalIDs is to enable attachments on the feature service.  When you run the tool, check the option to download attachments as well.  This should create a new field in the output feature class called GlobalID_str.  This will be the original GlobalIDs.

Just to post my findings: the issue I was having with GlobalID/ParentGlobalID values as {00000000-0000-0000-0000-000000000000} when implementing .load() on the FeatureSet with the service url was reproduced by ESRI support.  This appears to be a problem with ArcGIS Desktop 10.3.1 (32/64 bit) and 10.4 (64 bit).

My workaround was to load a json object and parse that to generate the desired output.  Luckily, my requirement was to simply output individual csv files for the contents of the service layers/tables, so it was a matter of implementing the csv.DictWriter() to perform that task.

One other issue that came up was with date columns would be written to the csv outputs as epoch format, so I needed to run a conversion on any date column prior to writing the csv.

Im having similar issues, using CURRENT_DATE. Im trying to pull the data that has been edited in the last 24hrs. I have tried using CURRENT_TIMESTAMP, yet it still doesn't solve the issue. Its very strange, as the script worked for a period of time, now gives me the error shown below.

Below is the setup i am trying to get consistently working. Whats even stranger that if i change the WHERE clause to EditDate >= CURRENT_DATE - 10, it pulls the data correctly. Any help or guidance would be much appreciated. Im using ArcGIS Desktop 10.5.

Ashley Rymer‌ go to My Organization tab in ArcGIS Online > Edit Settings > Security tab.  I found that 'Allow only standard SQL queries' needs to be checked for the supported SQL functions to work.

I have checked our organisation tab and 'Allow only standard SQL queries' is checked. It just seems strange why CURRENT_DATE - 1 doesn't work, yet CURRENT_DATE - 10, works fine.

Go to the REST URL of the service and try entering the query there.  Does it work when entering it there?

I just tried that query and it seems to work fine. Showing the results below. When i run using the script, it seems to run for a period of time then, give me the error i posted earlier.

ashley.rymer_EEGGIS‌ feel free to share the service to an ArcGIS Online Group and invite my user account (jskinner_CountySandbox).  I can take a look at it and see if there are any issues.

Jake Skinner‌ i have created a group, add the shared the two main layers that we are having issues with. You should have been invited to join the group. Both the layers were working Sunday and Monday, now only one is working.

eeg_admin‌ the tool executed successfully for only the e_a_kc_gy_inspection service.  The other service may have been corrupted some how.  I would recommend downloading the entire service, and then re-upload to ArcGIS Online to see if this fixes the issue.

I have added spatial filter https://github.com/nicogis/DownloadService

Hello, I am running into this error. I am using ArcMap 10.4 in a Citrix Environment. 

Executing: DownloadService true false https://services1.arcgis.com/KbxwQRRfWyEYLgp4/ArcGIS/rest/services/BLM_UT_Cultural_Sites_Hosted/Feat... # true false # # T:\UT\GisData\ut\so\projects\UT_AGOL\BLM\UASF.gdb\Sites_Mobile true #
Start Time: Tue Apr 03 14:51:26 2018
Running script DownloadService...
Failed script DownloadService...

Traceback (most recent call last):
File "T:\UT\GisData\ut\so\projects\UT_AGOL\BLM\ConfigFiles\DownloadServicewithAttachments.py", line 1, in <module>
import arcpy, requests, json, os, math, sys, linecache, shutil
ImportError: No module named requests

Failed to execute (DownloadService).
Failed at Tue Apr 03 14:51:26 2018 (Elapsed Time:

'requests' is not built in module so you need install it.

In Windows you can make pip install requests

Scott Kichman‌ you can install the requests module with the following steps:

1.  Open a command prompt as an administrator

2.  Navigate to the Scripts folder in your python installation directory.  Ex:

3.  You can then run the following command:

pip install requests

After this installs see if the tool executes successfully.

Hello all, thank you for the help. I unfortunately am working in a Citrix

environment with our ArcGIS deployment and do not have administrative

access to the servers.

Could you possibly write those procedures out as I can then forward this

through our help remedy system that they install the correct module.

We are still using ArcGIS 10.4

Thank you!

Thank you so much for creating this tool. Is there anyone who can show me what their parameters look like when the tool is set up?

Daniel Huneycutt‌ if you scroll through the comments you'll see some screen shots.  Ex:

skichman@blm.gov_BLM_EGIS‌ not sure what you mean by write the procedures out.  I would just copy the steps in my comment and forward to an administrator for your ArcGIS environment.

Version history
Revision #:
1 of 1
Last update:
‎07-17-2015 06:12 AM
Updated by:
 
Contributors