Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

[#1]

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

[JakeSkinner]

Anthony Von Moos‌ the attachments need to be downloaded to a directory if you want them to exist within the feature class.  The attachments are downloaded to disk, and then added to the attachment table.  The attachments can be deleted from disk afterwards, as they then exist within the attachments table.

If you do not want attachments at all, then you will want to set the getAttachments = 'false'.

[Anonymous User]

Awesome, I was able to get that to work with one of my features. When I try to use that for a table with attachments I get the following error.

Traceback (most recent call last):
File "C:\Road Approches\Subgrade_isnp.py", line 663, in <module>
if hasrow == True:
NameError: name 'hasrow' is not defined

Any ideas?

Thanks again!

[Anonymous User]

If I add you to a group could you take a look at my table with attachments I'm trying to download? I have had no luck getting this to work.

Thank you

[MikeSteven]

Apologies Jake Skinner, I wasn't actually putting in the server name (the variable in that part of the script is named 'server' and I was just using that variable name).

I don't know the server name so what I had used in the script was:

http://<domain name>:6080/arcgis/admin/generateToken

but it still gives the same error message. Any ideas why?

[JakeSkinner]

Anthony Von Moos‌ this is a bug with table's and attachments.  The tool creates a replica of the data, and the bug appears to not retain any attachments when doing so for a table.

One possible workaround would be to use the ArcGIS API for Python.  Here is a sample script that will download hosted feature service to a local zip file:

ArcGIS-Python-API-Samples/Backup Hosted Services.ipynb at master · jmirmelstein/ArcGIS-Python-API-Sa... 

[JakeSkinner]

Mike Steven‌ if you copy/paste the URL you reference above into a web browser, does it resolve correctly?

[MikeSteven]

Jake Skinner in a web browser if I go to:

http://<domain name>/arcgis/admin/generateToken 

then there is a message saying that administrative access has been disabled.

In a web browser if I go to:

http://<domain>:6080/arcgis/tokens or http://<domain>:6080/arcgis/tokens/generateToken 

then the browser says that the connection timed out.

In a web browser if I go to the same urls as above but without the :6080, i.e.:

http://<domain>/arcgis/tokens or http://<domain>/arcgis/tokens/generateToken 

then it eventually does load a page which would generate a token but it is taking a long time (in the region of 30-60 seconds for the page to load). However putting the same urls into the python script gives the error message about "URL is incorrect...". Is it possible that the problem is with the amount of time it seems to be taking to generate the token?

[Anonymous User]

Do you know how to run a Jupyter notebook on a schedule? I'm looking to run this on a daily basis and can't find a simple way to run my .ipynb on a schedule.

[JakeSkinner]

Anthony Von Moos‌ do you have ArcGIS Pro installed?  You can have the code in a .py file and run this on the server where Pro is installed.  For example, open notepad and add the following lines:

cd "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3"

python "C:\path\to\script.py"‍‍‍

Save the text file and then rename the extension to .bat.  You can use Windows Task Scheduler to execute the bat file on a scheduled basis.

[Anonymous User]

Awesome, Thanks for the info!

[Anonymous User]

I have another ArcGIS  script I need assistance with if you have time. Currently I have some editable features that our organization edits and uses on a daily basis. I’m wanting to create copies of these hosted feature services with the edit capabilities disabled for public maps I have setup. Do you know of a way with ArcGIS Python Api to do this? I would like the public layer to be updated on a daily basis.

[JakeSkinner]

You could create an application using Web AppBuilder for the public.  As long as you don't add an Edit widget, they won't be able to edit.

If you're concerned about someone hacking the REST URL of your feature service, you can limit the usage of the service to a specific web map/application.  See the section Limit Usage of Your Shared Service Item in the following link:

ArcGIS Server web services—ArcGIS  Help | ArcGIS 

[Anonymous User]

My concern would be that the layer is still available to edit if somebody searched the feature layer through the arcgis website. I ended up finding the Create View Layer tool which would allow me to keep the editable layer private and also provide a version to the public that has no editing capabilities.

[deleted-user-s-XH6KNGak26]

Hi Jake Skinner‌

When I tried to use the tool, I got the following error any idea about this.

Ps: I used this tool before and it was fine, but I tried to use it again and doesn’t work with me, may you please help in this.

Cheers,

Rawan

[JakeSkinner]

Rawan Saleh‌ you may be using an older version of this tool.  Can you try downloading the tool again and try to execute?

If you still receive the error, can you share the service you are trying to download to a Group in ArcGIS  and invite my user account (jskinner_CountySandbox)?

[deleted-user-s-XH6KNGak26]

Hi Jake Skinner ,

I use the latest version for this tool, and here the link for the services which I tried to download:

http://services.ga.gov.au/gis/rest/services/NEXIS_Building_Exposure/MapServer/7

Cheers,

Rawan

[JakeSkinner]

I think the service is timing out.  There are 581 records, but when trying to query this, it errors.  The query is the following:

http://services.ga.gov.au/gis/rest/services/NEXIS_Building_Exposure/MapServer/7/query?where=objectid...

[deleted-user-s-XH6KNGak26]

Thanks for checking this, Do you have any idea about slovong this problem?

[865_FCGIS]

We are trying to use this tool to download feature service data from our secure portal environment, and we are getting the attached URL/login error.

I have triple checked the Username and Password (same as the login to the portal environment) and copied and pasted the service URL directly from the REST service location. Is there any other reason why we may be getting this error?

The feature service URL is https://geo.forsythco.com/server/rest/services/Hosted/StormGreenInfrastructure/FeatureServer/1  

Thank you!

[JakeSkinner]

Rawan Saleh‌ it looks like the query will work when there's less than 100 features queried.  You would need to update the code so that it is downloading less than 100 features at a time.

[PayneRingling]

Hi Jake Skinner - I have the script running without any errors, though I was curious if there was a way to assign a domains description instead of coded value. 

I see in the updated on 8/10/17 you changed it to the coded value, could you possibly inform me on how to change that to the description instead?

Thanks a bunch, this tool is very useful 

[JamesCrandall]

Running into an issue when executing the DownloadServicewithAttachments.py script from the IDE with hardcoded variables (rather than the toolbox).  (we want to execute this via task scheduler is the reason why).  This is the message that prints in the IDE interpreter window:

C:\Python27\ArcGISx6410.4\lib\site-packages\requests-2.6.0-py2.7.egg\requests\packages\urllib3\connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
C:\Python27\ArcGISx6410.4\lib\site-packages\requests-2.6.0-py2.7.egg\requests\packages\urllib3\connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)

[SeanRedar]

Ever manage to resolve that issue?  I get the same error using Desktop 10.6.

[JakeSkinner]

James Crandall‌ add the following lines in the script under the import statements:

# Disable warnings
requests.packages.urllib3.disable_warnings()

[JakeSkinner]

Daniel Wade‌ and Sean Redar‌ I'll have to look into updating this code for federating environments.  The url will change on how the token is generated (i.e. https://server.esri.com:7443/arcgis/sharing/rest/generateToken/ )

[SeanRedar]

I am attempting to extract from our AGOL Organization account services directory (ex https://services3.arcgis.com/ ).  I did change the token URL to our org which works with another python script against the services directory so its a little puzzling why this doesn't.

[JakeSkinner]

Sean Redar‌ can you send a screen shot of how you have the tool setup? 

Also, can you share the service to an AGOL Group and invite my user account (jskinner_Countybox).  I can take a look at it.

[SeanRedar]

Jake, I believe the request has been sent and permissions should be ok but with AGOL is always a bit of crap shoot.  FYI, I could only find jskinner_CountySandbox.

Update, the tool did work when I supplied the URL of the Feature Layer versus the URL of the Feature Service (which also has a related table).   

[JakeSkinner]

Sean Redar‌ you will be able to download the features using the following URL:

https://services3.arcgis.com/aRSRPMqAe0fNWYeU/arcgis/rest/services/service_a5923f758eaa4817b1f34afeb...

And the associated table:

https://services3.arcgis.com/aRSRPMqAe0fNWYeU/arcgis/rest/services/service_a5923f758eaa4817b1f34afeb...

However, you won't be able to download the attachments in the table.  There's an issue with REST API that will not grab attachments when they are coming from a table. 

A workaround would be to use the ArcGIS API for Python.  Here is a sample script that will download hosted feature service to a local zip file:

ArcGIS-Python-API-Samples/Backup Hosted Services.ipynb at master · jmirmelstein/ArcGIS-Python-API-Sa... 

[SeanRedar]

Thanks Jake, we will be able to accomplish what we need too with this additional information.

[JamesCrandall]

What do you recommend to do if the hosted FS has over 1gb of attachments?

We'd like to only pull down after a specific "CreateDate" but any modifications to the .py source we've attempted has failed due to issues with GlobalId values between the parent feature and child attachment.

[JakeSkinner]

James Crandall‌ are you specifying a query within the script?  Ex:

CreateDate > date '3/15/2018'

[JamesCrandall]

Correct.  It seems that the feature service downloads the data with the query applied, but all of the attachments are being pulled down rather than only the related ones.

[JakeSkinner]

I could not reproduce this.  The query definition only downloaded the related attachments.  Could you share the service to an ArcGIS  Group and invite my user account (jskinner_CountySandbox)?

[JamesCrandall]

Interesting.  I'll take another look before bothering.

[YvanSojdehei1]

Im having an issue with bringing down a particular service:(https://utility.arcgis.com/usrsvcs/servers/793651fa1f2c48d29765d66b28ae1715/rest/services/Permits/DI... ) that shoots me this error:

Traceback (most recent call last):

  File "C:\Users\john\Documents\ArcGIS\DownloadService(COPY).py", line 185, in <module>
    data = response.json()
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 Failed to execute (Test).

Ive used this tool for several other services and I cant figure out why this particular one is giving me this error

[JakeSkinner]

Yvan Sojdehei‌ this appears to be the URL of a secure service that you've added to ArcGIS .  Can you try the initial URL that you added as an item to ArcGIS ?

[XiujuZhou]

appreciate the tool you provde. however I always got error, I setup the service as not secure. 

-------------------------------------------------------

Executing: DownloadService true false http://services1.arcgis.com/XpTq0H1nJ5DHgofs/arcgis/rest/services/service_2d983d48b0c8435897ca1481fe... # false false judyzhou@COH ***** C:\Data\VehicleTripInspection.gdb\test true C:\Data
Start Time: Fri Oct 19 14:10:15 2018
Running script DownloadService...

Generating Token

URL is incorrect. Or, Service is secure, please enter username and password.

Completed script DownloadService...
Failed to execute (DownloadService).
Failed at Fri Oct 19 14:10:16 2018 (Elapsed Time: 1.55 seconds)

[XiujuZhou]

appreciate the tool you provde. however I always got error, I setup the service as not secure. 

-------------------------------------------------------

Executing: DownloadService true false http://services1.arcgis.com/XpTq0H1nJ5DHgofs/arcgis/rest/services/service_2d983d48b0c8435897ca1481fe... # false false judyzhou@COH ***** C:\Data\VehicleTripInspection.gdb\test true C:\Data
Start Time: Fri Oct 19 14:10:15 2018
Running script DownloadService...

Generating Token

URL is incorrect. Or, Service is secure, please enter username and password.

Completed script DownloadService...
Failed to execute (DownloadService).
Failed at Fri Oct 19 14:10:16 2018 (Elapsed Time: 1.55 seconds)

[JakeSkinner]

Xiuju (Judy) Zhou‌ you need to specify the index value of the service.  Ex:

 http://services1.arcgis.com/XpTq0H1nJ5DHgofs/arcgis/rest/services/service_2d983d48b0c8435897ca1481fe...

[XiujuZhou]

Thanks Jake for you prompt response! I can run successfully now. but I don't see attachment saved to my specified folder c:/data, the message is:

Executing: DownloadService true false https://services1.arcgis.com/XpTq0H1nJ5DHgofs/ArcGIS/rest/services/service_2d983d48b0c8435897ca1481f... # true false # # C:\Data\VehicleTripInspection.gdb\test2 true C:\Data

Start Time: Fri Oct 19 14:41:16 2018

Running script DownloadService...

Generating Token

Copying features with ObjectIDs from 0 to 1

Retrieving Attachments

Service does not have 'Sync' operation enabled

[JakeSkinner]

Xiuju (Judy) Zhou‌ can you share the service to an ArcGIS  Group and then invite my user account (jskinner_CountySandbox)?  I can try and execute the tool and see what's going on.

[XiujuZhou]

Thanks! Yes. I invited your account my AGO group.

That is an ArcGIS Online Survey123 app with a form and feature service hosted by AGO. Please let me know if anything further.

Thanks!

Xiuju (Judy) Zhou

Systems Consultant

Houston Public Works-IT-GIS

611 Walker

832-395-2670

[JakeSkinner]

Xiuju (Judy) Zhou‌ go to the Feature Service details page > Settings tab.  Make sure 'Sync' is enabled:

[XiujuZhou]

Thanks Jack! I am able to download attachments now and see the folder with globalID named in hard drive . when I open the feature layer in ArcMap and identify a feature, I see globalID field without link to attachment files. How to open attachment file by clicking the globalID field or by anyway?

[JakeSkinner]

Xiuju (Judy) Zhou‌ you will click in the middle where it says 'Attachments':

[XiujuZhou]

Thank you so much Jack! I found it. it is working now. I am also wondering how to run a python file. see screenshot. I run failed by either way of two below.

• When I click it in ArcCatalog, a black command window shows in one second and disappeared.
• when I run it in windows command line, I got error:

[JakeSkinner]

You will need to edit the python file (.py) in a text editor or IDE (i.e. PyScripter).  You will want to edit the section under #Variables.  Here is an example:

[XiujuZhou]

Thanks lots Jack! I really appreciate your great HELP! it works now. A further question, is there anyway to same data to enterprise database SDE (SQL Server or Oracle)? if yes, where to save attachments to? database or hard drive?

[XiujuZhou]

Thanks lots Jack! I really appreciate your great HELP! it works now. A further question, is there anyway to same data to enterprise database SDE (SQL Server or Oracle)? if yes, where to save attachments to? database or hard drive?