Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

58416
501
07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

Attachments
Comments

I was able to poll SAML using this code here.  https://github.com/DOI-BLM/requests-arcgis-auth

My script does not "sync" it just sends a request to download a GDB, then renames and adds that GDB to a local directory each night.

I can download feature data, but not related tabular.  Script fails at line 391, "fs.load(fsURL)": RecordSetObject: Cannot open table for Load.

Going through all the comments here, this seems to have been resolved a while ago.  I've installed all the python dependencies that error logging caught (request, chardet, idna, certifi).  Can't seem to figure out if this issue is related to settings on the feature service or not, or with the OBJECTID field.... 
 

Chris Czerwinski‌ can you share the service to a Group in ArcGIS Online and invite my user account (jskinner_CountySandbox).  I'll take a look to see if I can figure out what's going on.

Thanks , you should have access now.

Chris Czerwinski‌ I could not reproduce this error in ArcGIS Desktop for either table.  Here is how I had the tool set up:

Weird, as far as I can tell I'm using the same input parameters.  As mentioned, I had to add a few python modules (request, chardet, idna, certifi) to get this tool to go.  This error message is a bit more ambiguous for me, but I'm wondering if I'm still missing something.  Tool works great for retrieving the feature class, but continues to crash when retrieving either of the two related tables (GlobalID to GUID relationship class 1:M).  Could I be missing something in my python libraries?  

Chris Czerwinski‌ what version of ArcGIS Desktop are you using?  I tested with 10.6.1.

Currently 10.4.1.  
I tested this small script suggested here: https://pro.arcgis.com/en/pro-app/arcpy/functions/recordset-and-featureset-properties.htm to see if I could work with featureset in a local environment, and I get this:

Chris Czerwinski I would try upgrading to 10.6.1 or 10.7 and see if you receive the same error.

Okay - thanks for your time on this.  Unfortunately upgrading is out of my control.  Until that happens, what would you suggest as an alternative data retrieval workflow.  I'm really not a fan of downloading a GDB to my AGOL account, and then downloading that locally.  Especially when some of our local geodatabases, need to be updated weekly, or even daily. 

Chris Czerwinski‌ if you have access to ArcGIS Pro, you can use the Table to Table tool there.  You will enter the URL to the service, i.e. https://services1.arcgis.com/d0ZCwU7eGKVeNiEE/arcgis/rest/services/RVCA_PLF_SiteVisit_Service/Featur... , for the input table. 

First time using ArcPro, but that works on the feature class and related tables.  Unfortunately, I can't preserve the GlobalIDs (in the FC) and their counterpart GUID (in the related tables) that make up the relationship classes.  There's an environment setting to preserve this data for the Feature Class to Feature Class tool (but doesn't seem to work), but not the table to table tool.  I tried the append (since it has a preserve GlobalID environment setting), but that also doesn't work.

Jake Skinner

When I ran the script, it downloaded the features and attachments, but not the related table. I have a Survey123 hosted feature class with a repeat in a related table that I would like to download via script so that it could be scheduled. If I go on AGOL, to the Survey123 data page, I can Export to a File Geodatabase manually and it does bring down the attachments AND the related table records. Is there any way to add that to the above tool, which is otherwise fabulous?

Thank you!

Christal Higdon‌ currently this tool will not download related data.  You can run the tool again and download the related table, but the relationship class will need to be created afterwards.  Creating the relationship class could also be a script.  You can execute the Create Relationship Class tool in ArcMap and then copy the result as a python snippet from the Results window.

I'll be working on a script soon to automate downloading multiple services (with related tables) from AGOL on a scheduled task (probably weekly).  If successful, I'll share the workflow here. 

Thank you Chris! I’m looking forward to it ☺

These scripts work for me in ArcPro or Jupyter Notebook (python3).  There's only a few lines that need editing or revisions to make it work for a specific service layer:

https://community.esri.com/thread/208467-download-feature-service-as-file-geodatabase

https://esri.jiveon.com/docs/DOC-10441-downloading-feature-layer-attachments?commentID=56737

F
or lines 123 and 137 in the DownloadAttchments.py, I had to edit from "renameFile(currentAttachmentPath, newAttachmentPath)" to "renameFile(currentAttachmentPath[0], newAttachmentPath)".

Was looking all over for this. It worked great for me. Thank you!

I have a weird issue - I don't get any errors and I don't get the feature class either. Any ideas why?

Have you tried it without the [County = 'Ventura'] condition?  

Not sure why this tool is failing, but ArcGIS Pro's Feature Class to Feature Class tool will work:

This tool works great.

I wonder if you could show me what to change in the script if I want to export the same feature service to the same geodatabase every day.

Thank you very much.

Keisuke Nozaki‌ if you are using ArcGIS Desktop you can execute the tool and then go to the Results window > right-click on the result > Copy as Python Snippet.  This is explained in the below link:

Understanding tool syntax—Help | ArcGIS Desktop 

The same can be applied using ArcGIS Pro.  You can then setup Windows Task Scheduler to execute the python script at a given interval.

I think variables need to be changed but do not understand a difference between agsService and baseURL.

May I also ask what cwd is?

Thank you.

Keisuke Nozaki‌ the cwd is the path to where the Attachments will be written.  Here is an example of how the variables are set up for an ArcGIS Online hosted feature service.  This will download the data, including attachments, to a feature class called Property_Damage:

It worked, thank you very much.

Is it possible to download my feature service in .csv format?

Anthony Von Moos‌ you can add the script tool to a Model, and then simply append the Table to Table tool at the end to write the feature class table to a CSV.

I need some help with getting this tool to run.  Here is my current setup:  

If I download it as Tabular Data, everything runs smoothly; however, if I try to do this as a Feature Data, I received this error:  

I am using ArcGIS Pro 2.4.0.  

Hervey Lee‌ this will be a little difficult to trouble shoot with the REST service not being externally accessible. 

  • Are you able to download other Map Services from the same instance?
  • What version is this ArcGIS Server instance?

Hi Jake, one quick question, can Waze data be consumed other than GeoEvent?

Thanks!

Xiuju (Judy) Zhou

Systems Consultant

Houston Public Works-IT-GIS

611 Walker

832-395-2670

Xiuju (Judy) Zhou‌ if you have access to their API, you can query it using something such as Python and write the data to a feature class.

Hi Jack,

Thanks for the reply.  I am unable to download off a different map service from the same Map instance (10.5). 

I have also tried it on Host Feature Services from AGO but I receive a different error. 

  • URL is incorrect.  Or, Service is secure, please enter username and password -> I entered my username/password and does not work still. 

I have also tried it on our Feature Instance, which is on 10.5, and it gave me the same error as my Map Instance.  

Hi Jake Skinner,

Is there a way to have to script start at a specific object ID? I'm downloading a dataset that has 129,420 records, and it got to 129,000 and gave an error. I looked through the script to try and get it to start at a specific object ID, but it looks like the way it iterates, there's no simple way to have it just start at a specific ID.

Matthew Bradbury‌ you can specify a where clause within the tool.  Ex:  OBJECTID > 129420

Jake Skinner‌, that was the first thing I did. I tried a few different ways of writing the object ID (OID, OBJECTID, objectID, ObjectID, ect), and none of those worked. It gives this error every time I try to pass it any kind of where clause, even for a field that I know exists in the data: 

Error loading features: RecordSetObject: Cannot open table for Load
Completed script DownloadService...
Failed to execute (DownloadService).
Failed at Tue Nov 5 11:29:46 2019 (Elapsed Time: 3.47 seconds)

I'm trying to see if there's a way to force it to just download the last 420 records in the set, as it downloaded 129k records without an issue. This is an ArcGIS Server service. If you'd like to see what I'm seeing, this is the service URL:

Layer: Geology Polygons (ID: 138) 

Matthew Bradbury‌ I believe there is an issue with the service.  If you go to the REST service URL:

https://spatialservices.conservation.ca.gov/arcgis/rest/services/CGS/QuaternarySurficialDepositsSout...

And try to execute the query OBJECTID > 129000 it will fail:

It appears to be something with the geometry.  If you choose not to return the geometry the query is successful.

That's quite a strange error; I don't think I've ever seen this one before. I guess I'll try to finish out the remaining records manually. I can see them in ArcGIS Pro, so I know that the geometry exists (at least for most of the polygons that are missing. I wonder if there is a record that doesn't have any geometry that's throwing this error. Thanks for the script, Jake Skinner. It seem to work well provided the service doesn't have any issues.

Hi Jake Skinner‌ and Adam Eversole‌,

I've using ArcGIS Pro's  Feature Class to Feature Class tool: that you mention above in a script tool for other staff to use and it has been working great to download hosted AGOL feature layer data. However today I noticed that even though the Feature Class to Feature Class—Conversion toolbox | Documentation  says it works with Basic, Standard and Advanced I'm getting a warning that the tool completes but with errors.  The error is 001102: You must have at least the standard license to run this tool with the specified data. Apparently features with attachments must have a license level other than ArcGIS Desktop Basic to be copied. Ugh I had been working with an advanced ArcGIS Pro license but I'm glad I switched to basic to see this error since the script tool/workflow process I'm working on is for staff that use the basic version. I'm frustrated to find this error and wonder why a person has to have a standard license or higher to download attachments?

Also just for clarification, the feature class to feature class tool does not allow hosted AGOL feature layer downloads in ArcMap correct only thru your tool Jake? 

Thanks for your help,

Kathy

Kathy Smikrud‌ a Standard license is required to enable attachments on a feature class.  See the licensing information for the Enable Attachments tool:

Enable Attachments—Data Management toolbox | Documentation 

Also just for clarification, the feature class to feature class tool does not allow hosted AGOL feature layer downloads in ArcMap correct only thru your tool Jake? 

Yes, that is correct.

Hey Jake, I've had mixed results with both your script and James Tedrick‌'s...Currently I am trying to run it on a hosted feature layer. Back story: I have a survey built with a hosted feature layer and a related table; the related table has attachments with photos. Your tool will download each individually (which, if you ask me, the next enhancement should be the ability to download both at the same time), but I cannot get it to download the attachments. My latest attempt resulted in this error:

Traceback (most recent call last):
  File "D:\GIS_Workspace\EBA\GIS_Library\Toolboxes\DownloadServicewithAttachments.py", line 661, in <module>
    if hasrow == True:
NameError: name 'hasrow' is not defined

Now, I was able to retrieve the pictures as part of a two step manual process using the MapPhotos toolbox that I believe James created, but I had to first export then download the hosted feature data as a geodatabase.That's not super ideal and I'd rather get yours to run without having to constantly download a file geodatabase offline. Thoughts?

FWIW, we opted to implement James' solution and have it running on a Windows Server 2012 via a scheduled task.  Not sure if Jake's is the same but I liked the idea of appending to the existing FGDB originally downloaded rather than creating a new copy.  This is mostly because we are moving features down from an ESRI tracker service to archive prior to any purge.

From what I recall, I had to work the .py source just a bit to make it work (nothing too crazy) but may be an issue if you're without any python dev support.  One big mistake I made initially was to name the FC's with an underscore "_" in them, which this caused most of the failure I had been having with it.  Once I did that it all worked pretty smoothly.

John Watson‌ do you have ArcGIS Pro installed?  You can use the Table to Table tool to download the related table.  This tool will download the attachments as well.

I'll have to check, trying to run the tool remotely on a client's machine with their data. I would assume they do have Pro, but I can definitely test it locally. Thanks for the quick reply....

Hi Jake, thanks for share your tool. I was learning more python just inspecting it. Im curious about this:

Which WhereClause is faster to load features:

   a)  where = "{0} IN ({1})".format(OID, ids)

   b)  where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)

   c) Make a list of OID's and put it directly in the URL query like this: "......./query?objectIds={[OID's List]}&returnGeometry=true...

What are the differences betwen the these? In which situations use them? Cold you tell me where can i find documentation about this?

Thank you for your time.

Regards

Juan Julio

Hi Jake, 

Thank you for the tool. I amusing it for some time. And now the Fc to Fc tool in the ArcPro does the job as expected. 

I was trying to use the tool to download the records from the Tracker for ArcGIS service that is in use for some time. The service by default have maxRecordCount value set at 28800. The service have more records, but when either of the two tools used,  the resulting FC will only contain 28800 records equivalent to maxRecordCount limit.  I believe the tool works as expected for other services. Does the Location Tracking service have different configueration? If so Is there a way to overcome this issue?

Thanks, 

A

Many thanks for this tool. It has saved me much time that would have otherwise been spent comparing multiple browser windows to answer questions that the Map App authors didn't anticipate. (Are polygons X & Y from 2013 or 2015? click, change layers, click, change layers, click, ...) 

Also note that even though FeatureClass to FeatureClass in ArcGIS Pro in can also convert services to local data, this tool will retrieve features from services that fc2fc fails with. So keep this one in your toolbox stash too!

The script works great when downloading a hosted feature layer from AGOL to a file geodatabase in ArcPro.  However, when I change the Output Feature Class/Table parameter to an enterprise geodatabase I get the error:

When I choose the output feature class it automatically adds [Database name].SDE in front of the feature class name:

Is there a way to get this to work with an enterprise geodatabase?

thanks

Jamie Leitch‌ instead of using this tool, you can simply use ArcGIS Pro's Feature Class to Feature Class tool.

Version history
Revision #:
1 of 1
Last update:
‎07-17-2015 06:12 AM
Updated by:
 
Contributors