Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

73686
501
07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

Attachments
Comments

The attachments are written to disk temporarily and then loaded into the Attachments table (which is stored in the Geodatabase).  The attachments can be deleted from disk after the tool is executed.

I tried with specified outputFC, but failed:

outputFC = r"C:\Users\e124802\AppData\Roaming\ESRI\Desktop10.5\ArcCatalog\GIMSDB_COHSDE.sde\test_login 

the program tried to save data to outputFC = r"C:\Users\e124802\AppData\Roaming\ESRI\Desktop10.5\ArcCatalog\GIMSDB_COHSDE.sde\sde\test_login 

why program added \sde subfolder to it?

Jake,

I tried to output to SDE enterprise database with specified outputFC, but failed:

  • outputFC = r"C:\Users\e124802\AppData\Roaming\ESRI\Desktop10.5\ArcCatalog\GIMSDB_COHSDE.sde\test_login
  • the program tried to save data to outputFC = r"C:\Users\e124802\AppData\Roaming\ESRI\Desktop10.5\ArcCatalog\GIMSDB_COHSDE.sde\sde\test_login

why program added \sde subfolder to outputFC?

Xiuju (Judy) Zhou‌ not sure why this is occurring, but try the following for the outputFC variable instead:

r"Database Connections\GIMSDB_COHSDE.sde\test_login"

Jake, yes, I changed outputFC, but get warning and no data are downloaded to database. could you advise?

baseURL = "https://services1.arcgis.com/XpTq0H1nJ5DHgofs/ArcGIS/rest/services/service_2d983d48b0c8435897ca1481f..."

outputFC = r"Database Connections\GIMSDB_COHSDE.sde\test_login"

--------------------------

Xiuju (Judy) Zhou‌ under the first line in the python script, add the following:

# Disable warnings
requests.packages.urllib3.disable_warnings()

Thanks Jack! Do you think if I disable the warning, data will be saved to SDE? Because data was not able to save after the warning. I am not sure Saving data is affected by warning message.

Thanks!

Xiuju (Judy) Zhou

Systems Consultant

Houston Public Works-IT-GIS

611 Walker

832-395-2670

Yes, it should.  If it does not, then it's most likely an enterprise geodatabase permissions issue.  One way to trouble shoot this is to execute the same script but output the feature class to a File Geodatabase.

I still can’t do it. The warning message is disabled. But data is not saved to SDE and PDF file is not saved without error message.

Hi Jake,

I still can’t do it. The warning message is disabled. But data is still not saved to SDE and PDF file is not saved without error message. is any successful case for SDE database?

Xiuju (Judy) Zhou‌ I believe I figured out the issue.  When running this script outside of ArcGIS Desktop/Pro you will need to specify the full feature class name.

SQL Server:

<database name>.<owner>.<feature class>

Oracle:

<owner>.<feature class>

Here are the parameters I was able to get to work.  Take note of the outputFC:

THANK you soooooooooooooooo much Jack! it is successfully! that is what I need finally! I really appreciate your great help!

Jake,

Thanks for the tool.  It works great to download individual hosted layers, but can it also be used to download the entire hosted feature as a GDB, similar to the functionality of Export in AGO?

Larry Jahn‌ no this will not download a File Geodatabase.  There is a sample on how to do this here. It is the Backup Hosted Services sample.  These samples leverage the ArcGIS API for Python, which requires Python 3.x.  Python 3.x is downloaded on any machine that has ArcGIS Pro installed.

Thanks for the quick response Jake.  I'll try that.

Would I be able to copy an existing feature service in AGOL, to another (new) feature service using this?

joe rodmey‌ you can use a combination of this tool and the following tool:

https://community.esri.com/docs/DOC-11621-update-hosted-feature-service-with-feature-class 

Anonymous User

Hi Jake,

I have used this tool successfully in the past but today I ran into this problem, any idea how to fix this:

Error: AttributeError: 'NoneType' object has no attribute 'utf_8_decode'

Thanks in advance

PJ

Pankaj Jamwal‌ can you share the service to an ArcGIS Online Group and invite my user account (jskinner_CountySandbox)?

Anonymous User

Thanks for your quick reply. I just shared the service with your account.

Pankaj Jamwal‌ try executing the tool again.  I was unable to reproduce this:

Anonymous User

Thanks. Yes I didn't get this error but got another error. Just wondering if you made some changes to fix attribute 'NoneType' error.

Any idea how to fix this new problem?

Thanks for your help

Hi, I am using 10.1 and if I download the zip-file, the toolbox is empty and the script not associated to the tbx. What am I doing wrong?

Stephanie Wegscheider‌ you will need a later release of ArcGIS Desktop.  10.3. is the oldest version you can use with this tool.

This is highly appreciated. I was able to download GIS data. However, there' s small change to piece of code if we're downloading Table.

if agsTable == 'true':
......
# arcpy.AddMessage('Copying features with ObjectIDs from ' + str(data['objects']) + ' to ' + str(data['objects']))
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))


Thanks again.

Meghan Kulkarni‌ thank you for finding this.  I went ahead and made this change and updated the tool.

You're welcome Jake.

Jake Skinner‌, do you have any additional info you can share about the following error I receive? You replied to an earlier comment about this error.

line 670, in <module>
if hasrow == True:
NameError: name 'hasrow' is not defined

Is it something I can avoid? Your script was working well until I ran it this morning and received the "hasrow is not defined" error for the first time. I'm trying to figure out if there was something I tweaked with the feature service that caused the bug to appear? As you mentioned in your earlier reply, the bug only occurs when including attachments in the export process. Otherwise the script still runs well. 

Jake Skinner‌, I spoke too soon. I discovered the error occurred because I had attachments on my feature service but deleted them. So, when the script ran this morning, there were no attachments to export and caused the script to fail. I re-ran the script after making sure there was an attachment in the service and it is now running smoothly again.

I'm running into issues with downloading feature services with more than 1000 records. The issue seems to occur when its appending (it only copies 1000 records at a time). Has anyone else had this issue and found a workaround? 

Hello David,

Does it work with access controlled Feature Service?

Melissa Grove‌ can you share the service to an ArcGIS Online Group and invite my user account (jskinner_CountySandbox)?  I can take a look and see what's going on.

Hi Jake Skinner‌, this may not be the ideal solution, but I was able to bypass this issue by treating every service like there were less than 1,000 records, and it works now. I changed line 215 to < 1,000,000,000 instead of less than 1,000. The only issue I am still having is the global id's are not preserved, which means the attribute table has nothing to relate too. Any ideas?

Melissa Grove One way to workaround this issue is to enable attachments on the feature service, then add an attachment.  When downloading the service, choose the option Get Attachments.  This will create a field in your feature class called GlobalID_Str that will be the original GlobalIDs.

Mmm probably not.  I mostly use it to download municipal data that is publicly available, but might not have the export option enabled.

Hi Jake,

I found this script that you created. I have searched in the web and this is the one that will provide me what I need. I tried using the toolbox in both ArcGIS Pro and ArcGIS desktop but I get the error message below in both. The data is in our Portal for ArcGIS environment and have admin privileges.

Start Time: Monday, 14 January 2019 2:29:48 PM

Running script Download Service...

Generating Token

URL is incorrect. Or, Service is secure, please enter username and password.

Completed script Download Service...

Failed to execute (DownloadService).

Failed at Monday, 14 January 2019 2:29:49 PM (Elapsed Time: 0.54 seconds)

Below are the parameters in both ArcGIS Pro & desktop:.

In ArcGIS Pro

ArcGIS Pro

In desltop

It would be great if you can help me. Thanks.

Dionisio

Dionisio battad‌ it looks like your portal is using another form of authentication other than IWA (Integrated Windows Authentication) and built-in.  Possibly ADFS?  Currently, IWA and built-in security are the only two supported for this tool.

Hello, 

When we run this script we get the following error message

I have attached below the setting entered.

We are only using arcgis online, and have shared this to public.  Are there settings in arcgis online that could prevent the token from being generated.

Thanks

Rory Bennison

Hi Rory,  The Feature Class to Feature class tool in ArcGIS Pro does a good job in copying your Hosted Service into a File Geodatabase.

Hi Melissa, Please! DO NOT EDIT!  your service data In ArcGIS Pro if you have a global ID containing {00000000-0000-0000-0000-000000000000} your data will be damaged.

We have a bug logged in support for this issue, BUG-000119312 - Editing a single feature from a hosted feature layer in ArcGIS Pro edits all the features instead when a GolbalID is duplicated.

- Pro edits the feature based on the globalID

Your Global ID data should never contain multiple {00000000-0000-0000-0000-000000000000} values, If you are running into this or can explain how the data got this way, I would strongly urge you to Contact Esri Technical Support and let us know whats going on.

Regards

- Adam

Esri Support Services

Thanks Adam, have tried that and it worked perfectly.

Cheers

Rory Bennison‌ your missing the index in the url (i.e. TwoStageTest/FeatureServer/0).  One advantage of this tool is that maintain any domains that your current feature service is using.  I don't believe the Feature Class to Feature Class will do this.

Jake Skinner

Hey Jake-

I'm trying to run the tool (Packaged with ArcGIS Pro 2.3, not yours ) and it fails with error 000210: Cannot create output .—Help | ArcGIS Desktop 

Which is pretty vague at best- I'm writing the output to a brand new fgdb on my c drive. Do you have any experience with the tool?  It's hitting a 10.6.1 server instance.  We have a python script that we use to download features via the url, but it has stopped working since our upgrade.  I have TEch Support attention on that issue, and we found this post as a potential fix...

Joe Borgione‌ can you share the URL that you are trying to download?  You can also add the URL to AGOL, share it to a group, and then invite my user account (jskinner_CountySandbox) to the Group.

Jake Skinner‌  Sorry it's taken me this long to reply.  Here is what I got to work:  Add the service to an ArcGIS Pro 2.3 session; open the tool and instead of adding the actual URL for the service, I add the  layer name as shown in the TOC as the input.  Works just fine!

I'm getting this issue trying to use the tool.  I've copied the same string from the Service URL parameter to the Feature Class to Feature Class tool and it works there.  Any ideas?

Brian Berquist‌ try adding the service to your Pro project by going to Add Data and browsing to Portal > My Content, or click drop down arrow and choose Data From Path:

After the service is added to Pro, try executing the Feature Class to Feature Class tool and choose the newly added layer as the input.

I have an enterprise logon, which I believe is causing a problem with being able to authenticate into the account.  Any idea on the correct input for the username?

from my profile in ArcGIS Online, my username is spelled out as:

username@xxxxxxx.net_TownofSomewhere

Richard Freer‌ it looks like you are using SAML, such as ADFS, for your ArcGIS Online authentication.  Currently, this tool does not support SAML.  Do you have access to ArcGIS Pro?  You can try using the Feature Class to Feature Class tool in Pro and see if that works.

Version history
Revision #:
1 of 1
Last update:
‎07-17-2015 06:12 AM
Updated by:
 
Contributors