Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

[#1]

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

[GeoffSchwitzgebel1]

Doug,

I'm not using any reserved words as field names.

Jake,

I had a support request opened with syncing issues in collector, but I think this is all related.  The case # is 01845850 if that helps you at all.  The support analyst is currently a member of the group.  I just added you to the group too.  The feature service is USC_Backflow_Valves.

[JakeSkinner]

gschwitz‌ I update the tool where it will download multiple attachments now.

[GeoffSchwitzgebel1]

Thanks!  That was a big help.  Is there anyway to include the ability to export any attribute domains within the feature service?

[DaveTenney]

running into SSL issue with this tool, see image below.

running 10.2.2

Dave

[JakeSkinner]

dtenney‌ when you enter the REST URL to the service in a web browser, do you receive a warning that the web site is not trusted?

[DaveTenney]

i do not, it is however a secured service so i am prompted to enter my un/pw, which is the same as i enter into the tool.

      are there any known issues with secured services or any known issues with Desktop Version issues?

thanks

Dave

[DaveTenney]

Finally got it to work...

   one question...

We have a service that is published to arcgis server.

   1) view is created in arcgis desktop from a sql database connection

   2) view shows QC errors that are populated within the sql database as client updates data

   3) the created view is then published to arcgis server for the client to view.

      our current test service only has 7 features in it and the download service tool took almost 6minutes to download those 7 features. is it normal for it to take that long? our real world service will have 100s if not 1000s of features, see the problem?

thanks

dave

[JakeSkinner]

Dave Tenney‌, it should not take that long.  Couple things to try:

1. Does downloading a service based on a feature class, rather than a view, take as long?

2. Try downloading a service from another resource, i.e.:

https://sampleserver6.arcgisonline.com/arcgis/rest/services 

[DaveTenney]

feature class (countywide roads) downloads very fast. so it seems that the tool may not like the view that is published.

  thanks for your help

dave

[rawansaleh1]

Hi All,

Could i use this tool to extract an orthophoto services?

Best

Rawan

[DaveTenney]

Generating Token

URL is incorrect. Or, Service is secure, please enter username and password.

Completed script DownloadService...
Failed to execute (DownloadService).
Failed at Wed Dec 07 14:15:53 2016 (Elapsed Time: 4.24 seconds)

im not really sure why its saying the URL is incorrect, the service is secure and i provided the appropriate un/pw.

dave

[JakeSkinner]

RawansSaleh‌ this tool will only work with map/feature services.

[DaveTenney]

jskinner-esristaff‌

   it appears to be a public vs secure service issue. 

i briefly made my service public and the tool ran great. the minute i returned it back to private the tool fails...any thooughts?

im am sure that i am giving the correct un/pw

dave

[JakeSkinner]

Dave Tenney‌ how do you have the service secured, using Active Directory or ArcGIS Server Built-In?  If you are using Active Directory, do you know if the authentication tier is at the Web Server or GIS Server?  It may be helpful to include a screen shot of ArcGIS Server Manager > Security tab.  Ex:

[DaveTenney]

we are using the server built-in security...

[JakeSkinner]

dtenney‌ I would need to take a look at the service URL.  If you would like, you can send me a personal message with the URL and a username/password I could use.

[DaveTenney]

sure thing

[PatrickMcKinney1]

I encountered an issue where the tool doesn't appear to work on Map Services from ArcGIS Server with over 100,000 records.  The tool copied features up to 100,000, but then failed.  I'm using a version of the tool I downloaded on July 20, 2015 (not sure if there have been changes since).

Error message:

Traceback (most recent call last):
File "F:\Scripts\ArcGIS Desktop\Export Feature Service\Download Service with Attachments.py", line 170, in <module>
fs.load(fsURL)
File "c:\program files (x86)\arcgis\desktop10.3\arcpy\arcpy\arcobjects\arcobjects.py", line 175, in load
return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args)))
RuntimeError: RecordSetObject: Cannot open table for Load

Failed to execute (DownloadService).
Failed at Fri Dec 16 13:53:27 2016 (Elapsed Time: 17 minutes 39 seconds)

[PankajJamwal1]

Hi Jake,

Thanks for this tool. I was able to use this tool but I am not able to download the Subtypes from the feature services. I have 6 subtypes (Taxon- Herp, Bird, Plant, Invert, Mammals, Fish) in GDB but after using this tool I got integer instead of Taxon names as shown below.

Any help would be highly appreciated.

[JakeSkinner]

pmckinneyccpa‌ can you share the service to a AGOL Group and invite me to the group to take a look.  My AGOL username is jskinner_CountySandbox.

[JakeSkinner]

pankajjamwal‌ does not currently support downloading domains or subtypes.  You will need to recreate the subtypes for the feature class.

[PatrickMcKinney1]

I sent you an invite. To provide some background, our regional public safety Task Force is looking into updating our regional datasets from each county using web services.  This tool is very promising if we can get all records from services.

Thanks for looking into it,

Patrick

[JakeSkinner]

Patrick McKinney‌ I believe this may be something specific with the service.  I tested your service and the features stop downloading after 100,000, but I don't receive an error.  I tested with a service (100,635 features) I published to ArcGIS Server and was able to successfully download it.  I am also using ArcGIS Server 10.5, and I noticed you are using 10.3.1.  If you would like to try the data I used, you can download it from here.

[PatrickMcKinney1]

Thanks for looking into it jskinner-esristaff‌.  I'll test with some other services.  I believe we will be upgrading to 10.5 after the first service pack is released.

[DougBrowning]

FYI on this script from the Survey123 guys.  It syncs to SQL.

https://community.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-s... 

[NickHarvey]

Hi Louis - If it helps I started testing a bare-bones method below (ArcGIS Pro 1.4) to download simple AGS FS records. I was anxious to see the process work with Pro/Python 3.4 because this is how we update many of our regional base layers from partnering agencies.  As our test case was simple, the code below accounts for none of the error checking, case types, or number-of-records-handling that Jake's tool provides (which we use in production).

*Note: Even when the source AGS services' max record count is set above 1k I found that the process can bomb depending upon the combination of number of features, number of fields, etc. In my case the largest number of features I downloaded in one go was 10k parcels. Its likely that you'll have to account for a > 1k record set somehow (Jakes' tool accounts for this). But in one test I simply split the records: One url for the 1st half, One url for the 2nd half of the records...merged after.

I tried it on the service below and it appears to work. Click supported operations:'query' at bottom of page to ui-build json query (url).

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/Landscape_Trees/FeatureServer/0

In Pro python window (below):

localfc = r'C:\LocalFolder\Local.gdb\wwctrees'
url = 'Insert your resulting url from json format Query (GET) here'
fs = arcpy.FeatureSet()
fs.load(url)
arcpy.CopyFeatures_management(fs, localfc)

HTH

-Nick

[EricAdolfson]

Hi Jake, I appear to be having the exact same issue as Dave Tenney. The tool works if I turn off security using ArcGIS Server security. Once I use security I get the message: URL is incorrect. Or, Service is secure, please enter username and password. Were you able to resolve the issue that Dave Tenney had??

Thanks, 

Eric

[JakeSkinner]

EA1‌, if you are using Web Tier authentication you will need to bypass your web adaptor to download the service.  Ex:

https://ags1.esri.com:6443/arcgis/rest/services/Parcels/MapServer/0

[WesleyKoros]

Hi Jake

I've just tried to using this awesome tool, how can I handle proxy connexion issue. Kindly help

Wesley

[JakeSkinner]

Wesley Koros‌ you may want to take a look at the following:

Using an HTTP PROXY - Python - Stack Overflow 

[WesleyKoros]

Thanks Jake, will have a look at it tomorrow and let you know if outcome. Cheers!

Sent from my iPhone

[WesleyKoros]

Hi Jake,

I tried fixing the proxy but getting the following attached error:

Kindly advise here:

Wesley

[WesleyKoros]

Hi Jake,

Awesome! The tool works fine when i use a desktop with version 10.3

installed , but for version 10.4 it does not work, i get the following

error:

Is the tool constrained to version 10.3 and below?

Wesley

[JakeSkinner]

Wesley Koros‌  I have successfully run this tool at 10.2, 10.3, 10.4, and 10.5.  Were you on a different server when you ran it from 10.3?  May want to compare the LAN settings for each server by going to Start > Internet Options > Connections tab > LAN settings.

[MonaSabbah]

Hi Jake, this is what I get when I try to run the tool! I am new to GIS and I have no idea what that means, also I had my GIS teacher take a look at it, and he didn't know what the case was either.  It doesn't say what the error is. 

[JakeSkinner]

sabbahmona‌ I would recommend writing the output to a File Geodatabase rather than a shapefile.  With the number of features this service has, it may exceed the shapefile 2 GB limit.  I was able to get this tool to execute, here is how I had it setup:

[MonaSabbah]

Unfortunately I still have the same problem, and I also have that red mark on top! even before I insert anything I still have that red error mark on top.  

[JakeSkinner]

The only thing I can think of is to try and re-download the tool.  Also, can you try executing the tool on another server? 

[AmandaOpp]

Jake Skinner‌

Hi there Jake,

I have used your tool successfully on several different projects and to say that it is handy is an understatement! I am trying to run it on a new data set and I keep getting the following error. I literally publish in the same way and follow the same work flow as my other data but this one errors out. Would you be able to shine some light as to why? Thank you!

[JakeSkinner]

amandaopp‌ can you share the service to an AGOL Group and invite my username to it:  jskinner_CountySandbox

[AmandaOpp]

jskinner-esristaff‌

You should be able to see that group, I added you automatically so hopefully you can see it.  Let me know if you need anything else.  Thank you!

[JakeSkinner]

Amanda Opp‌ I can't seem to figure out why the tool is failing on this service.  The features download, but it fails on the attachments.  One workaround you can do is add the hosted feature service to ArcMap and create a local copy for editing.

[AmandaOpp]

Jake Skinner<https://community.esri.com/people/jskinner-esristaff?et=notification.mention>

I wasn’t even able to get the features to download so that is progress. I will take another shot at it and let you know if I come up with any solutions. Thank you for your help Jake!

[MonaSabbah]

It finally worked. Your tool is perfect, the problem was with my Arcgis! I just had to repair it in the control panel...
Thanks you're brilliant! 

[AmandaOpp]

Jake Skinner‌

I am trying to use your tool on two sets of data that come from Survey123. Both forms are feature services. I am able to download just the features but get the same error on both for the attachment portion. Would you take a look at it and let me know if there is a fix for this if you have time? I will share it with my test group so you can see it. Thanks!

[JakeSkinner]

amandaopp‌ I was able to get this working.  It appears Survey123 creates the GlobalID field as 'globalid' rather than 'GlobalID'.  I updated the tool to check for this, it should now work.

[AmandaOpp]

I literally tried every version of GLOBALID but that!  Thank you for getting back to me and helping with this.

[DougBrowning]

Have you thought about having the ability to sync the data over to SQL like the Survey123 guys script does?  I tried using that script for other HFS but it fails.  A sync to SQL is more handy then just a download.

https://community.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-s... 

[DougBrowning]

For Survey123 use their sync script.  Sync right to SQL versus the whole thing everytime

https://community.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-s... 

[DougBrowning]

I tried giving a SQL db as the output location.  It does add the records but also said it failed with this error

Error Copying Features: FILE: PATH\Download Service with Attachments.py, LINE: 260
"arcpy.Append_management(fs, outputFC)": ERROR 000466: C:\Users\hidden\AppData\Local\Temp\arcF24\mx987E9D92D4E54932AA9208308B7A60DC.gdb\ScratchRecordSet does not match the schema of target PATH\WorkingGIS.sde\WorkingGIS.DBO.LaGrandeSurvivalTrees
Failed to execute (Append).

Also will it wipe and reload all each time?

Thanks