Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

52351
500
07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

Attachments
Comments

Hi Jake Skinner ,

I've linked many times to your toolbox since it provides a great solution for downloading data from ArcGIS for Server and/or ArcGIS Online.

I see it is also available on Search ArcScripts (BETA) , however the Language (Python) and Software (ArcGIS for Desktop) are not specified:

Is it possible to correct this? 

Hi Xander,

I'll look into updating this.

Hi Jake Skinner

i tried your amazing tool it's work properly and i think its a very useful tool but i have a problem with it, sometime when i use it to download a map service that contains an Arabic text

Could you please manage the code to be working fine with Arabic text also

Thanks in advance

Rawan

Hi Jake Skinner​, the entire discussion on the Arabic signs that rawan saleh refers to, can be found in this thread: Is there a way to download web map service published by ArcGIS Server?

I'm new to working with ArcGIS Online and Server. Does this tool allow you to download data that is served up by WMS connections I have set up in ArcCatalog or is it only the data you access in ArcMap from the Add Data/Add Data From ArcGIS Online option?

Thanks for your help.

What should I fill in at Service URL?  I tried the rest url of my hosted feature layer but after much trying I can't seem to contact my agol

I can't get past this:

nogo.PNG

I'm sure the username and password are ok as I use the same to log in to agol. I tried and tried using the script as a starting point for my own python scripts and also tried to use example scripts from the arc rest package from github but I could not get any result ...

any help would be appreciated, after two days i'm out of idea's

Hi Bert,

It looks like you are trying to download a tiled service.  This tool only supports hosted feature services in AGOL, or ArcGIS Server map/feature services.

Hello Jake,

You are right, the screendump was one of many attempts but changing that doesn't make any difference:

nogo2.PNG

I tried this on my locally installed arcgis for desktop 10.2.2 and on a 10.3.1 on a citrix virtual desktop but I can see no difference.

I would be happy to write my own python scripts but same issue, I just can't seem to get a connection to my hosted features...

Can someone point me to a step-by-step or tutorial about scripted/automated management of agol? That in the end is our goal here. In the mean time i've found this:

http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/The_ArcGIS_REST_API/02r3000000540000... but it's unclear to me what I need to do.

kind regards, have a nice day.

Bert

Bert,

Specify the index of the layer at the end of your URL.  For example:

http://services2.arcgis.com/hzdfa5jbraKip/arcgis/rest/services/Toekomstbomen/FeatureServer/0

Sorry, same as before with the /0 added

Whatever I seem to do I can't get past this error message.

Thanks for your reaction though, it's much appreciated.

regards,

Bert

Try with the following serivce:

https://services.arcgis.com/dlFJXQQtlWFB4qUk/arcgis/rest/services/PropertyDamage/FeatureServer/0

It's shared with 'Everyone', so you won't need a username/password.  The hosted service also contains attachments.

Success! So something must be wrong in my credentials ... Thanks and I'll investigate from here

agol_feature_OK.PNG

In the mean time I was pointed to: https://ago-assistant.esri.com/ which also privides a lot of information.

Ok, so my arcgis for home use works as expected but our company agol doesn't.

I'm taking this to the helpdesk. Seems odd when I can connect using my home use details to connect and obtain a token but no token when trying to connect to kdhl.maps.arcgis.com We're only talking connection here, getting a token

I am seeing this also.  It is for a 3rd party app though.  If I login use corp account it just keeps showing the login screen over and over.  If I login with a personal account it works fine.  If I login with the personal then switch and login with a corp it works.  If i then shut the app and reopen it goes back to not working.

Also have an issue with the Share package tool not working.  It leads 8 parts then gets a bad token error.  Bring it home and it works fine.

Let me know if you get anywhere on this.

I swear this all started happening when AGO was updated last time.....

Hi Jake Skinner

I get this error when running the script - ImportError: No module named 'urllib2', do you know what the problem could be?

Cheers

Hi Matthew Skewes​,

The urllib2 module is used throughout the script to make URL requests throughout the script.  To troubleshoot, try opening a python IDE (i.e. PythonWin, IDLE) and run the command:

import urllib2

Do you receive an error?

Hi Jake Skinner

I think I get that error because I'm running it from ArcGIS Pro with the new version of Python. I have tried running the tool in ArcGIS 10.1 but when I add the toolbox and open the tool I just get  this

Matthew Skewes

I created a 10.1/10.2 toolbox and copied the script tool there.  Try re-downloading the tool to see if it works in 10.1 for you.

Hi Jake Skinner

That did the trick, and the script was able to pull the project down when the export to FGDB function was failing to do so, big thanks for your help.

Cheers

Hi Doug,

Following advice from the helpdesk in the Netherlands (great people) we recreated all useraccounts. No underscores nor colons anymore. They might have done something else but that's unknown to me. Now I can connect and get a token using python without problem.

Our usernames are just our email addresses so no colons or anything either.  I did create a new user with no @ for our 3rd party developer to test with and he is having the same issue.  Can not figure it out so far.  His app works great with his company but not ours.  We can give credentials to the login box with no errors or anything. But then when we try to access anything it brings up the login again.  Personal account on the same device works fine.  Really weird.  With 107 accounts not looking forward to recreating them all.

This weekend I found out the script throws an error when the layer has no objects:

This error disappears as soon as there is at least one object in the layer. I have added an extra try - except:

Capture.PNG

Hi Jake Skinner

I'm also running into an issue establishing a connection to the hosted feature service:

Traceback (most recent call last):

  File "C:\ArcREST\Download Service\Download Service with Attachments.py", line 59, in <module>

    response = urllib2.urlopen(req)

  File "C:\Python27\ArcGIS10.3\Lib\urllib2.py", line 127, in urlopen

    return _opener.open(url, data, timeout)

  File "C:\Python27\ArcGIS10.3\Lib\urllib2.py", line 404, in open

    response = self._open(req, data)

  File "C:\Python27\ArcGIS10.3\Lib\urllib2.py", line 422, in _open

    '_open', req)

  File "C:\Python27\ArcGIS10.3\Lib\urllib2.py", line 382, in _call_chain

    result = func(*args)

  File "C:\Python27\ArcGIS10.3\Lib\urllib2.py", line 1222, in https_open

    return self.do_open(httplib.HTTPSConnection, req)

  File "C:\Python27\ArcGIS10.3\Lib\urllib2.py", line 1184, in do_open

    raise URLError(err)

URLError: <urlopen error [Errno 8] _ssl.c:510: EOF occurred in violation of protocol>


Failed to execute (DownloadService)

Any idea what the issue could be here?

Thanks

Hi Vlad Romanescu​,

Can you provide a screen shot of how you have the tool setup?

Certainly, I've attached a screenshot of the tool dialogue:

The feature service URL is at https://services6.arcgis.com/FAgEZRLybpkpIX4O/arcgis/rest/services/REQUESTS_FOR_SERVICE/FeatureServe... (shared with the organization, not the public).

Thanks

Vlad, I sent you an invitation to an AGOL group called 'User Data'.  Can you share the service with this Group so I can test trying to download it?

Jake, I've shared the service with the User Data group, you should be able to see it

Vlad, I could not reproduce this.  I could not tell from the screen shot, but make sure that you have the layer specified in the URL by adding '/0' at the end.  Ex:

Yes, the service URL has a '/0' at the end. Still the same error message. Could it be something in my organization's firewall blocking access when ArcGIS for Desktop tries to go to the feature service URL?

Thanks for your help.

Vlad 

Hi Jake,

Thanks for a great tool. I have it working just fine on a "standard" FS but when I have related tables it only brings down the top layer /0. Sure I am using "...FeatureServer/0" but I have tried a few things including using /1 or /3 but no joy?Should the tool work for these related tables i.e am I being stupid?

I swear I posted this already not sure where it went.  You could also try the ArcRest tools to do it.  You can specify the layers you want (0,1,2,3 etc).  Note i did have an issue when i put in too many layers it died.  I have an open ticket for this.

Here is the code I use.  Note you must install ArcRest.  Also note this just changed due to ArcRest changes.

from arcrest.security import AGOLTokenSecurityHandler
from arcrest.agol import FeatureService
from arcrest.common.filters import LayerDefinitionFilter

username = "put here"
password = "put here"
url = "put here"
outDir = r"put here"


#Get a local replica of the HFS

# Set layers list to what you want
proxy_port = None
proxy_url = None
agolSH = AGOLTokenSecurityHandler(username=username,password=password)
fs = FeatureService(url=url, securityHandler=agolSH, proxy_port=proxy_port, proxy_url=proxy_url, initialize=True)
result = fs.createReplica(replicaName='Backup', layers=[0,1,2,3,4], dataFormat='filegdb', out_path=outDir, wait=False)

# can look at this result and see if it worked

Hi CableClear,

dougbrowning‌ is correct that you can do this using ArcREST.  ArcREST will use the 'CreateReplica' command to create a File Geodatabase of the data.  However, SYNC needs to be enabled on the feature service in order to use the CreateReplica command.

I went ahead and updated the tool so that you can export data from hosted feature service tables.  So if your related data is in a table, you can now download this.  Just make sure you have 'Downloading Tabular Data' checked.  Also, it will not maintain the relationship class.  You will have to recreate this.

Hello Jake, Doug, Many thanks for taking a look.

Jake I have downloaded (v14) of your tool  and tried but I seem to get errors trying it on a dataset which has one Feature Class and  two related table. and which are joined with  a Relationship Class on BayID.

I have uploaded a sample reduced dataset and made it public so no username/PW.There are two layers /0 (bays) and /1(customers).

https://services7.arcgis.com/uQ6sp1S1EdnR7QVS/arcgis/rest/services/DownloadServiceTest/FeatureServer

Be very appreciative if you could take a look.

Cheers, Graham

Hi GrahamMi‌,

When you download the related table do you have 'Tabular Data' checked.  I was able to download both the feature class and table successfully.  You will need to recreate the relationship class, but the data downloaded successfully.  Below is how I had the tool set up:

Jake,

Neat tool! I'm able to get the features, but am having issues with the attachments. See error below.

mimiller‌,

Can you share the service to a Group and invite my ArcGIS Online account to this group?  My username is jskinner_CountySandbox.  I can test downloading the service with attachments.

Hi Jake,

Thanks so much for testing. 

I download the Service to C:\Test , extract the zip file and from inside ArcMap v 10.4.1 run the script from this shared folder.

When I start your Download Service toolbox it looks like this.

However as soon as I tick "Hosted Service layer" the "Download Feature Data" and "Download Tabular Data" grays out. I am then unable to tick the box.

I have found that if I tick the "Download tabular data"  first (before anything else) I can tick it but then as soon as I tick "Hosted Feature Service" it grays out again.

Possible a version issue in ArcMap? Or me being stupid?

Hope thsi helps and you can see the issue. Really need to get his working after spending so much time setting up a  related tables solution. Will massively appreciate your help.

Many thanks Jake.

Hi Jake,

This should blow your mind.

Messing around I tried using the "ArcGIS Server Service" even though it is hosted on AGOL. I get none of the greying out issues as post above, and it seems to work perfectly! I get the Customer data out fine.

If I then try and download ...FeatureServer/0 using "ArcGIS Server Service" and selecting "Downloading Feature Data" it also seems to works fine!

I checked again downloading ...FeatureServer/0 using "ArcGIS Feature Service" and it also works fine!

Too much for my little mind!

Graham Misselhorn‌, if you downloaded the latest version of the tool, you should have the option to check 'Downloading Feature/Tabular Data' when 'Hosted Feature Service' is checked:

Jake,

I sent you an invite to this group: http://iplan.maps.arcgis.com/home/group.html?id=4cdf9a4328404f8d868d6bf7ca703eb2 . Only one feature layer in the group.

Thanks!

Hi, 

Seem to be a great tool, but unfortunately, it doesn't work for me with Arcgis Pro 1.3

here is the message error i get even with my FL or a public FL

Any ideas?

Thank you

Michael Miller‌,

I believe I found the issue.  The OBJECTID field in AGOL is OBJECTID_1.  When you download the feature service to a feature class, the OBJECTID_1 field is converted to OBJECTID.  This was causing the script to fail.  Open the script in a text editor and you can the following line before line 329:

OID = "OBJECTID"

Ex:

lodecombe_DirEtudes‌,

Unfortunately, this tool will not work in ArcGIS Pro.  Pro uses Python 3.4, while ArcGIS Desktop uses Python 2.7.  This tool was written using Python 2.7.

Thanks Jake! Really appreciate sleuthing it out and sharing the solution.

Hi Jake

 

works perfectly with Arcmap. thanks.

However, is it possible to adapt the script because it seems that we have some proxy issues with our company. Work fine with personal internet connexion but not with company connexion.

Thanks

NB: i've used also Show Attachments in Web Map Popup. work fine at home but not in my company. For sure, it's a proxy connexion issues. Any idea to resolve it? Where to put the proxy adress?

Hi Jake,

I have multiple attachments/photos with the majority of points in a feature class, but only one image attachment is downloading with the script.  Is there a way to enable all attachments to download?  I've tried multiple other suggested options such as creating a replica via Rest/JSON, but the service continually fails to download.

Thanks,

Geoff

Have you tried the ArcRest method I posted?  If so when you look at the result out it will tell you the error it is getting.  Also make sure Sync is enabled on the HFS.

Thanks.  Sync is enabled on the feature service.  Just getting a generic error of "Exporting data for layer 0 failed." in the Rest web interface.  I also tried installing ArcRest and using the code you provided above, adding in the attachment parameters and still no luck.  The error code is the same and it says please check parameters:

{u'error': {u'message': u'Unable to create replica. Please check your parameters.', u'code': 500, u'details': [u'Exporting data for layer 0 failed.']}}

I also tried this github sample: ArcREST/create_replica_fs.py at master · Esri/ArcREST · GitHub with no luck. In both cases I tried just creating a replica of the features with no attachments with no luck.  I feel like somehow my feature service has become corrupted.  In both cases the feature service has been successfully referenced when the script runs, so no issues with authentication.

Geoff

Do you have a field called Date?  That is reserved and was my issue when that happened.

gschwitz‌, would you be able to create a Group in AGOL, share the service with this group and then invite my user account (jskinner_CountySandbox)?  I can take a look to see what's going on.

Version history
Revision #:
1 of 1
Last update:
‎07-17-2015 06:12 AM
Updated by:
 
Contributors