topic Re: Export data from Map Service in ArcGIS Enterprise Questions

Export data from Map Service

HaniDraidi — Thu, 23 Oct 2014 10:35:52 GMT

Hi,

Is there a tool (in ArcMap) to extract data from a map or feature service I am accessing?

Thank you,

Hani

Re: Export data from Map Service

AlexPerez — Tue, 12 May 2015 23:32:19 GMT

Not specifically no. If you can filter your search so that you want fewer records than the server's record limit, you can query the map service. Open up the map service in a browser. Navigate to the service you want and there should be a QUERY link at the bottom. Fill out the form and request the results in json (if you want everything you can always try WHERE 1=1). Copy paste them into a text file and you can use a JSON to features tool that will build the polygons with attributes for you. It can be time consuming but it is possible.

Re: Export data from Map Service

ModyBuchbinder — Wed, 13 May 2015 05:56:02 GMT

For image service if the download capability is checked (not default) you can use right click, Data, Download selected rasters.

You must select some rasters before the option is enabled.

Re: Export data from Map Service

JakeSkinner — Wed, 13 May 2015 11:57:14 GMT

Hi Hani,

Take a look at the following tool:

http://epro.maps.arcgis.com/home/item.html?id=16e5cc64178941839eca62837f168ec9

Re: Export data from Map Service

NickHarvey — Wed, 08 Jul 2015 18:57:23 GMT

Thanks for posting this tool Jake - REALLY HELPFUL. I was trying to use ArcMap to export 120k+ features from an AGS feature service that one of our partnering agencies had set up....I kept getting a "client timeout/server not responding error". I noticed that when I selected a small number of features the AGS feature service export would complete successfully. I know that on our own AGS feature services that the max number of records returned by server is set at 1000 (default presumably). And from looking at the code you have provided, this tool appears to dice up the download/extract in chunks of 1000 records by ObjectID. Am I connecting the dots correctly here? - That the 1000 record limit is expected per this client/server process and being accounted for in your code? Or, let me know otherwise....

Thanks again!

-Nick

Re: Export data from Map Service

JakeSkinner — Thu, 09 Jul 2015 16:23:02 GMT

Yes, that is correct. The script appends 1000 features at a time.

Re: Export data from Map Service

JoshWhite — Sat, 11 Dec 2021 21:25:45 GMT

Jake Skinner,

I was going to try using your tools. It is a secure service so I entered my credentials. The script acts as if it cannot generate a token. One thing I notice is that when I type in my password and move on to the next box the number of characters change in the password box, I'm hoping it is not defaulting to another password that I cannot see.

Executing: DownloadService false true http://<serviceurl>/countymap/rest/services/Clip_Ship/CL_ParcelData/FeatureServer/8 true false JWhite ***** "Database Connections\Connection to GIS-SERVER.sde\acgis.DBO.Address\acgis.DBO.AddressPoints" false #
Start Time: Thu Sep 24 09:33:31 2015
Running script DownloadService...
Generating Token
　
URL is incorrect.  Or, Service is secure, please enter username and password.
Failed script DownloadService...
Traceback (most recent call last):
  File "C:\Users\Josh White\AppData\Roaming\ESRI\Desktop10.3\ArcToolbox\My Toolboxes\Download Service with Attachments.py", line 53, in <module>
    iteration = int(data['objectIds'][-1])
KeyError: 'objectIds'


Failed to execute (DownloadService).

Re: Export data from Map Service

JakeSkinner — Thu, 24 Sep 2015 16:51:59 GMT

Hey Josh,

I believe I fixed the issue, and updated the tool:

http://countysandbox.maps.arcgis.com/home/item.html?id=2b1f198be4ea44288418fd39f1d0aff0

Re-download the tool and let me know if you run into the same issue.

Thanks.

Re: Export data from Map Service

JoshWhite — Wed, 30 Sep 2015 15:55:48 GMT

It comes back with a bit different error this time:

Traceback (most recent call last):

File "C:\Users\Josh White\AppData\Roaming\ESRI\Desktop10.3\ArcToolbox\My Toolboxes\Download Service with Attachments.py", line 46, in <module>

params = {'where': '1=1', 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}

NameError: name 'token' is not defined

Failed to execute (DownloadService).

I have another question, is the intent to do one layer at a time? If that is the case, I would need the ID number for the layer, is that correct? I've tried it both ways and the error is the same for both methods so it is catching on something else first. It appears it is still "catching" because of the secure service. I'm afraid I don't know enough python to know what I am looking at.

Re: Export data from Map Service

JakeSkinner — Wed, 30 Sep 2015 18:19:50 GMT

Yes, the tool is designed to download a feature layer, so you will need to specify the index of the layer you wish to download.

Try running the tool with the following service URL below. I just tested it, and it worked successfully. Be sure that you download the most recent version of the tool.

http://54.209.148.150:6080/arcgis/rest/services/Voting_Districts/MapServer/0

username: viewer1

password: viewer1

Re: Export data from Map Service

OLANIYANOLAKUNLE — Fri, 16 Oct 2015 23:42:45 GMT

Hi Jake Skinner, I found your tool very useful and helpful but I still get the error below;

iteration = int(data['OBJECTIDS'][-1])

KeyError: 'OBJECTIDS'

See my code block below as well;

username = "ASDERT"
        password = "@######"
        baseURL = 'http://services5.arcgis.com/b23aIR8OqGRlfJlE/arcgis/rest/services/PropertyEnumerationForm/FeatureServer/0'

        arcpy.AddMessage('\nGenerating Token\n')
        tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
        params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
        req = urllib2.Request(tokenURL, urllib.urlencode(params))
        response = urllib2.urlopen(req)
        data = json.load(response)
        token = data['token']

        # Return largest objectIds
        params = {'where': '1=1', 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
        req = urllib2.Request(baseURL, urllib.urlencode(params))
        response = urllib2.urlopen(req)
        data = json.load(response)
        try:
            data['objectIds'].sort()
        except:
            arcpy.AddWarning("\nURL is incorrect. Or, Service is secure, please enter username and password.\n")
        iteration = int(data['OBJECTIDS'][-1])
        minOID = int(data['OBJECTIDS'][0]) - 1
        OID = data['objectIdsFieldName']

Kindly assist me with this, Thanks

Re: Export data from Map Service

JakeSkinner — Mon, 19 Oct 2015 11:42:01 GMT

Hi Olaniyan,

When running the tool on the service, I did not receive any errors. Ex:

Re: Export data from Map Service

OLANIYANOLAKUNLE — Sat, 11 Dec 2021 21:25:47 GMT

Thanks Jake,

I was trying to run the script behind a button, see script below;

import pythonaddins
import os
import datetime
import time
import os, sys
import urllib, urllib2, arcpy, json
import uuid
arcpy.env.overwriteOutput = True
class AGOLClass1(object):
    """Implementation for AGOL_addin.button (Button)"""
    def __init__(self):
        self.enabled = True
        self.checked = False
    def onClick(self):
        username = "TAXXXX"
        password = "#######"
        baseURL = 'http://services5.arcgis.com/b23aIR8OqGRlfJlE/arcgis/rest/services/PropertyEnumerationForm/FeatureServer/0'
         
        
        arcpy.AddMessage('\nGenerating Token\n')
        tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
        params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
        req = urllib2.Request(tokenURL, urllib.urlencode(params))
        response = urllib2.urlopen(req)
        data = json.load(response)
        token = data['token']
        # Return largest objectIds
        params = {'where': '1=1', 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
        req = urllib2.Request(baseURL, urllib.urlencode(params))
        response = urllib2.urlopen(req)
        data = json.load(response)
        try:
            data['objectIds'].sort()
        except:
            arcpy.AddWarning("\nURL is incorrect.  Or, Service is secure, please enter username and password.\n")
        iteration = str(data['objectIds'][-1])
        minOID = str(data['objectIds'][0]) - 1
        OID = data['objectIdsFieldName']

and I get the error below;

>>>

Traceback (most recent call last):

File "C:\Users\KUNLE\AppData\Local\ESRI\Desktop10.2\AssemblyCache\{322E160C-A3DE-48C9-BEBC-86802FD9C785}\AGOL_addin.py", line 40, in onClick

iteration = str(data['objectIds'][-1])

KeyError: 'objectIds'

>>>

But the script runs well as a script tool under a toolbox. Kindly assist, thanks.

Re: Export data from Map Service

TanuHoque — Wed, 08 Apr 2020 15:02:10 GMT

Thanks Jake Skinner‌ for the python script. This is awesome!

Just an FYI for all - in ArcGIS Pro, you can use geoprocessing tools such as 'Feature Class to Feature Class' to export data from a layer off a map service. And you can download all data not just first 1,000 or whatever is MaxRecordCount is set to (as long as the server is not an older version).

Steps

Copy the url of the layer - make sure to the url ends with '/MapServer/<number>' not '/MapServer'
e.g. https://sampleserver6.arcgisonline.com/arcgis/rest/services/SampleWorldCities/MapServer/0
Click Data From Path
Paste the url
OK
Open Feature Class to Feature Class gp tool
Select the layer
Specify output
Run

Re: Export data from Map Service

Esad — Thu, 16 Jul 2020 19:37:25 GMT

I am trying to fetch data from a map server that I do not manage and I am receiving the following error:

Proxy server got bad address from remote server (verify the server is running). (status code 504).