Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

117179
503
07-17-2015 06:12 AM

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

I've come across a lot of users that have requested the ability to download ArcGIS Online Hosted Feature Services and ArcGIS Server Feature/Map Services. See that attached tool to do just this.  The tool will also allow you to download attachments from ArcGIS Online Hosted Feature Services.  Hope users find this helpful and useful.

Update 8/10/17:  Tool will now download geodatabase coded value domains.  Also, consolidatde the 2x and 3x version into one tool.  The tool can be run from either ArcGIS Pro or ArcGIS Desktop.

Update 11/13/17:  Tool now supports using a where clause to query the service.

Update 3/14/17:  Updated to a single script for ArcMap/Pro using the requests module

Update 1/17/19:  Thanks to Adam Eversole‌ for pointing out that this tool's functionality can now be handled by ArcGIS Pro's Feature Class to Feature Class tool:

Attachments
Comments

Jake Skinner‌ Unfortunately I don't have access to ArcGIS outside of our network but I have tested it by reducing parts of the code and it looks like it opens the proxy gets the URL and then next time it goes to see a URL the proxy is closed.  I think I need to find some code that keeps the "session" open.  My python skills are limited, more of a copy/paste and test type of guy

Does this tool work for ArcMap 10.6?

I am getting the error "line 661, in <module>
if hasrow == True:
NameError: name 'hasrow' is not defined"

Brita Austin‌, yes this works with ArcGIS 10.6.  Can you share you the service to an ArcGIS Online Group and invite my user account (jskinner_CountySandbox)?  I can try to download the service and troubleshoot.

I've created the group and shared as you asked. Thank you so much! I really appreciate the help 

Brita Austin‌ this may be a possible bug.  The script calls the CreateReplica function.  When doing this manually by going to:

https://services5.arcgis.com/CvSVG6rs7CUXohUs/arcgis/rest/services/PLANO/FeatureServer/createReplica 

the returned JSON is empty for the features and attachments in the JSON file.  This is only occurring for tables.  I'm checking with the ArcGIS Online team if this is expected, a bug, or a limitation.  I'll let you know ASAP.

This tool is great and really helpful. However, I am having trouble downloading attachments. Not all of my points hosted on ArcGIS online have attachments but with the points containing attachments I would like to download them. When using the tool I check get attachments and have tried several places of storing the photo directory but the attachments are not downloading and the results bar says service has no attachments, despite me checking in ArcGIS online and several points having attachments. I am just wondering if you have tips for how to download the attachments with the points

Brad, I have made a similar tool only for arcgis pro: I have added also spatial filter. You can try see if it download attachments GitHub - nicogis/DownloadService: Download data from services of arcgis server or host service  

hi,

 i am having issues downloading attachments using this tool i can successfully download without attachments. but if i tick the "download Attachments" box i get a warning saying "Service does not have sync operation enabled" it is enabled as i can do editing (including offline) 

the error that follows is:

 urllib.urlretrieve(replicaUrl, cwd + os.sep + 'myLayer.json')
NameError: name 'replicaUrl' is not defined

and if i check the json there is no key called replicaURL 

here is the error message i am getting 

ok so i have figured out part of it ,

the createReplica URL was incorrect, it contained the layer number at the end 

i.e. the end of the url looked like FeatureServer/0createReplica instead of FeatureServer/createReplica

so i changed line 273 from 

crUrl = baseURL[0:-7] + r'createReplica'

to 
crUrl = baseURL[0:-8] + r'createReplica'

to strip the last character and i no longer get that error,

now the script does not throw an error but it exits early and i get a message that the "Service does not contain attachments", which is not true because one feature does 

and this is being caused by the replicaURL 

line 290 - 292 

urllib.urlretrieve(replicaUrl, cwd + os.sep + 'myLayer.json')

f = open(cwd + os.sep + 'myLayer.json')

it is not copying myLayer,json anywhere because the replicaURL is not returning anything

if i copy the replicaURL into a browser i get an error "SSL required"

so does it need the token generated earlier in script to be appended?

i manually tried this by appending the token to the end of the replicaURL but maybe thst not the correct position for the token. in what position within the url should the token be?

Further to the above,

 the SSL error is caused because replica url is using http, i have modified so it uses https

now i need to figure out where to include the token in the url

------------SOLVED ------------------

i had to add the following line 

replicaUrl=("https{}").format(replicaUrl[4:])+"?token={}".format(crValues['token'])

before 
urllib.urlretrieve(replicaUrl, cwd + os.sep + 'myLayer.json')

around the line 293 area

now it downloads attachments. YAY!

Brad Wilson‌ can you share your service to an ArcGIS Online Group and invite my user account (jskinner_CountySandbox)?  I can take a look and see if I can reproduce.

Anthony, could you please share your full updated script? I would love to try it out! 

here is updated script,only slight changes to Jake Skinner's script as documented above 

import arcpy, urllib, urllib2, json, os, math, sys
from arcpy import env
env.overwriteOutput = 1
env.workspace = env.scratchGDB

hostedFeatureService = arcpy.GetParameterAsText(0)
agsService = arcpy.GetParameterAsText(1)

baseURL = arcpy.GetParameterAsText(2) + "/query"

agsFeatures = arcpy.GetParameterAsText(3)
agsTable = arcpy.GetParameterAsText(4)

username = arcpy.GetParameterAsText(5)
password = arcpy.GetParameterAsText(6)

# Generate token for hosted feature service
if hostedFeatureService == 'true':
try:
arcpy.AddMessage('\nGenerating Token\n')
tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken'
params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'}
req = urllib2.Request(tokenURL, urllib.urlencode(params))
response = urllib2.urlopen(req)
data = json.load(response)
token = data['token']
except:
token = ''

# Genereate token for AGS feature service
if agsService == 'true':
try:
arcpy.AddMessage('\nGenerating Token\n')
server = baseURL.split("//")[1].split("/")[0]
tokenURL = 'http://' + server + '/arcgis/tokens/?username=' + username + '&password=' + password + '&referer=http%3A%2F%2F' + server + '&f=json'
req = urllib2.Request(tokenURL)
response = urllib2.urlopen(req)
data = json.load(response)
token = data['token']
except:
token = ''
pass

# Return largest ObjectID
params = {'where': '1=1', 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
req = urllib2.Request(baseURL, urllib.urlencode(params))
response = urllib2.urlopen(req)
data = json.load(response)
try:
data['objectIds'].sort()
except:
arcpy.AddWarning("\nURL is incorrect. Or, Service is secure, please enter username and password.\n")
iteration = int(data['objectIds'][-1])
minOID = int(data['objectIds'][0]) - 1
OID = data['objectIdFieldName']

# Code for downloading hosted feature service
if hostedFeatureService == 'true':
if iteration < 1000:
x = iteration
y = minOID
where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)
fields ='*'

query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query

fs = arcpy.FeatureSet()
fs.load(fsURL)

arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
outputFC = arcpy.GetParameterAsText(7)
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
outputFC2 = outputFC.split(".")[-1]
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), outputFC2)
else:
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), os.path.basename(outputFC))

else:
newIteration = (math.ceil(iteration/1000.0) * 1000)
x = minOID + 1000
y = minOID
firstTime = 'True'

while x <= newIteration:
where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)
fields ='*'

query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query

fs = arcpy.FeatureSet()
fs.load(fsURL)

if firstTime == 'True':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
outputFC = arcpy.GetParameterAsText(7)
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
outputFC2 = outputFC.split(".")[-1]
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), outputFC2)
else:
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), os.path.basename(outputFC))
firstTime = 'False'
else:
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
insertRows = arcpy.da.InsertCursor(outputFC, ["*","SHAPE@"])
searchRows = arcpy.da.SearchCursor(fs, ["*","SHAPE@"])
for searchRow in searchRows:
fieldList = list(searchRow)
insertRows.insertRow(fieldList)
elif desc.workspaceFactoryProgID == '':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
arcpy.Append_management(fs, outputFC, "NO_TEST")
else:
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
arcpy.Append_management(fs, outputFC)
x += 1000
y += 1000

try:
del searchRow, searchRows, insertRows
except:
pass

# Check to see if downloading a feature or tabular data from a ArcGIS Server service
if agsService == 'true':
if agsFeatures != 'true' and agsTable != 'true':
arcpy.AddError("\nPlease check 'Downloading Feature Data' or 'Downloading Tabular Data'\n")

# Code for downloading feature data
if agsFeatures == 'true':
if iteration < 1000:
x = iteration
y = minOID
where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)
fields ='*'

query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query
fs = arcpy.FeatureSet()
fs.load(fsURL)

arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
outputFC = arcpy.GetParameterAsText(7)
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
outputFC2 = outputFC.split(".")[-1]
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), outputFC2)
else:
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), os.path.basename(outputFC))

else:
newIteration = (math.ceil(iteration/1000.0) * 1000)
x = minOID + 1000
y = minOID
firstTime = 'True'

while x <= newIteration:
where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)
fields ='*'

query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query

fs = arcpy.FeatureSet()
fs.load(fsURL)

if firstTime == 'True':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
outputFC = arcpy.GetParameterAsText(7)
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
outputFC2 = outputFC.split(".")[-1]
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), outputFC2)
else:
arcpy.FeatureClassToFeatureClass_conversion(fs, os.path.dirname(outputFC), os.path.basename(outputFC))
firstTime = 'False'
else:
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
insertRows = arcpy.da.InsertCursor(outputFC, ["*","SHAPE@"])
searchRows = arcpy.da.SearchCursor(fs, ["*","SHAPE@"])
for searchRow in searchRows:
fieldList = list(searchRow)
insertRows.insertRow(fieldList)
elif desc.workspaceFactoryProgID == '':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
arcpy.Append_management(fs, outputFC, "NO_TEST")
else:
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
arcpy.Append_management(fs, outputFC)
x += 1000
y += 1000

try:
del searchRow, searchRows, insertRows
except:
pass

# Code for downloading tabular data
if agsTable == 'true':
if iteration < 1000:
x = iteration
y = minOID
where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)
fields ='*'

query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query

fs = arcpy.RecordSet()
fs.load(fsURL)

arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
outputFC = arcpy.GetParameterAsText(7)
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
outputFC2 = outputFC.split(".")[-1]
arcpy.TableToTable_conversion(fs, os.path.dirname(outputFC), outputFC2)
else:
arcpy.TableToTable_conversion(fs, os.path.dirname(outputFC), os.path.basename(outputFC))

else:
newIteration = (math.ceil(iteration/1000.0) * 1000)
x = minOID + 1000
y = minOID
firstTime = 'True'

while x <= newIteration:
where = OID + '>' + str(y) + 'AND ' + OID + '<=' + str(x)
fields ='*'

query = "?where={}&outFields={}&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query

fs = arcpy.RecordSet()
fs.load(fsURL)

if firstTime == 'True':
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
outputFC = arcpy.GetParameterAsText(7)
desc = arcpy.Describe(os.path.dirname(outputFC))
if desc.workspaceFactoryProgID == 'esriDataSourcesGDB.SdeWorkspaceFactory.1':
outputFC2 = outputFC.split(".")[-1]
arcpy.TableToTable_conversion(fs, os.path.dirname(outputFC), outputFC2)
else:
arcpy.TableToTable_conversion(fs, os.path.dirname(outputFC), os.path.basename(outputFC))
firstTime = 'False'
else:
desc = arcpy.Describe(os.path.dirname(outputFC))
arcpy.AddMessage('Copying features with ObjectIDs from ' + str(y) + ' to ' + str(x))
arcpy.Append_management(fs, outputFC)
x += 1000
y += 1000

try:
del searchRow, searchRows, insertRows
except:
pass

# Code for retrieving attachments
getAttachments = arcpy.GetParameterAsText(8)

if getAttachments == 'true':
# Create Replica to retrieve attachments
arcpy.AddMessage("\nRetrieving Attachments\n")
cwd = arcpy.GetParameterAsText(9)
crUrl = os.path.join(baseURL[0:-8], 'createReplica') # varied AHAY 20180712

crValues = {'f' : 'json',
'layers' : '0',
'returnAttachments' : 'true',
'token' : token }


crData = urllib.urlencode(crValues)
crRequest = urllib2.Request(crUrl, crData)
crResponse = urllib2.urlopen(crRequest)
crJson = json.load(crResponse)

try:
replicaUrl = crJson['URL']
except KeyError:
arcpy.AddWarning("\nService does not have 'Sync' operation enabled\n")

replicaUrl=("https{}").format(replicaUrl[4:])+"?token={}".format(crValues['token']) # added AHAY 20180712
urllib.urlretrieve(replicaUrl, cwd + os.sep + 'myLayer.json')

f = open(cwd + os.sep + 'myLayer.json')

lines = f.readlines()
f.close()

for line in lines:

if not 'attachments' in line:
arcpy.AddWarning("\nService does not contain attachments\n")
os.remove(cwd + os.sep + 'myLayer.json')
sys.exit()

# Get Attachment
with open(cwd + os.sep + 'myLayer.json') as data_file:
data = json.load(data_file)

dict = {}
x = 0
while x <= iteration:
try:
dict[data['layers'][0]['features']['attributes'][OID]] = data['layers'][0]['features']['attributes']['GlobalID']
x += 1
except IndexError:
x += 1
pass

fc = arcpy.GetParameterAsText(7)
arcpy.AddField_management(fc, "GlobalID_Str", "TEXT")

for key in dict:
with arcpy.da.UpdateCursor(fc, [OID, "GlobalID_Str"], OID + " = " + str(key)) as cursor:
for row in cursor:
row[1] = dict[key]
cursor.updateRow(row)

arcpy.EnableAttachments_management(fc)
arcpy.AddField_management(fc + "__ATTACH", "GlobalID_Str", "TEXT")
arcpy.AddField_management(fc + "__ATTACH", "PhotoPath", "TEXT")

# Add Attachments
# Create Match Table
try:
for x in data['layers'][0]['attachments']:
gaUrl = x['url']
gaFolder = cwd + os.sep + x['parentGlobalId']
if not os.path.exists(gaFolder):
os.makedirs(gaFolder)
gaName = x['name']
gaValues = {'token' : token }
gaData = urllib.urlencode(gaValues)
urllib.urlretrieve(url=gaUrl + '/' + gaName, filename=os.path.join(gaFolder, gaName),data=gaData)


rows = arcpy.InsertCursor(fc + "__ATTACH")
hasrow = False
for cmtX in data['layers'][0]['attachments']:
row = rows.newRow()
hasrow = True
row.setValue('GlobalID_Str', cmtX['parentGlobalId'])
row.setValue('PhotoPath', cwd + os.sep +cmtX['parentGlobalId'] + os.sep + cmtX['name'])
rows.insertRow(row)

if hasrow == True:
del row
del rows
arcpy.AddAttachments_management(fc, 'GlobalID_Str', fc + '__ATTACH', 'GlobalID_Str', 'PhotoPath')

try:
arcpy.MakeTableView_management(fc + '__ATTACH', "tblView")
arcpy.SelectLayerByAttribute_management("tblView", "NEW_SELECTION", "DATA_SIZE = 0")
arcpy.DeleteRows_management("tblView")
arcpy.DeleteField_management(fc + '__ATTACH', 'GlobalID_Str')
arcpy.DeleteField_management(fc + '__ATTACH', 'PhotoPath')
except:
pass
except KeyError:
pass

os.remove(cwd + os.sep + 'myLayer.json')

Hey Jake,

This tool works great, but I would like to download only features that have been edited within the last hour. Is this possible with a Where Clause? I have a feature service hosted in arcgis online with editor tracking turned on.

Thanks!

Drew MerrillSQL functions are supported in the query.  Try the following:

EditDate > (CURRENT_TIMESTAMP - 1)

You will replace EditDate with your field name.

Anonymous User

Is there a simplified version of this as just a ArcPy script that I can run independently without the use of toolbox. I'm looking for a way to just download a hosted feature service I own on AGOL. 

Anthony Von Moos‌ when you download the tool, it will include the python script.  You can edit this and change the arcpy.GetParameterAsText to your parameters and then you can run this on a scheduled task if that's what you're looking to do.  For the parameters that you check/uncheck in the tool, you will want to specify 'true' or 'false'.

Anonymous User

Thanks for the reply. I updated the script and ran it but I'm receiving the error:  NameError: name 'token' is not defined. 

Anonymous User

also what is the cwd parameter?

Thanks.  Is there where a way to auto populate the current time into a field as I add new features to a hosted feature class?

Anthony Von Moos‌ that is the directory where your attachments will be stored if you are download them. If you want to share your service to an ArcGIS Online group and invite my user account (jskinner_CountySandbox).  I can take a look.

Drew Merrill‌ you can enable editor tracking on the hosted feature service by going to the Settings tab and checking 'Keep track of who created and last updated features':

This will add 4 additional fields (Creator, CreateDate, Editor, EditDate) to your service.

Anonymous User

Done! I'm wanting to be able to run that script and download a shapefile copy of that Test_Approaches feature layer to my C drive.

Perfect thanks that worked. I missed the fields being added when I enabled editor tracking the first time, but I see them now.

Anthony Von Moos‌ here were the parameters I used to get this to work:

Anonymous User

It looks like that did the trick! I appreciate all your help on this.

Jake,

I've been trying to apply an additional filter on the initial whereClause variable so that it only pulls feature from my Hosted Feature Service after a specified date.  I tried at about line # 220 of the script so that the params variable includes the updated whereClause -- this is what I'm changing from:

    # Return largest ObjectID
    if whereClause == '':
        whereClause = '1=1'
    params = {'where': whereClause, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
    data = urllib.parse.urlencode(params)
    data = data.encode('ascii') # data should be bytes
    req = urllib.request.Request(baseURL, data)
    response = urllib.request.urlopen(req)
    data = response.read().decode("utf-8")
    json_acceptable_string = data.replace("'", "\"")
    data = json.loads(json_acceptable_string)‍‍‍‍‍‍‍‍‍‍‍

To:

# Return largest ObjectID
    #whereClause = '1=1'
    whereClause = "CreationDate>'{}'".format(inputD)
    if whereClause == '':
        whereClause = '1=1'

    #params = urllib.urlencode({'f': 'pjson', 'where': "CreationDate>'{}'".format(inputDate), 'outFields': '*', 'token': token, 'returnGeometry': 'true'})
    params = {'where': whereClause, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
    req = urllib2.Request(baseURL, urllib.urlencode(params))
    print baseURL
    response = urllib2.urlopen(req)
    data = json.load(response)‍‍‍‍‍‍‍‍‍‍‍‍

However, anywhere else in the code where it checks for "if whereClause != '1=1':" then it seems to run into issues.  Perhaps because it is using a Replica (which is out of sync with the queries feature service?).

Is there a straight forward way to alter this script so that the initial whereClause/params filtered on an input date?

Thanks

I am having trouble with the attachments (images) not being able to be opened. The Geometry and Attributes seem to come through fine, but the Photo1.jpg (similar for all) gives me the error "It looks like we don't support this file format". I get similar results opening with Windows Photos as well as paint.  Any help would be greatly appreciated.  

On another note, I occasionally get an error when running the script that "GlobalID_Str" has too many characters for the arcpy add field module.  

ArcMap 10.6

Jonathan 

Jonathan Holt‌ can you share your service with an ArcGIS  group and invite my user account (jskinner_CountySandbox)?  I can see if I can reproduce.

James Crandall‌ can you share the service you are working with to an ArcGIS  group and invite my user account (jskinner_CountySandbox)?  If so, what is the query you are trying to execute?  I'll try setting up the parameters to execute the script with the query.

Hi Jake -- I'm running this as a scheduled task (a . bat executes the .py script), so no Toolbox implementation. 

The whereClause is how I posted in my last reply -- I just want to change to only use features from the service after an input date using the CreationDate column.  The whereClause works and applied correctly and the script runs most the way thru.  It fails when attempting to add attachments and create the match table.

Here's the first whereClause statement(s) setup (whereClause2 is for the date filter, I left the original whereClause the same):

    # Return largest ObjectID
    whereClause = '1=1'
    whereClause2 = "CreationDate>'{}'".format(inputD)
    if whereClause == '':
        whereClause = '1=1'

    #params = urllib.urlencode({'f': 'pjson', 'where': "CreationDate>'{}'".format(inputDate), 'outFields': '*', 'token': token, 'returnGeometry': 'true'})
    params = {'where': whereClause2, 'returnIdsOnly': 'true', 'token': token, 'f': 'json'}
    req = urllib2.Request(baseURL, urllib.urlencode(params))
    print baseURL
    response = urllib2.urlopen(req)
    data = json.load(response)
    try:
        data['objectIds'].sort()
    except:
        arcpy.AddError("\nURL is incorrect.  Or, Service is secure, please enter username and password.\n")
        #sys.exit()

    OIDs = data['objectIds']
    arcpy.AddMessage(OIDs)
    count = len(data['objectIds'])
    iteration = int(data['objectIds'][-1])
    minOID = int(data['objectIds'][0]) - 1
    OID = data['objectIdFieldName']

I get the correct list of OIDs.  Also, I can see that it retrieves the attachments & creates replica just fine

It fails in the section

        # Add Attachments
        # Create Match Table
        if whereClause != '1=1':
            attachmentGlobalIDs = []
            for x in data['layers'][0]['features']:
                if x['attributes'][OID] in OIDs:
                    attachmentGlobalIDs.append(x['attributes']['GlobalID'])

        try:
            for x in data['layers'][0]['attachments']:
                if whereClause != '1=1':
                    for attachmentGlobalID in attachmentGlobalIDs:
                        if x['parentGlobalId'] == attachmentGlobalID:
                            gaUrl = x['url']
                            gaFolder = cwd + os.sep + x['parentGlobalId']
                            if not os.path.exists(gaFolder):
                                os.makedirs(gaFolder)
                            gaName = x['name']
                            gaValues = {'token' : token }
                            gaData = urllib.urlencode(gaValues)
                            try:
                                urllib.urlretrieve(url=gaUrl + '/' + gaName, filename=os.path.join(gaFolder, gaName),data=gaData)
                            except:
                                PrintException("Error Retrieving Attachments")

                            rows = arcpy.da.InsertCursor(fc + "__ATTACH", ["GlobalID_Str", "PhotoPath"])
                            rows.insertRow((x['parentGlobalId'], cwd + os.sep + x['parentGlobalId'] + os.sep + x['name']))
                            del rows
                            hasrow = True

Specifically, it throws an error here

         if hasrow == True:
                try:
                    arcpy.AddMessage("Adding attachments")
                    arcpy.AddAttachments_management(fc, 'GlobalID_Str', fc + '__ATTACH', 'GlobalID_Str', 'PhotoPath')
                except:
                    PrintException("Error Retrieving Attachments")

            try:
                arcpy.MakeTableView_management(fc + '__ATTACH', "tblView")
                arcpy.SelectLayerByAttribute_management("tblView", "NEW_SELECTION", "DATA_SIZE = 0")
                arcpy.DeleteRows_management("tblView")
                arcpy.DeleteField_management(fc + '__ATTACH', 'GlobalID_Str')
                arcpy.DeleteField_management(fc + '__ATTACH', 'PhotoPath')
            except Exception, e:
                print "Error removing empty rows from attachment table"
                PrintException("Error removing empty rows from attachment table")
                pass

The except block throws the print "Error removing empty rows from attachment table".

Jonathan Holt‌ I believe something is corrupt with the attachments.  When trying to view them in a Web Map I get a 404 error.

James Crandall‌ I ran the script in PyScripter with the following parameters:

I did not make any changes to anything else within the script and it downloaded the data with attachments successfully.

I get a KeyError: 'GlobalID' on the last line in this block:

        # Add Attachments
        # Create Match Table
        if whereClause != '1=1':
            attachmentGlobalIDs = []
            for x in data['layers'][0]['features']:
                print x['attributes'][OID]
                if x['attributes'][OID] in OIDs:
                    attachmentGlobalIDs.append(x['attributes']['GlobalID']) <--KeyError thrown here

If you can, share the service to a group and invite my user account (jskinner_CountySandbox).  I can try executing the script to see if I receive the same error.

Thanks!  Invite to group sent.  There's only 1 hosted FS in there.

I'm having the same error KeyError: 'GlobalID' at the same location as James

I get this error when I use a where clause and get attachments.

The tool runs successfully with either a where clause or "get attachments", but throws the error when I try to do both.

The data that I'm trying to download is a hosted feature layer originally from Survey 123

Thanks for that.  I should have mentioned the additional detail you provided, it's probably important:

Hosted FS

Source to a Survey123 survey

Attachments

Thanks again!

Anonymous User

How would I alter this code if my feature service has related tables?

James Crandall‌ I had to change GlobalID to globalid in the below segment of code:

# Add Attachments
# Create Match Table
if whereClause != '1=1':
   attachmentGlobalIDs = []
   for x in data['layers'][0]['features']:
       if x['attributes'][OID] in OIDs:
          attachmentGlobalIDs.append(x['attributes']['globalid'])‍‍‍‍‍‍‍

It appears Survey123 uses this field casing for the GlobalID field.  There didn't seem to be any attachments, so I added one to OBJECTID 1 and the script executed successfully with the following parameters:

Lianne Scott

Anthony Von Moos‌ this tool will not maintain the relationship class.  You can download the related tables by executing the tool for the features, and then again for the table(s).  You would need to recreate the relationship class once the data is downloaded.

Hi Jake Skinner, I'm trying to use your lovely tool to download data from a secure ArcGIS Server feature service but I'm getting an error message saying "URL is incorrect.  Or, Service is secure, please enter username and password." I've checked the URL and username and password that I'm entering and I'm sure I've entered them correctly because I used the same credentials to view the data in ArcGIS Desktop. Before I got this error message I was getting a different error message, an InsecureRequestWarning saying that an "Unverified HTTPS request is being made." I trust the connection so after some brief investigations I added a couple of lines to the top of your script to disable warnings about the connection being insecure:

from requests.packages.urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

But now I get the message "URL is incorrect.  Or, Service is secure, please enter username and password."

I'm now unsure as to what to do next. Are you able to offer any advice about why I'm seeing this message or how I might overcome it?

Mike Steven‌ if your ArcGIS Server instance is not using Windows Authentication, can you private message me on GeoNET with the URL and username/password?  I can investigate the issue.

Thanks Jake Skinner. It says that I can only send a message to people that are following me, please can you follow me?

I believe the issue is with how the token is being generated.  You do not have administrative access enabled, so you will need to update the portion of the Generate Token code:

# Genereate token for AGS feature service
if agsService == 'true':
    try:
        arcpy.AddMessage('\nGenerating Token\n')
        server = baseURL.split("//")[1].split("/")[0]
        tokenURL = 'http://' + server + '/arcgis/admin/generateToken'
        params = {'username': username, 'password': password, 'client': 'requestip', 'f': 'pjson'}
        response = requests.post(tokenURL, data = params, verify = False)
        token = response.json()['token']
    except:
        token = ''
        pass

Change the tokenURL in the above to the fully qualified domain name of your ArcGIS Server instance with port 6080.  Ex:

tokenURL = 'http://ags1.esri.com:6080/arcgis/admin/generateToken'

Let me know if that works for you.

Thanks for the suggestion, unfortunately it's still showing the same error message. If I change the script to:

tokenURL = 'http://' + server + ':6080/arcgis/admin/generateToken'

It still gives the "URL is incorrect.  Or, Service is secure, please enter username and password" message. If I go to that url in a browser, the page times out from taking too long to respond.

Still in a browser if I go to http://<domain>/arcgis/tokens or http://<domain>/arcgis/tokens/generateToken it loads a page that you can generate a token on. But if I then change the script to:

tokenURL = 'http://' + server + '/arcgis/tokens/generateToken'

It still gives the "URL is incorrect.  Or, Service is secure, please enter username and password" message.

You will not want to specify the server variable in the URL.  This will take the portion of URL that, I believe, is your web server where the web adaptor is installed.  You will want to specify the fully qualified domain name of the server where ArcGIS Server is installed.  For example, my ArcGIS Server machine is ags1.esri.com, and my web server where the web adaptor is installed is webserver.esri.com.  For the URL, the following would not work:

http://webserver.esri.com:6080/arcgis/admin/generateToken

But the following would:

http://ags1.esri.com:6080/arcgis/admin/generateToken

You organization might be enforcing SSL handshakes all of the sudden. With URLib, you'll need to hard code the path to your org's ssl cert. 

Anonymous User

If I set GetAttachments = 'True' and leave the CWD blank it still downloads the photos to my root directory. Would I have to set the CWD to false for it to stop downloading them to a directory?

Version history
Last update:
‎07-17-2015 06:12 AM
Updated by:
Contributors