Download feature service as file geodatabase

15175
27
Jump to solution
01-24-2018 03:09 AM
XabierVelasco_Echeverría
Occasional Contributor

Hi,

I apply the following workflow to download a hosted feature service as a file geodatabase:

from arcgis.gis import GIS
gis = GIS("https://www.arcgis.com", USER, PASSWORD)
servicio = gis.content.get('XXXXXXXXXXX')
itemDescargar = servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True')
fgdb = gis.content.get(itemDescargar ['exportItemId'])
fgdb.download(save_path='c:\modelbuilder')

If I execute like this, the downloaded ZIP has 1 KB. If I wait several seconds to download, the downloaded ZIP has the proper size (14MB). That means, servicio.export is still working while I try to download as I am calling from a variable.

If I instead execute servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True') without a variable, the process finishes correctly before the next line of code is executed, but then I do not know what is the value of exportItemId in order to download it. How do I get the results of servicio.export? I mean, how do I get values from the output message? i.e.  {'type': 'file', 'size': 61145088, 'jobId': '359e30e3-0fc7-4925-8e92-354a4ccdd478::XXXXXXXX', 'serviceItemId': 'XXXXXXXX', 'exportFormat': 'fileGeodatabase', 'exportItemId': 'XXXXXXXXX'}

27 Replies
RichardFisher
New Contributor III

Hello Xabier - The compressed file is 1,052,933 KB.

So there were two issues as you can see, the one updating the records with the script login is the most painful as it alters the content and is not just a harmless download of data as a back up.

Currently if I don't use the look it works well except the parameter to not update the 'creator' and the 'edited by' fields are ignored and updated by the login running the script.

If I include the loop it also finishes with a complete download but fails on the bool issue testing the zip file which I believe but can't prove is back in the arcpro py File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 816, in get

  •    params['f'] = 'json'
  • TypeError: 'bool' object does not support item assignment
0 Kudos
ChrisCzerwinski
New Contributor III

I've been running this script on multiple feature service for a couple years now.  I just upgraded to ArcPro2.6 and now I'm getting this looping issue at fgb.download().  Can't seem to resolve it??

0 Kudos
RichardFisher
New Contributor III

I'm on 2.5.2 pro and trying the upgrade 

I am not using the loop at all I just have one export line with params and a wait statement following .

    AGOLitem.export(GDBname,'File Geodatabase'parameters=pub_params, wait=True)

I have one large fgdb download that works and one that does not getting errors about indices which I have not found an answer to yet.

File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 7340, in export

    if status['status'] == 'failed':

TypeError: string indices must be integers

The other strange thing is that where previous I could import arcgis.gis that no longer worked and had to switch to the format

from arcgis.gis import GIS as g to make it work.

I am currently upgrading to see if it helps or hurts.

I will also try to rebuild the indexes to see if that helps.

0 Kudos
ChrisCzerwinski
New Contributor III

Thanks for the quick reply Richard Fisher‌.  Can you share a sample script in context?  Hoping to see your changes in context.  Thanks!

0 Kudos
RichardFisher
New Contributor III

Sure - There is more there because it exports and puts away several file geodatabase files with the download that creates folders elsewhere but the general flow for one of them is here.

It was working for over a year and now, fails on one which may be due to a change my customer made in layer assignment still trying to sort that out and the changes I had to make to arcgis.gis for some reason..? ArcPro upgrades not sure.. but it's running on Python 3.6.10 64bit and works for one Hosted Feature... hope this helps with some context.

#import arcgis.gis

from arcgis.gis import GIS as g

# Snipped from calling python file to file doing the detail export work

start = timer()
print ("Export Hosted Feature")

from AGOL_aaaaaaaaa import export_xxxxxxxx_from_agol

export_xxxxxxxx_from_agol()

end = timer()

print ("aaaaaaaaaaaaa export in AGOL Done " + str((end - start)/60) ) # Time in minutes


# ---- python general flow of export on one of the hosted features called by the 'main' python snippet above

import time
import datetime
import os
import zipfile
#import arcgis.gis

from arcgis.gis import GIS as g

from zipfile import ZipFile
from datetime import datetime
import pathlib

from timeit import default_timer as timer
from datetime import datetime, date
from dateutil.relativedelta import relativedelta

import csv, sys
# --------------------------------- vars -----------------------------------------------
# Where the GDB will be extracted to after extraction and then download of that extraction
outputFolder=r"\\xxxxxxxxx\yyyyyyyyyy\zzzzzz"
# Path to credentials file for Prod
credentialsFile=r"C:\\aaa\\bbb\\cccc\\yourcreds.csv"
# Gather site url and login credentials
cred = ''
# log into AGOL object
gis = ''
# name of AGOL temporary GDB saved from Hosted feature class
GDBname = "xxxxxxxxx_fgdb"
# item_id is the service_ID of the hosted Prod CRUB feature layer to download
item_id='aaaaaaaaaaaaaaaaaaaaaaaaaaa'
#Parameters for editor tracking try and preserve
pub_params = {"editorTrackingInfo" : {"enableEditorTracking":True, "preserveEditUsersAndTimestamps":True}}

# --------------------------- called functions ------------------------------
# open(file, mode, buffering, encoding, errors, newline, closefd, opener)
def read_credentials():
with open(credentialsFile, newline='') as f:
reader = csv.reader(f)
for row in reader:
print("row[0] :" + str(row[0]))

return row

def export_xxxxxxxx_from_agol():
print ("AGOL_ADA_Export_CURB_Prod.py mndot.maps.arcgis.com")
# Gather site url and login credentials
cred = read_credentials()
# log into AGOL object
gis = g(cred[0], cred[1], cred[2])

print ("GDBname : " + GDBname )

# search AGOL for the item to export via the item id
AGOLitem = gis.content.get(item_id)

print ("Exporting Hosted ccccccccc Feature Layer to temp AGOL file..." + str(GDBname))

AGOLitem.export(GDBname,'File Geodatabase', parameters=pub_params, wait=True)

#add 20 seconds delay to allow export to complete
time.sleep(60)

print ("Export Hosted xxxxxxxxx Feature Layer to temp AGOL file Done ..." + str(GDBname))

return GDBname

# --------------------------- called functions ------------------------------

def search_for_agol_fgdb(monthFolder):
"""This function searches for the name of the AGOL
file geodatabase exported from the hosted feature class"""
try:
print ("Try targeted search for :" + str(GDBname) + ".zip")
# Gather site url and login credentials
cred = read_credentials()
# log into AGOL object
gis = g(cred[0], cred[1], cred[2])
print ("GDBname : " + GDBname )
search_fgb = gis.content.search(query=(str(GDBname) + ".zip"), item_type = 'File Geodatabase',sort_field="created" ,sort_order="desc")
fgb_item_id = search_fgb[0].id
fgb = gis.content.get(fgb_item_id)
print ("Finished search. Starting download to folder..:" + monthFolder)
fgb.download(save_path=monthFolder) #download file gdb from ArcGIS Online to your computer
return fgb
print ("Finished targeted search")
except Exception as e:
print("type error: " + str(e))
print ("Was not able to find exact match to download :" + (str(GDBname) + ".zip") )
return "Error : " + str(e) + " : No data found"
finally:
print ("Finished search.")


# backup one month and year if needed for folder name to store downloads
def create_month_directory():
mydate = datetime.now()
backdate = mydate - relativedelta(months=1)
# create the output folder
pathlib.Path(outputFolder + '\\' + str(backdate.strftime("%b")) + '_' + str(backdate.strftime("%Y"))).mkdir(exist_ok=True)

monthFolder = (outputFolder + '\\' + str(backdate.strftime("%b")) + '_' + str(backdate.strftime("%Y")))

print("monthFolder : " + str(monthFolder))

return monthFolder

0 Kudos
ChrisCzerwinski
New Contributor III

Thanks.  Still can't seem to get things going.  I might try going back to the ArcPro version I was on when this was working... just to test.  Please let me know if you come up with anything new.

0 Kudos
ChrisCzerwinski
New Contributor III

I downgraded back to ArcPro2.3.0 and on python 3.6.6.  Now everything is working as expected.   Basically using the same script shared by Thomas Laue‌ ("ArcGIS API AGOL Download Feature Layer as GDB_Python3.py").  Tells us something I guess, but I don't know what yet....

0 Kudos
ChrisCzerwinski
New Contributor III

Reviving this topic again since I'm back on ArcPro2.6, and now the problem is back.  Like @XabierVelasco_Echeverría I can cancel the script once it's in a loop and get the expected data download, but I'd like to be able to pin point the error here. 

0 Kudos