Download feature service as file geodatabase

XabierVelasco_Echeverría · ‎01-24-2018

Hi,

I apply the following workflow to download a hosted feature service as a file geodatabase:

from arcgis.gis import GIS
gis = GIS("https://www.arcgis.com", USER, PASSWORD)
servicio = gis.content.get('XXXXXXXXXXX')
itemDescargar = servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True')
fgdb = gis.content.get(itemDescargar ['exportItemId'])
fgdb.download(save_path='c:\modelbuilder')

If I execute like this, the downloaded ZIP has 1 KB. If I wait several seconds to download, the downloaded ZIP has the proper size (14MB). That means, servicio.export is still working while I try to download as I am calling from a variable.

If I instead execute servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True') without a variable, the process finishes correctly before the next line of code is executed, but then I do not know what is the value of exportItemId in order to download it. How do I get the results of servicio.export? I mean, how do I get values from the output message? i.e. {'type': 'file', 'size': 61145088, 'jobId': '359e30e3-0fc7-4925-8e92-354a4ccdd478::XXXXXXXX', 'serviceItemId': 'XXXXXXXX', 'exportFormat': 'fileGeodatabase', 'exportItemId': 'XXXXXXXXX'}

DarrylKlassen1 · ‎06-28-2019

import os
import arcpy
from datetime import datetime
n = datetime.now()
datestamp = n.strftime("%B_%d_%Y")
mxd = arcpy.mapping.MapDocument(r"D:\temp\AGOL_Data.mxd")
df = arcpy.mapping.ListDataFrames(mxd)[0]
destPath = r"D:\temp"
outgdb = destPath + "\\GarbageCanSDE.gdb"
for lyr in arcpy.mapping.ListLayers(mxd, '', df):
    if lyr.isFeatureLayer == True:
        d = lyr.name.encode('ascii', 'ignore')
        d = d.replace(' ','_')
        #file_name = d + "_" + datestamp + ".shp"
        file_name = outgdb + "\\" + d + "_" + datestamp
        dest1 = os.path.join(destPath, file_name)
        print(file_name)
        print(type(file_name))
        print(dest1)
        if arcpy.Exists(dest1):
            arcpy.Delete_management(dest1)
            print("deleted")
        try:
            arcpy.CopyFeatures_management(lyr, dest1)
        except:
            pass

XabierVelasco_Echeverría · ‎07-01-2019

Hi Darryl,

In that piece of code there is no reference to the ArcGIS Python API...

Best regards,

Xabier

Danielle_Journey · ‎07-23-2019

Has anyone tried using the Windows Task Scheduler to run this script when not logged in to their computer?

I have an issue were the Task Scheduler will start the script, creates the temporary file geodatabase in AGO and then stops? To set up the task, I followed: Schedule a Python script or model to run at a prescribed time: 2019 update stored my credentials, and ran with highest privileges. I've also checked the box to "Allow others to export to different formats" on the hosted feature service. For reference, the Windows Task Scheduler runs the script perfectly if I am logged in to the computer. Any thoughts?? Tom Laue‌ Xabier Velasco Echeverría‌

RichardFisher · ‎07-30-2019

We will be trying to do so with FME software.

SurveyAdmin · ‎05-20-2020

This is a great script and I have gotten the .zip file to download. However, the script seems to be getting stuck and never gets passed "Zipping exported geodatabase for download...", When I finally stop the script, the .zip file is there with all of the expected data, but not sure why I can't get the script to complete. Can you offer any insight?

JDSF_PeterRowland · ‎03-03-2022

This has been giving me grief. as far as I can tell its trying to reference an undeclared variable?

RichardFisher · ‎07-30-2019

While borrowing from many of these comments and script ideas you all have posted, I have an error in python right after the download from AGOL to the local zip file. Where it seems the test for valid zip is not done fast enough to prevent one extra loop from trying the download, or it is some kind of timeout as this runs for an hour.

This is the part that finishes creating the correct zip file download but errors off right at the end in the ArcGIS Pro connection.py and request.py.

I believe that I can put in a try catch and work around this, but would prefer to understand the issue and have it work with out that tactic.

'''while statement runs until a valid zipped file is created'''

zipfullpath=os.path.join(outputFolder,GDBname+".zip")

while zipfile.is_zipfile(zipfullpath)==False:

print("looping: " + str(datetime.now()))

fgb.download(save_path=outputFolder)

zf = ZipFile(os.path.join(outputFolder,GDBname+ ".zip"))

looping: 2019-07-29 13:26:10.354947
Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 849, in get
resp = opener.open(url)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
error.HTTPError: HTTP Error 498: 498
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\xxx\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\ptvsd_launcher.py", line 43, in <module>
main(ptvsdArgs)
File "c:\Users\xxx\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\__main__.py", line 434, in main
run()
File "c:\Users\xxx\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\__main__.py", line 312, in run_file
run_path(target, run_name='__main__')
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\wip\sandbox\agol_py\ArcGIS API AGOL Download Feature Layer as GDB_Python3.py", line 45, in <module>
download(save_path=outputFolder)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 7032, in download
out_folder=save_path, try_json=False, force_bytes=False)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 924, in get
return self.get(newpath, ssl, try_json, is_retry=True)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 816, in get
params['f'] = 'json'
TypeError: 'bool' object does not support item assignment
At the bottom it also seems that in connection.py the bool test is not valid around line 815

if try_json:

params['f'] = 'json'

RichardFisher · ‎07-31-2019

I did change the code to load the bool check of is a proper zip file and it ran through with the result finishing and creating the valid zip file of the download but with an error still

Reason for exception 'bool' object does not support item assignment

I took this to mean it was still failing in the arcpro connection.py on line 816 trying to check for a Boolean json.

I have not tried this yet but possibly I am missing a parameter of 'f' and setting it to fgd ?? Hard to find the documentation on this as it relates to fgb.dowload.

As of now it just takes in the file name, but in the manual web interface you do have to choose a shapefile or fgdb or etc....

fgb.download(save_path=outputFolder)

If I find an answer I will update this memo....

File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 816, in get
params['f'] = 'json'
TypeError: 'bool' object does not support item assignment

RichardFisher · ‎08-06-2019

Removing the loop seems to solve the issue and keeps the correct data downloaded.

There is a new issue that I will enter in a new thread - The manual or via python download of AGOL Hosted feature to a fgdb changes the creator and edited by fields to be the account doing the download - I did take some input from a post regarding params for publishing that would preserver tracking data and when I try it, the export still runs but does not preserve the data.

XabierVelasco_Echeverría · ‎08-06-2019

Hi Richard, what is the size of the data you are downloading? I had tu use the loop as there were issues while downloading big geodatabases (i.e. 1.8GB) without the loop.