topic Re: Download feature service as file geodatabase in ArcGIS API for Python Questions

Download feature service as file geodatabase

XabierVelasco_Echeverría — Wed, 24 Jan 2018 11:09:04 GMT

Hi,

I apply the following workflow to download a hosted feature service as a file geodatabase:

from arcgis.gis import GIS
gis = GIS("https://www.arcgis.com", USER, PASSWORD)
servicio = gis.content.get('XXXXXXXXXXX')
itemDescargar = servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True')
fgdb = gis.content.get(itemDescargar ['exportItemId'])
fgdb.download(save_path='c:\modelbuilder')

If I execute like this, the downloaded ZIP has 1 KB. If I wait several seconds to download, the downloaded ZIP has the proper size (14MB). That means, servicio.export is still working while I try to download as I am calling from a variable.

If I instead execute servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True') without a variable, the process finishes correctly before the next line of code is executed, but then I do not know what is the value of exportItemId in order to download it. How do I get the results of servicio.export? I mean, how do I get values from the output message? i.e. {'type': 'file', 'size': 61145088, 'jobId': '359e30e3-0fc7-4925-8e92-354a4ccdd478::XXXXXXXX', 'serviceItemId': 'XXXXXXXX', 'exportFormat': 'fileGeodatabase', 'exportItemId': 'XXXXXXXXX'}

Re: Download feature service as file geodatabase

XabierVelasco_Echeverría — Fri, 26 Jan 2018 08:55:28 GMT

For the API developers:

Summing it up, how may I access the dictionary that outputs from an item.export? You can do that with the function results in arcpy, what is the proper way to do it with the arcgis module? I guess it should be super easy, but could not find examples in the documentation

Re: Download feature service as file geodatabase

XabierVelasco_Echeverría — Tue, 30 Jan 2018 11:55:36 GMT

I found out a workaround for this bug.

from arcgis.gis import GIS
gis = GIS("https://www.arcgis.com", USER, PASSWORD)
servicio = gis.content.get('XXXXXXXXXXX')
servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True')
items = gis.content.search(nombre)
fgdb = gis.content.get(items[0].itemid)
fgdb.download(save_path=arcpy.env.scratchFolder)

The documentation for the API is a bit tricky, but if you search content you can get the item ID of the new item (FGDB). Anyway, the export function should not be asyncronous when you use the parameter wait = True. That means, if you call the export function inside a variable, why does it register as downloaded although the download is not finished yet?

Re: Download feature service as file geodatabase

Tom_Laue — Mon, 15 Oct 2018 21:12:33 GMT

I was also intermittently getting the 1 KB zipped file export.

After a lot of testing, I found a solution that has been working for me. I used Xabier Velasco Echeverría's script (above) as a starting point and made a couple of additions:

I added a line when exporting the hosted feature layer to an AGOL File GDB to wait 10 seconds to allow extra time while it searches ArcGIS Online for the item. This has avoided the error that it responds that the File Geodatabase the script created does not exist in ArcGIS Online.
I added a While statement to verify the zipped file downloaded is a valid zipped file (not a 1 KB file). If not, the download will run again and again until a valid zipped file is downloaded.

Attached is my script.

Re: Download feature service as file geodatabase

Sveinung_Bertnes_Råheim — Tue, 30 Oct 2018 08:08:48 GMT

I tested your script. Ran it in the arcgis pro version of python. And it worked first time.
Excellent documentation in the python script for novices like me!

Re: Download feature service as file geodatabase

DarrylKlassen1 — Thu, 20 Jun 2019 22:00:09 GMT

I tried this script - but am getting an error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis.py", line 2590, in __getattr__
raise AttributeError("'%s' object has no attribute '%s'" % (type(self).__name__, name))
AttributeError: 'Item' object has no attribute 'export'

Not sure why the item doesn't have this ability? In the settings in AGOL I have this turned on

Any thoughts?

Thanks

Darryl

Re: Download feature service as file geodatabase

XabierVelasco_Echeverría — Thu, 27 Jun 2019 09:50:35 GMT

Hi Darryl,

What is the size of the geodatabase that you are trying to download?

Re: Download feature service as file geodatabase

DarrylKlassen1 — Thu, 27 Jun 2019 14:59:27 GMT

I don't know the exact size since it is coming from AGOL, but it is a feature service hosted by AGOL with under 100 points, and only about 10 attributes. So quite small.

Re: Download feature service as file geodatabase

XabierVelasco_Echeverría — Fri, 28 Jun 2019 07:23:15 GMT

Ok, could you please post the code you are executing so that I may check what is going on?

Re: Download feature service as file geodatabase

DarrylKlassen1 — Fri, 28 Jun 2019 15:30:57 GMT

import os
import arcpy
from datetime import datetime
n = datetime.now()
datestamp = n.strftime("%B_%d_%Y")
mxd = arcpy.mapping.MapDocument(r"D:\temp\AGOL_Data.mxd")
df = arcpy.mapping.ListDataFrames(mxd)[0]
destPath = r"D:\temp"
outgdb = destPath + "\\GarbageCanSDE.gdb"
for lyr in arcpy.mapping.ListLayers(mxd, '', df):
    if lyr.isFeatureLayer == True:
        d = lyr.name.encode('ascii', 'ignore')
        d = d.replace(' ','_')
        #file_name = d + "_" + datestamp + ".shp"
        file_name = outgdb + "\\" + d + "_" + datestamp
        dest1 = os.path.join(destPath, file_name)
        print(file_name)
        print(type(file_name))
        print(dest1)
        if arcpy.Exists(dest1):
            arcpy.Delete_management(dest1)
            print("deleted")
        try:
            arcpy.CopyFeatures_management(lyr, dest1)
        except:
            pass

Re: Download feature service as file geodatabase

AllanKrygeris1 — Sun, 30 Jun 2019 16:23:41 GMT

Thanks for this.

I was getting the "'dict' object has no attribute 'download'" error until I changed the wait parameter from a string:

servicio.export(nombre, 'File Geodatabase', parameters=None, wait='True')

to a boolean:

servicio.export(nombre, 'File Geodatabase', parameters=None, wait=True)

Re: Download feature service as file geodatabase

XabierVelasco_Echeverría — Mon, 01 Jul 2019 07:42:03 GMT

Hi Darryl,

In that piece of code there is no reference to the ArcGIS Python API...

Best regards,

Xabier

Re: Download feature service as file geodatabase

Danielle_Journey — Tue, 23 Jul 2019 16:04:08 GMT

Has anyone tried using the Windows Task Scheduler to run this script when not logged in to their computer?

I have an issue were the Task Scheduler will start the script, creates the temporary file geodatabase in AGO and then stops? To set up the task, I followed: Schedule a Python script or model to run at a prescribed time: 2019 update stored my credentials, and ran with highest privileges. I've also checked the box to "Allow others to export to different formats" on the hosted feature service. For reference, the Windows Task Scheduler runs the script perfectly if I am logged in to the computer. Any thoughts?? Tom Laue‌ Xabier Velasco Echeverría‌

Re: Download feature service as file geodatabase

RichardFisher — Tue, 30 Jul 2019 17:49:47 GMT

We will be trying to do so with FME software.

Re: Download feature service as file geodatabase

RichardFisher — Tue, 30 Jul 2019 17:59:30 GMT

While borrowing from many of these comments and script ideas you all have posted, I have an error in python right after the download from AGOL to the local zip file. Where it seems the test for valid zip is not done fast enough to prevent one extra loop from trying the download, or it is some kind of timeout as this runs for an hour.

This is the part that finishes creating the correct zip file download but errors off right at the end in the ArcGIS Pro connection.py and request.py.

I believe that I can put in a try catch and work around this, but would prefer to understand the issue and have it work with out that tactic.

'''while statement runs until a valid zipped file is created'''

zipfullpath=os.path.join(outputFolder,GDBname+".zip")

while zipfile.is_zipfile(zipfullpath)==False:

print("looping: " + str(datetime.now()))

fgb.download(save_path=outputFolder)

zf = ZipFile(os.path.join(outputFolder,GDBname+ ".zip"))

looping: 2019-07-29 13:26:10.354947
Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 849, in get
resp = opener.open(url)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
error.HTTPError: HTTP Error 498: 498
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\xxx\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\ptvsd_launcher.py", line 43, in <module>
main(ptvsdArgs)
File "c:\Users\xxx\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\__main__.py", line 434, in main
run()
File "c:\Users\xxx\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\__main__.py", line 312, in run_file
run_path(target, run_name='__main__')
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\wip\sandbox\agol_py\ArcGIS API AGOL Download Feature Layer as GDB_Python3.py", line 45, in <module>
download(save_path=outputFolder)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 7032, in download
out_folder=save_path, try_json=False, force_bytes=False)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 924, in get
return self.get(newpath, ssl, try_json, is_retry=True)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 816, in get
params['f'] = 'json'
TypeError: 'bool' object does not support item assignment
At the bottom it also seems that in connection.py the bool test is not valid around line 815

if try_json:

params['f'] = 'json'

Re: Download feature service as file geodatabase

RichardFisher — Wed, 31 Jul 2019 20:41:41 GMT

I did change the code to load the bool check of is a proper zip file and it ran through with the result finishing and creating the valid zip file of the download but with an error still

Reason for exception 'bool' object does not support item assignment

I took this to mean it was still failing in the arcpro connection.py on line 816 trying to check for a Boolean json.

I have not tried this yet but possibly I am missing a parameter of 'f' and setting it to fgd ?? Hard to find the documentation on this as it relates to fgb.dowload.

As of now it just takes in the file name, but in the manual web interface you do have to choose a shapefile or fgdb or etc....

fgb.download(save_path=outputFolder)

If I find an answer I will update this memo....

File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 816, in get
params['f'] = 'json'
TypeError: 'bool' object does not support item assignment

Re: Download feature service as file geodatabase

RichardFisher — Tue, 06 Aug 2019 14:37:35 GMT

Removing the loop seems to solve the issue and keeps the correct data downloaded.

There is a new issue that I will enter in a new thread - The manual or via python download of AGOL Hosted feature to a fgdb changes the creator and edited by fields to be the account doing the download - I did take some input from a post regarding params for publishing that would preserver tracking data and when I try it, the export still runs but does not preserve the data.

Re: Download feature service as file geodatabase

XabierVelasco_Echeverría — Wed, 07 Aug 2019 05:49:52 GMT

Hi Richard, what is the size of the data you are downloading? I had tu use the loop as there were issues while downloading big geodatabases (i.e. 1.8GB) without the loop.

Re: Download feature service as file geodatabase

RichardFisher — Wed, 07 Aug 2019 12:24:07 GMT

Hello Xabier - The compressed file is 1,052,933 KB.

So there were two issues as you can see, the one updating the records with the script login is the most painful as it alters the content and is not just a harmless download of data as a back up.

Currently if I don't use the look it works well except the parameter to not update the 'creator' and the 'edited by' fields are ignored and updated by the login running the script.

If I include the loop it also finishes with a complete download but fails on the bool issue testing the zip file which I believe but can't prove is back in the arcpro py File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 816, in get

params['f'] = 'json'
TypeError: 'bool' object does not support item assignment

Re: Download feature service as file geodatabase

SurveyAdmin — Wed, 20 May 2020 13:24:44 GMT

This is a great script and I have gotten the .zip file to download. However, the script seems to be getting stuck and never gets passed "Zipping exported geodatabase for download...", When I finally stop the script, the .zip file is there with all of the expected data, but not sure why I can't get the script to complete. Can you offer any insight?