topic Re: Download Item Data in ArcGIS API for Python Questions

Download Item Data

LesiMai3 — Wed, 28 Dec 2016 19:57:55 GMT

Hi,

I was trying to use the item's download method to download the data associated with a map image layer on Portal. But it doesn't seem to be able to download the data and I got b' ' as the output. I also tried the get_data method, which returns an error message saying "the JSON object must be str, not 'bytes'".

I did a similar workflow on a hosted feature layer on ArcGIS Online. And this time, it was able to download the item. But it only returns the data schema in a dictionary. Is there way to download the actual data behind a hosted feature service? Can the download method work with map image layers?

Or did I miss something here?

Thank you!

Re: Download Item Data

RohitSingh2 — Sun, 12 Dec 2021 10:54:35 GMT

Hello,

You can download the data for file based item types. File based items include CSV, Service Definition, Shapefile, File Geodatabase, Powerpoint, etc.

Hosted feature layers and map image layers are not file based items, so you cannot use the item.download() method to download their data.

For hosted feature layers, you can use one of the following ways to get the data:

1) Use extract_data tool from the features.manage_data module (doc: arcgis.features.manage_data module — arcgis 1.0 documentation). It's easy to use, couple examples:

from arcgis.features.manage_data import extract_data
csv_lyr = gis.content.get('c8bc7d3c3b60415e9845bc00dcd777ed')
outputgdb = extract_data([csv_lyr])
outputgdb.download('C:\\xc')

Out[]: 'C:\\xc\\extract_data_20161025204926.zip'

# you can also specify the format:
outputcsv = extract_data([csv_lyr], data_format='CSV')

2) You can get the related data item for the layer, and download the data item used to publish the layer:

relitems = layer_item.related_items("Service2Data")

relitems[0].download()

Hope this helps,

Rohit

Re: Download Item Data

LesiMai3 — Thu, 29 Dec 2016 16:15:41 GMT

Hi Rohit,

Thank you for the detailed clarification!

I tried the your sample code on a hosted feature layer but got the following error message --

How can I fix it?

Thank you,

Lesi

Re: Download Item Data

RohitSingh2 — Thu, 29 Dec 2016 18:00:05 GMT

Hi Lesi,

I'm not sure why extracting data to a file geodatabase isn't working and the error message isn't very informative.

Please try specifying a different data_format such as Shapefile:

res = extract_data([lyr], data_format='Shapefile')

res.download('C:\\Test')

Thanks,

Rohit

Re: Download Item Data

LesiMai3 — Thu, 29 Dec 2016 18:41:38 GMT

Hi Rohit,

You're right! Changing to a data format other than file geodatabase works. Not sure why though...

Thank you for your help!

Lesi

Re: Download Item Data

LesiMai3 — Wed, 04 Jan 2017 18:04:00 GMT

Hi Rohit,

I have a follow-up question. After I downloaded the data using extract_data, I noticed that file item was added to AGOL. Is that the intended result?

Thank you,

Lesi

Re: Download Item Data

RohitSingh2 — Wed, 04 Jan 2017 18:59:13 GMT

Hi Lesi,

That is the intended result. You can then get the data downloaded to your computer using:

res.download('C:\\Test')

Thanks,

Rohit

Re: Download Item Data

AlistairFox — Fri, 27 Jan 2017 08:00:09 GMT

any idea whey when i use

from arcgis.features.manage_data import extract_data

I get No module name 'arcgis.features'

Re: Download Item Data

Anonymous User — Tue, 31 Jan 2017 18:20:37 GMT

Alistair Fox‌ Can you check if you have upgraded to version 1.0 of the API? The `features` module is new in v1.0 and not found in v0.3 or earlier.

To check, open your terminal (or Python command prompt if you installed using Pro) and type

conda list

and check the version next to `arcgis` package.

Re: Download Item Data

AlistairFox — Wed, 01 Feb 2017 02:58:26 GMT

yes that was it. all sorted now that i am actually on the 1.0 release

apart from the fact the export_data seems to ignore the output_name attribute? anyone got that to work?

outputgdb = extract_data([csv_lyr],output_name="fred")

layer exported is still the default one esri use when you supply nothing

I've tracked through the code and the ExtractData task is definitely getting passed the output name. When i switch to CSV it seems to use the service name (more usable) so I think it seems to be some hard coding inside the extract data task in online that for zipped file gdb outputs it forces to a set name structure. Can anyone confirm?

Re: Download Item Data

BlakeGardiner2 — Thu, 02 Feb 2017 09:19:39 GMT

I can also confirm I'm getting the same problem. Won't export as a file geodatabase, but it will as a shapefile. When i try to specify output as FileGeodatabase, i get "Item does not exist or is inaccessible" which is obviously not the case.

Re: Download Item Data

AhjungKim4 — Mon, 06 Feb 2017 20:45:57 GMT

rsingh-esristaff‌ I am also getting the "Item does not exist or is inaccessible" error when I try to download as file geodataabse. Is this a bug?

Re: Download Item Data

AlistairFox — Tue, 07 Feb 2017 08:12:46 GMT

we have been able to download file geodatabase, but some do fail .At the moment we have a support call in with ESRI. we seem to think it is related to our Survey123 created and published feature classes only.

We have also note that extract_data call also consumes credits ,where as manual export data does not. Anyone know if there is an equivalent that does not consume credits

Re: Download Item Data

AhjungKim4 — Tue, 07 Feb 2017 14:13:32 GMT

I checked our AGOL credit after reading this and it did, in fact, use credit! Glad you mentioned it otherwise I would have continued to test. Thank you!

Re: Download Item Data

PhilLarkin1 — Sun, 12 Dec 2021 10:54:38 GMT

Alistair Fox

You could query the service for the JSON. This doesn't consume credits. The method works best if your feature count is below the return limit of the service.

from arcgis.gis import GIS
from arcgis.features import FeatureLayer
import arcpy
gis = GIS("http://yourROOT.maps.arcgis.com/","User","Pw")
layer = FeatureLayer("https://services1.arcgis.com/oz19jk1209j09j39j/ArcGIS/rest/services/ServiceName/FeatureServer/0",gis)
featureSet = layer.query(where='1=1', out_fields='*')

rawJSON = featureSet.to_json

jsonFile = 'c:\\somelocalpath\example.json'
localFC = 'c:\\somelocalpath\\Example.gdb\\data'
with open(jsonFile, "w") as file:
    file.write(rawJSON)
arcpy.JSONToFeatures_conversion(jsonFile,localFc

Re: Download Item Data

AlistairFox — Sun, 12 Dec 2021 10:54:41 GMT

Hi Phil,

this looks like a really good solution if i can get it to work. 1st issue was it said arcpy didn't contain function

JSONToFeatures_conversion

I was running 10.3 Pro on the machine so I copied across a 10.4 arcpy module and now it complains i'm not running arcpy in a conda environment. 

Which i am, even when i run from the pro python interactive terminal it still complains about the environment

any ideas?

Re: Download Item Data

PhilLarkin1 — Thu, 23 Feb 2017 00:46:44 GMT

It sounds as if you have multiple python environments, neither of which have both arcpy and arcgis packages available.

Here is what I did to run these together:

1)Install Pro 1.4

2)Added arcgis using Pro's package manager

3)Run script from Pro's python installation. In my case: C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe

Please post a new question if this does not work for you.

See screenshot of the package manager:

Re: Download Item Data

AhjungKim4 — Thu, 23 Feb 2017 15:09:37 GMT

Thank you for posting this. I have written this as a back-up solution, but it is not ideal for feature services that have related records and attachment photos. Also, it does not preserve the original OID. Exporting feature service to file geodatabase and downloading it is a workflow supported in ArcGIS Online. But the Python API, which supposed to promote automating ArcGIS tasks did not include this in the API. But the "extract" workflow which is a premium service that cost a credit for 1,000 data points is somehow included in the API.

What's worse is now that I know this method consumes credit, I selectively chose a feature service that only has two data points. But it consumed 15.71 credits per day every time I run the script. It does not reflect the credit consumption rate Esri documented.

I have an Esri Support ticket open for this but have not received a satisfying answer yet.

Re: Download Item Data

PhilLarkin1 — Thu, 23 Feb 2017 16:39:21 GMT

Ahjung Kim‌
You might want to consider creating a replica. However, this might be a premium service.

The arcgis package appears to support this functionality:

arcgis.features.managers module — arcgis 1.0.1 documentation

Re: Download Item Data

AhjungKim4 — Fri, 24 Feb 2017 15:55:25 GMT

That was the answer I got from the Esri Support: create a replica. Below is the answer I got from the Esri Support. I tried it and was able to download with attachments. Thank you. Phil.

In terms of another way to download the data to file geodatabase, without consuming credits, you could create a replica using the Python API and then download. This is the programmatic equivalent of exporting to a file geodatabase from the Item Details page of a service, or creating a replica and manually entering paramters at the REST endpoint. You may want to test manually from the Item Details page first and ensure that it works there; if it doesn't it won't work programmatically either, and confirms our suspicion that there could be a data-specific problem.