topic Re: Hosted shapefile downloads for large datasets not working or working very slowly in ArcGIS Hub Questions

Hosted shapefile downloads for large datasets not working or working very slowly

JOY_detroit — Mon, 15 Apr 2024 15:23:57 GMT

We noticed that the few datasets that use the new Hosted Downloads on our open data portal are downloading as shapefiles much slower (or even not at all) compared to other datasets using the original download method. All datasets have the proper "Extract" settings configured.

For example: our Parcel dataset downloads quickly as a CSV, GeoJSON, File Geodatabse, etc. However, it will not download as a shapefile.

When inspecting the network calls, the following responses appear:

{
"replicaName": "",
"replicaID": "",
"submissionTime": 1713194227870,
"lastUpdatedTime": 1713194227997,
"status": "ProvisioningReplica",
"recordCount": 0,
"resultUrl": ""
}

A few second later....

{
"replicaName": "",
"replicaID": "",
"submissionTime": 1713194227870,
"lastUpdatedTime": 1713194228057,
"status": "ExportingData",
"recordCount": 0,
"resultUrl": ""
}

A few more seconds later...

{
"replicaName": "",
"replicaID": "",
"submissionTime": 1713194227870,
"lastUpdatedTime": 1713194233507,
"status": "ExportingData",
"recordCount": 500,
"resultUrl": ""
}

A few more seconds later...

{
"replicaName": "",
"replicaID": "",
"submissionTime": 1713194227870,
"lastUpdatedTime": 1713194238980,
"status": "ExportingData",
"recordCount": 1000,
"resultUrl": ""
}

And so forth. Refreshing the page and trying again resets everything.

Given there are 380K records, you can see where this doesn't work.

Has anyone else dealt with this problem or noticed it?

Re: Hosted shapefile downloads for large datasets not working or working very slowly

George_Thompson — Mon, 15 Apr 2024 16:44:51 GMT

I am wondering if there is a "size" issue with the shapefile. Shapefiles have a 2GB size limit.; https://desktop.arcgis.com/en/arcmap/latest/manage-data/shapefiles/geoprocessing-considerations-for-shapefile-output.htm#:~:text=Shapefile%20extensions-,Geometry%20limitations,roughly%2070%20million%20point%20features.

Can you export the feature as a shapefile from the downloaded file geodatabase?

If so, how big is the entire (all the) files?

Here is another post on this: https://community.esri.com/t5/data-management-questions/export-large-shapefile/td-p/1092341

Re: Hosted shapefile downloads for large datasets not working or working very slowly

JOY_detroit — Mon, 15 Apr 2024 17:14:28 GMT

I don't think it is a size issue.... according to the AGO item, the size is just under 500mb. A previous download of the shapefile from about 2 months ago (same number of rows) was 352 MB. It was working fine, as far as we know, up until mid to late last week.

Re: Hosted shapefile downloads for large datasets not working or working very slowly

KatieMorgan — Tue, 11 Jun 2024 21:23:50 GMT

Were you able to find a solution for this in the end? I'm experiencing the same issue with shapefile downloads on a large dataset.

Re: Hosted shapefile downloads for large datasets not working or working very slowly

JOY_detroit — Wed, 12 Jun 2024 12:17:46 GMT

I talked with ESRI support. They verified it was an issue and as of right now are still looking into it. There has been no update to the case since 5/22/2024. If you want to open your own issue, they assigned our issue bug number: BUG-000167697. You may be able to reference it to expedite support, though I am not sure.

In the meantime we turned of the Extract capabilities in the settings of our large AGO layers, and it reverts back to the old interface for downloading those items on our Open Data Portal.

Re: Hosted shapefile downloads for large datasets not working or working very slowly

KatieMorgan — Wed, 12 Jun 2024 15:10:12 GMT

Brilliant thank you, I'll include your bug reference in my case request to Esri. Will post here if anything is resolved.