AGOL Open Data download error

[AdamMesser1]

We are having problem with the “Full Dataset” download option for certain data layers on our open data site: http://data.mtfwp.opendata.arcgis.com 

When we try to download the shapefile or the spreadsheet, the dataset gets queued, but is never returned.  A specific example currently exhibiting this behavior is our Deer/Elk hunting districts:

http://data.mtfwp.opendata.arcgis.com/datasets/38cef4e1b592416c98acc2308ed33cc3_0

We have tried a few methods to fix the problem. From the Open Data Management page, we’ve tried resetting the dataset and updating the index for the site, neither of which helped.  Deleting and recreating the dataset solved the problem, but only temporarily.  We recreated the dataset on Friday. Full dataset downloads worked for the remainder of the day, but are now failing once again. 

We had also had a separate problem with other datasets in which the Full Dataset download is partial or empty. In the case of the spreadsheet download, we’ve seen empty or partial csvs.  In the case of shapefiles, we have seen zip files that did not include shp, dbf or shx files.  We have replaced the dataset that were exhibiting this behavior, so I don’t have any current examples. 

We have not seen any problem with the “Filtered Dataset” downloads.

Any thoughts on what to do to ensure performance?

Also, does anyone know how long to the full datasets get cached on ESRI’s servers before being cleared?

Courtney Claessens​

[BillDaigle]

Daniel Fenton​

[CourtneyClaessens · Accepted solution]

Hi Adam,

Looking at your site - things seem to be operating normally now. If you experience any other issues with empty downloads please let us know along with a link, and we can investigate.

Regarding dataset caches -

For non-hosted datasets: When a dataset download is requested for the first time, we generate a cache. We keep serving this cached download for 24 hours (unless the administrator clicks the reset button). After 24 hours, when a dataset download is requested, we generate a new cache. This 24-hour rule is not automatic-- if a user does not request a download for a few days the original cache will sit there until a new download is requested.

For hosted datasets: We cache the datasets and the 24 hour rule does not apply, since we know when the dataset has been last updated. When a user first requests the download, we generate a cache. When the user requests a subsequent download, we check to see if the cache is out of date as per the last edited date on the hosted item, and either serve them the old cache or generate a new one.

Hope this helps.

Courtney

[AdamMesser1]

A resolved ticket per this issue indicated that a webserver update last week fixed the problem with downloads hanging. We were unable to replicate the incomplete zip downloads, so assume those issues were also resolved. Thanks for the clarity on caching.