ArcGIS Hub site download issues

Question

Hi,

We have received reports from end users that some isolated dataset downloads are hanging in some formats on our ArcGIS Hub site (running for >18 hours).  I have replicated this issue locally.  Please investigate.  We have cleared the browser cache and refreshed the content catalogue with no effect.  Is this a widespread known issue or isolated to our ArcGIS Hub site only?  Please note, not all updated overwritten layers have been affected in this month's update cycle.

Home page: https://naturalengland-defra.opendata.arcgis.com

Screenshots below are in the Chrome browser.

See examples below - there may be more?

1. Dataset page: https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::ancient-woodland-england/about

2. Dataset page: https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::national-nature-reserves-england/about

3. Dataset page: https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::england-coast-path-route/about

4. Dataset page: https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::england-coast-path-coastal-margin/about

Kind regards,

Andy Lee

Natural England Open Data Geoportal Service Manager

Natural England, Department for the Environment, Food and Rural Affairs (Defra)

AndrewLee4 · Answer

Hi Patrick,

Thanks for noting that this is not an isolated situation and for supporting me with my comments above.  It's good to hear that our site is not the only one impacted across the ecosystem.  I hope Thomas and team are able to get to the root cause soon.  I may try the beta hosted downloads feature, but I would prefer that the existing capabilities continue to work as-is before exploring these further.

Kind regards,

Andy

AndrewLee4 · Answer

Hi Thomas,

I have followed your guidance and unshared and re-shared all the latest updated items this month with the site Content group.

I have also now officially raised a Technical Support ticket with ESRI UK on myesri.com:

#03163302 ArcGIS Hub Download format issues - Natural England Open Data Geoportal

The ticket has been picked up by Tom Cresswell - he has asked:

To gain some insight into your case, please can you provide the following details:

- Confirm where the datasets are being loaded from.
- A workflow detailing how and when the data was overwritten.
- Confirm the formats that appear to be hanging.
Thank you in advance for providing this information.

I will reply to Tom and ask him to pass this over to you now for further investigation and include the link to this page.

Some items completely regenerated their downloads successfully in all five formats, but some did not.

The following succeeded:

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::england-coast-path-route/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::ancient-woodland-england/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::sites-of-special-scientific-interest-units-england/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::sites-of-special-scientific-interest-england/about

The following items have started to regenerate some of the download formats, but not all the formats have become available yet.  They have been running all day (UK time) and appear to be hanging.  Many of these are critical UK national data products published for use by contractors, partners and public users inside the ESRI platform and outside.

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::national-nature-reserves-england/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::england-coast-path-coastal-margin/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::countryside-stewardship-scheme-2016-management-areas-england/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::countryside-stewardship-scheme-2016-management-options-england/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::environmental-stewardship-scheme-agreements-england/about

https://naturalengland-defra.opendata.arcgis.com/datasets/Defra::environmental-stewardship-scheme-options-england/about

To test if this improves download generation performance, I have turned on the Beta capability "Hosted Downloads" option on the Content item pages for each of the above items.

Batch updates are completed using a Python script which was created/adapted for us by ESRI UK Professional Services team. This reads a .CSV file list of the locally stored ArcMap MXD file-paths, parameters for the AGOL item fields and service definition configurations.  This checks if the service names exist, then generates a locally stored service definition .SD file, then pushes it up to AGOL and overwrites the existing services one by one.  Then some manual changes are made on the AGOL item pages to ensure that all previous configurations are retained as-is. e.g. Map Viewer visualisation, pop-ups etc.  There have been no issues at all with the overwrite scripts or AGOL item configuration, only the ArcGIS Hub downloads.

If you would like to view any of the above items in AGOL please click Open in ArcGIS Online from the bottom right of the ArcGIS Hub item About page.

Kind regards,

Andy

PatrickJankanish · Answer

Hi Thomas,

We've had an Esri support ticket (Esri Case #03111599) open for several weeks regarding the generation of download files on our Open Data site. We've been focusing the troubleshooting on one of our largest datasets, King County parcels (parcel_area). The examples cited by Andy Lee are similar to our experiences with this layer. When we first opened the ticket, choosing to generate a new download file would kick off a process that might never complete or might complete after many hours. Small datasets were not a problem. We did try some similarly large datasets on other Esri Open Data sites, such as the City of Seattle and LA County, and noted that those also exhibited the same behavior, so we thought there was something going wrong on the backend with the system that generates the download files.

If you can access the history on our support ticket, you can see some of the things we've been trying. Over the past week or so we've noted that on some days the download files for parcel_area generate in a matter of minutes for all file formats. On one occasion this week, that was true for some of the file formats, but it took more than 10 times as long for other formats (e.g., 3 minutes vs. 30 minutes). And there have still been instances this week when download file generation appeared to have timed out without producing anything.

Despite some successes, I've been reluctant to agree to close the ticket until we see consistent success.

I don't think the little bit of information I've provided here can help, but I wanted you to know that what Andy Lee has reported is not an isolated situation.

Thanks,
Patrick Jankanish
King County GIS Center