topic Open Data downloads getting stuck in ArcGIS Hub Questions

Open Data downloads getting stuck

MelissaBrooks — Mon, 30 Sep 2019 20:57:15 GMT

Hi there,

We are experiencing issues with (any type) downloads either

- "Gathering Data" continuously

- Taking a very long time

- Taking a very long time, then the second time "Gathering Data" continuously

It sounds very similar to this post,

What does Gathering Data mean

Can you please check this issue has not come back since the recent updates

Thanks

Melissa

New Plymouth District Council

New Zealand

Info: Our new open data portal is not yet ready for the public, but I can provide information if necessary.

Re: Open Data downloads getting stuck

HemantAngarkar1 — Tue, 01 Oct 2019 05:38:33 GMT

To troubleshoot initially;

1. Is it happening with particular dataset/s or all of them?

2. If it is particular dataset/s, try dropping cache (Manual Reset). Rebuilding cache may take some time which depends on how big/small corresponding feature service is (number of features).

3. If it is still not working try un-sharing and re-sharing the concerned item/s from Public and Open Data group.

4. If it is still the issue, try adding same Feature Service again to ArcGIS Online using Service URL with a different Name >> Share it with Public and Open data group. If it works then there is probable corruption of the item.

5. If it doesn't work create a new Open Data Group and try sharing services with that group and add that group to your site to see if issue is group specific.

This can go on, but if you can check few things for further troubleshooting;

1. How big are the datasets in question (How many features)?

2. Since when issue occurred? Did anything changed?

Hope this helps!

Re: Open Data downloads getting stuck

MelissaBrooks — Tue, 01 Oct 2019 19:44:19 GMT

Thanks for this list Hemant Angarkar‌ we will work through this, a logical order.

This issue is inconsistent, one item will work (after a long while) then next time it wont work (continues to try but never finishes) - or vice versa

Polylines tend to be more succeptable than points, but point layers still have issues.

Ranges from very small (4-30) up to med/large (20-40,000 features.)

No more than 20 layers per service as per recommendation.

Began this week, but we think we may have been spot checking last week, it may just be that we needed to focus on some of the more complex layers.

I'll go through this checklist today so I can eliminate any problem with the item and/or open data sharing and provide an update. Thanks again for your feedback

Appreciated!

Melissa

Re: Open Data downloads getting stuck

HemantAngarkar1 — Thu, 03 Oct 2019 05:00:02 GMT

You are welcome Melissa Brooks‌. Point of all those steps is to find out where issue is. Also I missed in my last post;

1. If services are hosted on ArcGIS for Server then try uploading same data as a service to ArcGIS Online and see if that has issues.

2. If Both Server and AGOL services has issues then I think it may the data that is causing it.Run Check Geometry to see if it has issues and run Repair Geometry to fix those issues. If no geometry issues then it might be having Complex Geometries that is too many vertices per feature (for line and polygon geometries obviously). Run Simplify tools can address the issues but they will slightly modify shape.

3. If it is only ArcGIS for Server services, try setting the Download Cache to "Manual" from Data Manager page for that dataset.

4. If that fixes the issue then I would recommend to check the Machine's (Where ArcGIS for Server is installed) resource utilization (CPU and RAM) from Task Manager while you hit the refresh cache button from Data Manager page for that dataset. If you see spike in Resource utilization then you have to work with your ArcGIS Admin and IT to address that issue(Also involving technical support would be good idea in this case).

After typing all that I am just basing these troubleshooting steps on the basis of my imagination of the issue. I think if you won't get it resolved after working through all the steps, going to technical support will be ideal where they can take a look at the issue in a better way.

Hope that helps!

Re: Open Data downloads getting stuck

KarolFallas — Fri, 04 Oct 2019 22:17:57 GMT

Hi,

We have the same issue. It happen recently.

Thanks

Karol Fallas

Municipalidad de Heredia

Costa Rica

Re: Open Data downloads getting stuck

HemantAngarkar1 — Mon, 07 Oct 2019 05:32:07 GMT

Hello,

Did you try steps mentioned above? Let me know if you need any further explanation.

Re: Open Data downloads getting stuck

HemantAngarkar1 — Mon, 07 Oct 2019 05:52:39 GMT

Please provide link to the dataset with issue if site is public.

Re: Open Data downloads getting stuck

KarolFallas — Mon, 07 Oct 2019 13:17:18 GMT

Hi,

This is an example:

Ubica, Municipalidad de Heredia

Days before it was working.

Thanks

http://ubica-herediasig.opendata.arcgis.com/datasets/zonas-homog%C3%A9neas

Re: Open Data downloads getting stuck

HemantAngarkar1 — Wed, 09 Oct 2019 05:33:22 GMT

Karol Fallas‌

Download worked for me today. I Think it was temporary. Try setting the Download Cache to "Manual" from Data Manager page for that dataset if dataset is not upadted frequently on Server.

Re: Open Data downloads getting stuck

KarolFallas — Wed, 09 Oct 2019 16:20:24 GMT

Hi,

Yes it seems like temporary, one day some data works and others don't.

I changed Download Cache to "Manual" but after a few seconds is retorned to automatic.

Re: Open Data downloads getting stuck

HemantAngarkar1 — Thu, 10 Oct 2019 05:02:41 GMT

Hello,

It shouldn't go back to automatic. Try Incognito mode in Google Chrome to do the change, browser cache sometimes causes these kinds of things. Get in touch with Technical support if you still find it problematic.

Re: Open Data downloads getting stuck

IvanBrown — Fri, 11 Oct 2019 14:08:18 GMT

We are also suddenly experiencing stuck downloads (w/ underlying data that hasn't changed).

The Data Manager--where you go to reset download caches--is suddenly missing. I would go there to try fixes like re-indexing the site or resetting caches; it's gone.

Re: Open Data downloads getting stuck

Todd_Metzler — Fri, 11 Oct 2019 16:40:59 GMT

Hello Esri Hub Administrators (Esri Technical Staff. Not customer Hub Site Administrators like me),

Please prioritize determining the cause of this problem and fixing in very, very soon. Our team has already expended over 16 human hours troubleshooting these problems ourselves.

I am customer 497 and my Esri case is #02419756.

Disappointed that Esri status shows no outages, conditions normal. Clearly that is not the case.

Todd M

ArcGIS Enterprise Administrator

Re: Open Data downloads getting stuck

Anonymous User — Fri, 11 Oct 2019 20:17:45 GMT

Hi all,

First off, apologies for the download issues over the past week. We observed a significant outage that caused much of the headaches you've experienced.

Over the span of about a week (10/1 – 10/8) the Hub backend experienced substantial delays with dataset indexing and download file preparation. We lost a substantial amount of cached files in our backend, requiring a rebuild across the Hub platform. Rebuilding the cache for all the content across the platform takes significant time. While the system is recovering you may experience unexpected behavior due to indexing failures or download caching failures that occurred during the noted period. We expect these issues to clear themselves in the coming days and will continue to monitor.

We agree that a lack of messaging and error reporting compounded the issues this week, which is why we're investing development energies into better error reporting and response. In addition to adding error display and management features to the application, we will be adding a status page to Hub in the coming weeks. It is our hope that this will help make sure you not only see what's wrong before your end users do, but that you don't waste valuable hours troubleshooting issues in vain.

Again, we apologize for the inconvenience and frustration this caused you. We're working to make sure it doesn't happen again.

Best,

Patrick, Hub Product Engineer

Re: Open Data downloads getting stuck

Anonymous User — Wed, 16 Oct 2019 18:22:55 GMT

Hi Patrick Hammons,

Thank you for recognizing the lack of communication around this issue.

I wholeheartedly support your planned investment into developing better error communication, reporting, and response tools similar to how ArcGIS Online does it with their status page (ArcGIS Online Health Dashboard).

Best,

Lucas, Open Data Site (HUB) User & Administrator

Re: Open Data downloads getting stuck

Todd_Metzler — Thu, 17 Oct 2019 12:46:33 GMT

2019-10-17T08:41:00(-0400)

Hello Again Esri Hub Administrators (Esri Technical Staff. Not customer Hub Site Administrators like me),

Hub behavior is inconsistent and unsatisfactory again today. My open data catalog <Maine GeoLibrary >doesn't even show content and hub site landing page fails to load about 50% of the time<ArcGIS Hub > .

All green <ArcGIS Online Health Dashboard >.

What do I tell my customers now?

My Esri case is comeing your way.

Todd Metzler

State of Maine

Enterprise GIS Platform Administrator

Re: Open Data downloads getting stuck

Anonymous User — Thu, 17 Oct 2019 15:20:16 GMT

Hi Todd,

Sorry to hear you're continuing to have technical issues. Unfortunately I'm not able to reproduce as your site looks fully functional on my end. If this continues to be an issue I would definitely recommend opening a support ticket with My Esri.

As mentioned above, Hub is not integrated with the ArcGIS Online dashboard. We are working to add a status page but this work is forthcoming.

Patrick