Reinstate option to download a file previously generated in ArcGIS Hub

[MaddieMooreDEQ]

The ability to download previously generated files from the ArcGIS Hub site was a vital feature for many users. For datasets that are updated infrequently, there is often no need for a new download each time the site is accessed. However, now, when hosting data from ArcGIS Server that is shared with the Hub site, some downloads can take an average of 30 minutes to an hour for certain datasets. The option to download previously generated files helped streamline this process and significantly reduces download times.

[DanNarsavage_IDWR]

Did you just notice this today (10 Sept 2024) too?  We're trying to figure out when this changed.  We're also currently having trouble downloading any data from Hub at all.  A service with two records says "Processing..." for dozens of minutes when saving to a shapefile or a CSV.

[JoanDelosSantos]

@DanNarsavage_IDWR @MaddieMooreDEQ This changed around August 13 or 14 with an AGOL update.  Per our Technical Account Manager, Esri is advising those who are experiencing this issue to submit tech support tickets or to contact your Esri team so that they can hear about your specific issues.  I believe that the Hub Dev team is trying to address it.  I think good if they hear from lots of users so that they understand how impactful this change has been.  It has had a profound (negative) impact on the functionality of our open data site.

[DanNarsavage_IDWR]

@JoanDelosSantosthank you.  Maddie's idea was the first thing we found while searching the community this morning.  We saw your comment on the blog post (and other such comments and posts) shortly afterward, which let us know we weren't alone and added to our understanding of the timing of this. We have plans to submit a support ticket ASAP after we scramble to respond to the FOIA request that prompted our discovery of this bug as well as contact the several people that have contacted us looking for data in the past month (whom we referred to the hub unaware that it was broken but who never contacted us back to report an issue). 

And if I may editorialize here just a moment . . .  AAAAAAAARGH!!!!  😄 😄 😄

[DanNarsavage_IDWR]

@JoanDelosSantos @MaddieMooreDEQ I submitted a service request this morning and lo and behold downloads are working again for us.  Are they working for you too? I don't see any movement on BUG-000170515 so I'm just wondering if I can expect whatever changed not to change back.

[NinaRihn]

@JoanDelosSantos @MaddieMooreDEQ @DanNarsavage_IDWR  I just got an update on BUG-000170515:

Status: In Product Plan (Learn More)
Additional Information:  We are working on a production change which will always download the cached file and provide a secondary option to generate a new file if editor tracking is not enabled on the service or if the service does not have the extract capability.

Hopefully they will give the user some indication of how old the cache is, and hopefully a newly generated file will download in a reasonable time.

[BillSchlatter]

@DanNarsavage_IDWRWe're having the same issues. Extremely long file generation processes ending with failure and cryptic error messages 60-70% of the time, even for the tiniest, simplest datasets. I have an active support ticket on the issue. 

Interestingly, one thing we've confirmed with this is that it's not just a Hub issue. One of the testing steps I was asked to do was to export the data via the Item Description page for where the item is hosted on ArcGIS Online. I've been getting similar failure rates, times, and error messages without Hub being in the loop at all.

[KristenWobbe1]

We noticed the first week of September that something had changed with the Aug 13th update and our users experienced workflow disruptions by ESRI taking away 'Data Downloads' for FGDB for Referenced/non-hosted Feature Services, which we have had in place for two years. After meeting with our ESRI Rep, we requested that ESRI restore download functionality for 'non AGOL hosted services'. 

It was our hope that existing capabilities not be removed until the replacement functionality is tested/implemented and in place for Hub users. We have not heard back on answers to our questions regarding the expected timeframe when FGDB would be released back as a download option for 'referenced/non-hosted' services. We did learn the Hub Team was on a 'pause' until Nov. 12th, and with that information we don't expect an immediate change or restore back. 

Some of our very large datasets are failing on the SHP download now. It's nice to hear from all of you that we are not alone, and that many of you have filed remedy tickets, and good to know there is a bug number logged too.

[NinaRihn]

If it helps, a few days ago I got this bug update:   https://support.esri.com/en-us/bug/attempting-to-download-arcgis-enterprise-services-from-bug-000170... 

in my initial testing it seems like certain layers are downloading better (not hanging up in processing at least)... I do need to do more broad testing though, because I don't know what impact there is if there isn't a cache available for large datasets. (or how to tell if there is a cache or how old it is)

[DanNarsavage_IDWR]

Yep, what Nina said (again).  The vast majority of our layers seem to be downloading reasonably again, but at least one of our largest is still seeing troubles.  But we've got bigger fires to fight at the moment.

FWIW, I like the new functionality--downloading an older copy of our large datasets will likely suffice for most people, and that happens really quickly now, which is great.  But I doubt that most people are savvy enough to our data update frequency to *know* whether or not it's worth it for them to request a fresh download (which is a significant time commitment).  I suppose the answer to that issue is, "The year is 2024, people. Just use the freakin' service."  😂

[DanNarsavage_IDWR]

Update:

The one layer that was seeing troubles is a map image layer.  Once we switched to sharing that feature layer through Hub, the problem ceased.  But I submitted another support request.  Below is the meat of it.

We’ve finally gotten a chance to vet the fix for BUG-000170515 more thoroughly, and we found an issue.  I’ll try to detail the steps to reproduce the issue below . . .

1. I clicked “Download” for a shapefile of our Wells layer, and a zip file immediately downloaded.  Yay! 
• But it didn’t contain the correct number of records—the data behind that service gets updated daily, and that shapefile was cached more than a day ago so it was missing a couple hundred new features.
2. So I clicked “Request new file”, and—after about twelve minutes of processing—I got another zip file of the same name.  Yay!
• But the file size was identical to the previously-downloaded zip file.  And sure enough, it contained identical data to the first download.
3. After a few minutes of head-scratching, a coworker visited the same page, clicked “Download” (not “Request new file”), and received a zip file that was slightly larger than the one I received in step 2 above.
• Sure enough, it contained the new number of records. Yay!
4. I revisited the same page and clicked “Download” (not “Request new file”), and was immediately blessed with the same updated file that my coworker received in step 3.

So it looks to us like AGOL correctly cached an updated copy at step 2 (and correctly passed it along in steps 3 & 4), but it handed off the old, cached shapefile to the poor chump who asked for the new stuff and was willing to wait for it.  Can you help?