nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Reinstate option to download a file previously generated in ArcGIS Hub

maddie.moore_VADEQ

The ability to download previously generated files from the ArcGIS Hub site was a vital feature for many users. For datasets that are updated infrequently, there is often no need for a new download each time the site is accessed. However, now, when hosting data from ArcGIS Server that is shared with the Hub site, some downloads can take an average of 30 minutes to an hour for certain datasets. The option to download previously generated files helped streamline this process and significantly reduces download times.

Find more posts tagged with

downloads

Community Management

data management

Status: Open

Comments

DanNarsavage_IDWR

Did you just notice this today (10 Sept 2024) too? We're trying to figure out when this changed. We're also currently having trouble downloading any data from Hub at all. A service with two records says "Processing..." for dozens of minutes when saving to a shapefile or a CSV.

JoanDelosSantos

@DanNarsavage_IDWR @MaddieMooreDEQ This changed around August 13 or 14 with an AGOL update. Per our Technical Account Manager, Esri is advising those who are experiencing this issue to submit tech support tickets or to contact your Esri team so that they can hear about your specific issues. I believe that the Hub Dev team is trying to address it. I think good if they hear from lots of users so that they understand how impactful this change has been. It has had a profound (negative) impact on the functionality of our open data site.

DanNarsavage_IDWR

@JoanDelosSantosthank you. Maddie's idea was the first thing we found while searching the community this morning. We saw your comment on the blog post (and other such comments and posts) shortly afterward, which let us know we weren't alone and added to our understanding of the timing of this. We have plans to submit a support ticket ASAP after we scramble to respond to the FOIA request that prompted our discovery of this bug as well as contact the several people that have contacted us looking for data in the past month (whom we referred to the hub unaware that it was broken but who never contacted us back to report an issue).

And if I may editorialize here just a moment . . . AAAAAAAARGH!!!! 😄 😄 😄

DanNarsavage_IDWR

@JoanDelosSantos @MaddieMooreDEQ I submitted a service request this morning and lo and behold downloads are working again for us. Are they working for you too? I don't see any movement on BUG-000170515 so I'm just wondering if I can expect whatever changed not to change back.

NinaRihn

@JoanDelosSantos @MaddieMooreDEQ @DanNarsavage_IDWR I just got an update on BUG-000170515:

Status: In Product Plan (Learn More)
Additional Information: We are working on a production change which will always download the cached file and provide a secondary option to generate a new file if editor tracking is not enabled on the service or if the service does not have the extract capability.

Hopefully they will give the user some indication of how old the cache is, and hopefully a newly generated file will download in a reasonable time.

BillSchlatter

@DanNarsavage_IDWRWe're having the same issues. Extremely long file generation processes ending with failure and cryptic error messages 60-70% of the time, even for the tiniest, simplest datasets. I have an active support ticket on the issue.

Interestingly, one thing we've confirmed with this is that it's not just a Hub issue. One of the testing steps I was asked to do was to export the data via the Item Description page for where the item is hosted on ArcGIS Online. I've been getting similar failure rates, times, and error messages without Hub being in the loop at all.

KristenWobbe1

We noticed the first week of September that something had changed with the Aug 13th update and our users experienced workflow disruptions by ESRI taking away 'Data Downloads' for FGDB for Referenced/non-hosted Feature Services, which we have had in place for two years. After meeting with our ESRI Rep, we requested that ESRI restore download functionality for 'non AGOL hosted services'.

It was our hope that existing capabilities not be removed until the replacement functionality is tested/implemented and in place for Hub users. We have not heard back on answers to our questions regarding the expected timeframe when FGDB would be released back as a download option for 'referenced/non-hosted' services. We did learn the Hub Team was on a 'pause' until Nov. 12th, and with that information we don't expect an immediate change or restore back.

Some of our very large datasets are failing on the SHP download now. It's nice to hear from all of you that we are not alone, and that many of you have filed remedy tickets, and good to know there is a bug number logged too.

NinaRihn

If it helps, a few days ago I got this bug update: https://support.esri.com/en-us/bug/attempting-to-download-arcgis-enterprise-services-from-bug-000170515

in my initial testing it seems like certain layers are downloading better (not hanging up in processing at least)... I do need to do more broad testing though, because I don't know what impact there is if there isn't a cache available for large datasets. (or how to tell if there is a cache or how old it is)

DanNarsavage_IDWR

Yep, what Nina said (again). The vast majority of our layers seem to be downloading reasonably again, but at least one of our largest is still seeing troubles. But we've got bigger fires to fight at the moment.

FWIW, I like the new functionality--downloading an older copy of our large datasets will likely suffice for most people, and that happens really quickly now, which is great. But I doubt that most people are savvy enough to our data update frequency to *know* whether or not it's worth it for them to request a fresh download (which is a significant time commitment). I suppose the answer to that issue is, "The year is 2024, people. Just use the freakin' service." 😂

DanNarsavage_IDWR

Update:

The one layer that was seeing troubles is a map image layer. Once we switched to sharing that feature layer through Hub, the problem ceased. But I submitted another support request. Below is the meat of it.

We’ve finally gotten a chance to vet the fix for BUG-000170515 more thoroughly, and we found an issue. I’ll try to detail the steps to reproduce the issue below . . .

I clicked “Download” for a shapefile of our Wells layer, and a zip file immediately downloaded. Yay!

But it didn’t contain the correct number of records—the data behind that service gets updated daily, and that shapefile was cached more than a day ago so it was missing a couple hundred new features.

So I clicked “Request new file”, and—after about twelve minutes of processing—I got another zip file of the same name. Yay!

But the file size was identical to the previously-downloaded zip file. And sure enough, it contained identical data to the first download.

After a few minutes of head-scratching, a coworker visited the same page, clicked “Download” (not “Request new file”), and received a zip file that was slightly larger than the one I received in step 2 above.

Sure enough, it contained the new number of records. Yay!

I revisited the same page and clicked “Download” (not “Request new file”), and was immediately blessed with the same updated file that my coworker received in step 3.

So it looks to us like AGOL correctly cached an updated copy at step 2 (and correctly passed it along in steps 3 & 4), but it handed off the old, cached shapefile to the poor chump who asked for the new stuff and was willing to wait for it. Can you help?

maddie.moore_VADEQ

Hey folks @JoanDelosSantos @MaddieMooreDEQ @DanNarsavage_IDWR

I saw the update for September 27th: https://hub.arcgis.com/pages/changelog

While for me this does fix the download speed issue it is super upsetting there is no information on when the last download file was generated. Also, having to first download the file and then the option to request an updated version appearing is not a clear workflow for anyone that uses hub.

I guess I will just have to add it to my list of things to do to ensure content library updates happen on a weekly basis and add a note in the site that all data is updated on a particular day every week.

DanelleMalget1

Hi all, I have also been digging in on this slowness issue and I have found the date of these 'cached download files' - or so it seems based on my data update schedules! It would be great to display these dates for the user so they can confidently choose to spend the time generating a new file or not (this may be coming in the future if I remember my blog reading correctly!)

After I saw the 9/27 changelog and tried my large file downloads again with developer tools/f12 running to capture the web request related to download generation, I was reminded of the location in the hub api where these cached files are stored or catalogued at a minimum. Swap in your itemid and the layer number for mine (67d000cb0ca84517a2145749356172f0_2) and check 'prettyprint' if in Chrome - and you'll see a list of the cached download links, along with their datetimestamp they were generated. The 'links' property at the end of each 'format' section is the download link provided to the hub users representing the cached download file.

https://hub.arcgis.com/api/v3/datasets/67d000cb0ca84517a2145749356172f0_2/downloads

To address what @DanNarsavage_IDWR noticed about the old file being sent with a 'Request New File' is a bit peculiar, because I would guess that the 'Request New File' operation would overwrite the cached file and the 'lastModified' date would update. It would be pure speculation, but i could see something like cached download being presented to the browser for download without the user noticing, then the 'Request New File' button was clicked and the browser still had the old one ready to go! Coming back to that later would have given a bit of time for that cached file to have been updated! Perhaps looking at that modified date while repeating this process would be informative!!

(Also - note that fgdb downloads are still being generated on the backend and accessible from the v3 api even though they are not displayed on the hub download gui - at least for the time being!)

DanelleMalget1

Followup on my speculation on how @DanNarsavage_IDWR different download file versions might have happened... I just tested a new file generation for shp on one of our larger datasets that hadn't been downloading successfully, and we now have it on enterprise server 11.1 so I wanted to see if it would now download.

During the download operation, i had the dev tools open, and was also watching the hub.arcgis.com/api/v3/datasets/itemid_lyrnum/downloads page i mentioned in my last post. A couple of things I noticed that might be informative for timing and version of resulting files:

1. When i click download for SHP in the gui, it seems like all of the types are getting updated and prepared for download, not just the shp I wanted to download

2. The FGDB option - that is not displayed on the hub gui right now to access easily - seemed to finish its update in 6 min tops, and I waited for about another hour for the other file types to finish (my previously cached files won't actually have any changes to capture in this test - no updates have happened, so i can't compare results to confirm).

3. After about 55 min, I did get the hub gui to present a download file to my browser for the shapefile - yeah... BUT - there was evidence on the download page for shp specifically that it had reset to 10% progress on converting and did not indicate it was ready to download! This feels suspicious that the user could get a download file in the gui while the api downloads page seems to still be processing a download. And sure enough, about 10 minutes later, the api downloads page showed shp was finished - so i captured another download (which i would assume to contain the new content - if there were any).

My main goal for this test was to just generate a successful new download (as there is no new data to capture) but i happened to notice this potential delay in the api finishing the file generation AFTER the user has been given a file to download... curious. Thoughts?

DanNarsavage_IDWR

@DanelleMalget1you've already put orders of magnitude more thought & effort into this than I have. 😛 Thank you for all your work & info! You might be right that the download simply doesn't wait for the new file to be created rather than what I was speculating. For my part, I'm waiting on Esri to troubleshoot their own stuff--Esri employs people who are far better at using browsers' dev tools than I am. 😄 Plus I'm feeling more strongly all the time that our users should just use our services directly if up-to-date data is a priority for them, and using the services also circumvents other issues like those raised by Maddie & others.

FWIW, I haven't heard back from Esri support about the last thing I posted, so I'll have to reach out with a NEW ticket rather than replying to the old one. Sigh . . .