Reinstate option to download a file previously generated in ArcGIS Hub

[MaddieMooreDEQ]

The ability to download previously generated files from the ArcGIS Hub site was a vital feature for many users. For datasets that are updated infrequently, there is often no need for a new download each time the site is accessed. However, now, when hosting data from ArcGIS Server that is shared with the Hub site, some downloads can take an average of 30 minutes to an hour for certain datasets. The option to download previously generated files helped streamline this process and significantly reduces download times.

[MaddieMooreDEQ]

Hey folks @JoanDelosSantos @MaddieMooreDEQ @DanNarsavage_IDWR 

I saw the update for September 27th: https://hub.arcgis.com/pages/changelog

While for me this does fix the download speed issue it is super upsetting there is no information on when the last download file was generated. Also, having to first download the file and then the option to request an updated version appearing is not a clear workflow for anyone that uses hub. 

I guess I will just have to add it to my list of things to do to ensure content library updates happen on a weekly basis and add a note in the site that all data is updated on a particular day every week. 

[DanelleMalget1]

Hi all, I have also been digging in on this slowness issue and I have found the date of these 'cached download files' - or so it seems based on my data update schedules! It would be great to display these dates for the user so they can confidently choose to spend the time generating a new file or not (this may be coming in the future if I remember my blog reading correctly!) 

After I saw the 9/27 changelog and tried my large file downloads again with developer tools/f12 running to capture the web request related to download generation, I was reminded of the location in the hub api where these cached files are stored or catalogued at a minimum.  Swap in your itemid and the layer number for mine (67d000cb0ca84517a2145749356172f0_2) and check 'prettyprint' if in Chrome - and you'll see a list of the cached download links, along with their datetimestamp they were generated. The 'links' property at the end of each 'format' section is the download link provided to the hub users representing the cached download file. 

https://hub.arcgis.com/api/v3/datasets/67d000cb0ca84517a2145749356172f0_2/downloads

To address what @DanNarsavage_IDWR noticed about the old file being sent with a 'Request New File' is a bit peculiar, because I would guess that the 'Request New File' operation would overwrite the cached file and the 'lastModified' date would update. It would be pure speculation, but i could see something like cached download being presented to the browser for download without the user noticing, then the 'Request New File' button was clicked and the browser still had the old one ready to go! Coming back to that later would have given a bit of time for that cached file to have been updated! Perhaps looking at that modified date while repeating this process would be informative!! 

(Also - note that fgdb downloads are still being generated on the backend and accessible from the v3 api even though they are not displayed on the hub download gui - at least for the time being!)

[DanelleMalget1]

Followup on my speculation on how @DanNarsavage_IDWR different download file versions might have happened... I just tested a new file generation for shp on one of our larger datasets that hadn't been downloading successfully, and we now have it on enterprise server 11.1 so I wanted to see if it would now download. 

During the download operation, i had the dev tools open, and was also watching the hub.arcgis.com/api/v3/datasets/itemid_lyrnum/downloads page i mentioned in my last post. A couple of things I noticed that might be informative for timing and version of resulting files: 

1. When i click download for SHP in the gui, it seems like all of the types are getting updated and prepared for download, not just the shp I wanted to download

2. The FGDB option - that is not displayed on the hub gui right now to access easily - seemed to finish its update in 6 min tops, and I waited for about another hour for the other file types to finish (my previously cached files won't actually have any changes to capture in this test - no updates have happened, so i can't compare results to confirm).

3. After about 55 min, I did get the hub gui to present a download file to my browser for the shapefile - yeah... BUT - there was evidence on the download page for shp specifically that it had reset to 10% progress on converting and did not indicate it was ready to download! This feels suspicious that the user could get a download file in the gui while the api downloads page seems to still be processing a download. And sure enough, about 10 minutes later, the api downloads page showed shp was finished - so i captured another download (which i would assume to contain the new content - if there were any). 

My main goal for this test was to just generate a successful new download (as there is no new data to capture) but i happened to notice this potential delay in the api finishing the file generation AFTER the user has been given a file to download... curious. Thoughts? 

[DanNarsavage_IDWR]

@DanelleMalget1you've already put orders of magnitude more thought & effort into this than I have.  😛  Thank you for all your work & info!  You might be right that the download simply doesn't wait for the new file to be created rather than what I was speculating. For my part, I'm waiting on Esri to troubleshoot their own stuff--Esri employs people who are far better at using browsers' dev tools than I am.  😄  Plus I'm feeling more strongly all the time that our users should just use our services directly if up-to-date data is a priority for them, and using the services also circumvents other issues like those raised by Maddie & others.

FWIW, I haven't heard back from Esri support about the last thing I posted, so I'll have to reach out with a NEW ticket rather than replying to the old one.  Sigh . . .