topic Re: Data updated on Open Data/ArcGIS Hub but the downloads are old in ArcGIS Hub Questions

Data updated on Open Data/ArcGIS Hub but the downloads are old

CortDaniel1 — Wed, 27 Mar 2019 22:36:04 GMT

We are hosting data on our Open Data web site. When data is updated, we see data updated on parts of the website. On the website, record counts and features change. But the downloaded data is old.

For example, we updated our Cities data at 12:30 pm. We see the feature that was changed when following the, "Create Webmap" link. The map below was created with webmap, and the highlight area is new.

It has been 2 - 3 hours and the data downloaded is still not current.

The date on the shapefile is old too.

We have refreshed the index for this data several times.

Here is the URL to this data: https://gisdata-piercecowa.opendata.arcgis.com/datasets/cities-in-pierce-county

Is there a way we can refresh all of the data in our Open Data web site at the same time?

Thanks,

Cort

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

CortDaniel1 — Tue, 10 Sep 2019 18:44:57 GMT

We have found a workaround that has been working for about a month.

The workaround is:

Update the data.
Un-share the data.
Wait until the data is no longer available on your Open Data Portal.
Re-share the data.

After taking these steps, the downloaded data is current.

Prior this, the workaround was to republish the data.

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

Anonymous User — Tue, 10 Sep 2019 19:30:14 GMT

Hi Cort,

About a month ago we made some changes to the download caching process so that caches are now generated once Hub detects a change to the service. It's still possible that users will get old downloads if they happen to request a format before the caching job has completed, but this change should resolve your issue if things are working as designed. Could you try not employing your workaround for a few days and see if the problem still occurs?

Thanks,

Patrick

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

CortDaniel1 — Thu, 12 Sep 2019 20:13:04 GMT

Patrick,

I reviewed the ArcGIS Hub change log going back to June. I do not see any mention that the download caching process has changed in the past month. I find the change log entries cryptic and may have missed it (https://hub.arcgis.com/pages/changelog).

Updates to our hosted feature service data happen irregularly, but we will try your suggestion. We will not employ the workaround and see if the problem still occurs.

Please keep in mind that our data is for the Public and Public agencies. There is an expectation that that data downloaded from Open Data is current. I do not feel this is not an unreasonable expectation. The time that the date was updated is posted on Open Data.

If the posted time changed after the cache was refreshed, the problem would be resolved.

Currently, we are experiencing two unrelated problems with FGDB downloads. I posted this on GeoNet, but no one has responded. Perhaps you can look at this one too (https://community.esri.com/thread/236218-open-data-fgdb-not-downloading-after-being-updated).

Thanks,

Cort

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

Anonymous User — Thu, 12 Sep 2019 21:19:01 GMT

No, you didn't miss anything. There are a few pieces to the app and the indexing piece that got the update receives such frequent updates that they often don't make the regular changelog. I hadn't seen your geonet post until your reply above otherwise I would have let you know about the update as soon as it went out.

Let us know if you see any further date discrepancies. Otherwise, see you over onOpen Data FGDB not downloading after being updated.

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

SusanCarlson — Tue, 07 Jan 2020 18:51:16 GMT

We are having this same issue except our services are hosted in local ArcGIS Server. We had data updated today at 6:20 AM and at 1:17 PM the downloaded data is still not updated.

Loudoun Land Use Existing Structures

The data tab shows the correct information:

The search results page shows the correct updated date but the old number of records:

Download Cache it set to automatic. I tried the manual reset button several times to no avail. The data takes seconds to download so I'm assuming it wouldn't take long to update the cache. I waited over an hour after I first clicked to update the cache.

For the two layers I'm currently having an issue with, all the data was replaced with new data, if that matters. We have other frequently downloaded data (addresses, parcels) that is typically updated 4 days a week. Those appear to be fine right now with the last update occurring 3 days ago.

(Similarly, when adding or removing a field in a layer I have to unshare and reshare with Open Data as the workaround stated above for the change to show up correctly.)

Thanks,

Sue

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

Anonymous User — Tue, 07 Jan 2020 20:31:10 GMT

Hi Susan,

Was the 124491 record count accurate as of the first update this morning?

Thanks,

Patrick

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

SusanCarlson — Tue, 07 Jan 2020 20:51:36 GMT

Hi Patrick,

The old record count was 124491 before this morning’s update (6:20 am Eastern) to 128668 records.

Thanks.

Susan Carlson, GISP

GIS Web Administrator

Loudoun County, Virginia

Office of Mapping & Geographic Information

1 Harrison St SE | PO Box 7000 | Leesburg VA 20177-7000

T: 703-771-5949 | M: 703-936-1240

susan.carlson@loudoun.gov<mailto:susan.carlson@loudoun.gov>

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

Anonymous User — Tue, 07 Jan 2020 22:10:59 GMT

Hi Susan,

Ok thanks. More can be found in our faq (FAQs—ArcGIS Hub | Documentation), but for on premises services, it can take up to 24 hours for updates to be reflected in downloads. It appears that a download request came in early this morning (~4am) which triggered a new cache. In the coming weeks we're adding the ability to manually refresh the download cache but for now check tomorrow and let us know if the issue persists.

Best,

Patrick

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

SusanCarlson — Tue, 07 Jan 2020 23:31:01 GMT

Thank you, Patrick!

Sue

GIS Web Administrator

Office of Mapping & Geographic Information

703-771-5949

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

SusanCarlson — Wed, 08 Jan 2020 14:50:15 GMT

Good morning Patrick,

The download data is good now.

Do you anticipate that the manual cache refresh will be able to be automated via script? We are a fast growing county and our data changes frequently.

Thanks, again!

Sue

GIS Web Administrator

Office of Mapping & Geographic Information

703-771-5949

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

Anonymous User — Wed, 08 Jan 2020 15:01:26 GMT

Glad to hear the download refresh came through. We're planning to add explicit control over download refresh in the UI in the next few months so you can set when the download cache is refreshed. We could open this up to scripting, but we're hoping adding a spot for this in the app itself will fully address the pain point of not having control over when caches are made.

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

NinaRihn — Fri, 26 Jun 2020 12:56:55 GMT

Patrick, has the cache control mechanism been added yet, or are you close to getting that?

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

ThomasHervey1 — Fri, 26 Jun 2020 18:10:31 GMT

Hi Nina,

We're hoping to release a new download UI within the next few weeks. After that, we will evaluate what steps need to be taken to add a cache control mechanism.

The likely first step will be to add an option under the more menu '...' when on a dataset's edit route (datasets/:id/edit) to re-cache a download. You can expect to see that feature in the next few weeks. Recent changes in our API has made it easier to control downloads without going through the UI. So, after the re-caching option is in place in the Hub UI, we plan to provide a way for authenticated users to re-cache downloads using our API.

Re: Data updated on Open Data/ArcGIS Hub but the downloads are old

ThomasKonzel — Wed, 20 Apr 2022 15:50:45 GMT

Has this manual cache process been implemented? We get emails asking for updated data when the data on our Open Data Portal Hub site is up to date. It's just that the reported data isn't accurate. It's very frustrating for years now, to have to explain to users that the data is up to date. If you're not able to create a process to manually cache the data, then AT LEAST give us a check box to remove any reported data due to inaccuracies.