Select to view content in your preferred language

Data updated on Open Data/ArcGIS Hub but the downloads are old

4395
14
03-27-2019 03:36 PM
CortDaniel1
Regular Contributor

We are hosting data on our Open Data web site.  When data is updated, we see data updated on parts of the website.  On the website, record counts and features change.   But the downloaded data is old. 

For example, we updated our Cities data at 12:30 pm.  We see the feature that was changed when following the, "Create Webmap" link.  The map below was created with webmap, and the highlight area is new.  

It has been 2 - 3 hours and the data downloaded is still not current.  

The date on the shapefile is old too.  

We have refreshed the index for this data several times.  

Here is the URL to this data: https://gisdata-piercecowa.opendata.arcgis.com/datasets/cities-in-pierce-county 

Is there a way we can refresh all of the data in our Open Data web site at the same time?


Thanks, 

Cort

14 Replies
CortDaniel1
Regular Contributor

We have found a workaround that has been working for about a month.  

The workaround is:

  1. Update the data.
  2. Un-share the data. 
  3. Wait until the data is no longer available on your Open Data Portal.
  4. Re-share the data.


After taking these steps, the downloaded data is current. 

Prior this, the workaround was to republish the data.  

0 Kudos
by Anonymous User
Not applicable

Hi Cort,

About a month ago we made some changes to the download caching process so that caches are now generated once Hub detects a change to the service. It's still possible that users will get old downloads if they happen to request a format before the caching job has completed, but this change should resolve your issue if things are working as designed. Could you try not employing your workaround for a few days and see if the problem still occurs?

Thanks,

Patrick

CortDaniel1
Regular Contributor

Patrick,

I reviewed the ArcGIS Hub change log going back to June.  I do not see any mention that the download caching process has changed in the past month.  I find the change log entries cryptic and may have missed it (https://hub.arcgis.com/pages/changelog). 

Updates to our hosted feature service data happen irregularly, but we will try your suggestion.  We will not employ the workaround and see if the problem still occurs. 

Please keep in mind that our data is for the Public and Public agencies.  There is an expectation that that data downloaded from Open Data is current.  I do not feel this is not an unreasonable expectation.  The time that the date was updated is posted on Open Data. 

 

If the posted time changed after the cache was refreshed, the problem would be resolved.

Currently, we are experiencing two unrelated problems with FGDB downloads.  I posted this on GeoNet, but no one has responded.  Perhaps you can look at this one too (https://community.esri.com/thread/236218-open-data-fgdb-not-downloading-after-being-updated). 

 

Thanks,

 

Cort

0 Kudos
by Anonymous User
Not applicable

No, you didn't miss anything. There are a few pieces to the app and the indexing piece that got the update receives such frequent updates that they often don't make the regular changelog. I hadn't seen your geonet post until your reply above otherwise I would have let you know about the update as soon as it went out. 

Let us know if you see any further date discrepancies. Otherwise, see you over onOpen Data FGDB not downloading after being updated. 

0 Kudos
SusanCarlson
Occasional Contributor

We are having this same issue except our services are hosted in local ArcGIS Server. We had data updated today at 6:20 AM and at 1:17 PM the downloaded data is still not updated.

Loudoun Land Use Existing Structures 

The data tab shows the correct information:

 

The search results page shows the correct updated date but the old number of records:

Download Cache it set to automatic. I tried the manual reset button several times to no avail. The data takes seconds to download so I'm assuming it wouldn't take long to update the cache. I waited over an hour after I first clicked to update the cache. 

For the two layers I'm currently having an issue with, all the data was replaced with new data, if that matters. We have other frequently downloaded data (addresses, parcels) that is typically updated 4 days a week. Those appear to be fine right now with the last update occurring 3 days ago.

(Similarly, when adding or removing a field in a layer I have to unshare and reshare with Open Data as the workaround stated above for the change to show up correctly.)

Thanks,

Sue

0 Kudos
by Anonymous User
Not applicable

Hi Susan,

Was the 124491 record count accurate as of the first update this morning?

Thanks,

Patrick

0 Kudos
SusanCarlson
Occasional Contributor

Hi Patrick,

The old record count was 124491 before this morning’s update (6:20 am Eastern) to 128668 records.

Thanks.

Susan Carlson, GISP

GIS Web Administrator

Loudoun County, Virginia

Office of Mapping & Geographic Information

1 Harrison St SE | PO Box 7000 | Leesburg VA 20177-7000

T: 703-771-5949 | M: 703-936-1240

susan.carlson@loudoun.gov<mailto:susan.carlson@loudoun.gov>

0 Kudos
by Anonymous User
Not applicable

Hi Susan,

Ok thanks. More can be found in our faq (FAQs—ArcGIS Hub | Documentation), but for on premises services, it can take up to 24 hours for updates to be reflected in downloads. It appears that a download request came in early this morning (~4am) which triggered a new cache. In the coming weeks we're adding the ability to manually refresh the download cache but for now check tomorrow and let us know if the issue persists.

Best,

Patrick

0 Kudos
SusanCarlson
Occasional Contributor

Thank you, Patrick!

Sue

GIS Web Administrator

Office of Mapping & Geographic Information

703-771-5949

0 Kudos