topic Re: Auto Update of Datasets on OpenData in ArcGIS Hub Questions

Auto Update of Datasets on OpenData

CraigSwadner — Thu, 02 Mar 2017 18:52:51 GMT

We have had several issues with our data updating. After talking to other municipalities it seems to be a huge flaw in the software for customers not to have the ability to choose an automatic nightly update. ESRI appears to have fixed the issue where you download a dataset it will create a new cache of data, however it could go days, weeks, months, maybe years before someone downloads that dataset again. There really needs to be a way customers can set their sites to initiate a nightly update of their dataset caches ....

Re: Auto Update of Datasets on OpenData

KevinDunlop — Thu, 02 Mar 2017 18:56:22 GMT

We have the same issue. We tried creating a script to download (thus refreshing the cache) each night but weren't able to get it to work. I would love to see an the refresh option expanded to manual, nightly, weekly, or on demand.

Re: Auto Update of Datasets on OpenData

CraigSwadner — Thu, 02 Mar 2017 19:00:01 GMT

We also worked with someone from esri on a similar script and that worked until about two weeks ago. Now nothing we try works, i use to be able to manually unshare / share data and it would update. That also doesnt work anymore, i currently have an open issue with esri about this. This isnt a enterprise solution, a patch at best.

Re: Auto Update of Datasets on OpenData

CraigSwadner — Thu, 02 Mar 2017 19:06:41 GMT

or how about making the automatic button on the admin page actually automatically update your data, instead of someones interpretation.

Re: Auto Update of Datasets on OpenData

CraigSwadner — Tue, 22 Jun 2021 08:38:04 GMT

hey kevin i created a new idea here https://community.esri.com/ideas/13187 if you wouldnt mind voting that would be greatly appreciated

Re: Auto Update of Datasets on OpenData

PhilLarkin1 — Sun, 12 Dec 2021 00:08:27 GMT

Hello-

I have had success with this problem when modifying ESRI's open data script on github. The refresh function didn't update the date-updated shown on our Open Data site so I changed it a bit. I found that if I updated a property of a dataset (disclaimer, description, ... ) the 'Updated' date would update. Because the cache settings are set to Automatic users are getting fresh data. This script is run nightly from Task Scheduler.

Cache Settings:

Modified refresh function.


import requests
from arcgis.gis import GIS

    def refresh(self):
        """
        Refreshes all Open Data datasets and download cache. Will also update the index.
        """

        gis = GIS("https://[yourAGOL].maps.arcgis.com",[yourUserString], [yourPasswordString])

        DataStatus = {}

        for dataset in self.OpenDataItems:

            #send refresh request for each dataset in Open Data.
            url = "https://opendata.arcgis.com/api/datasets/{0}/refresh.json?token={1}".format(dataset, self.token)
            HTTPresponse = requests.put(url, verify=False)

            #if refresh was successful, trigger Last Modified date by editing the description attribute. This will add
            #a space in the text of the description.
            if HTTPresponse.ok:
                #strip Open Data numbering suffix from item id ('_1','_2')
                itemId = dataset[:-2]
                item = gis.content.get(itemId)
                itemName = item['title']
                #add blank description
                item.update({'description':" "})
                logger.info("HTTP Response of: {0} for layer: {1}. Layer Updated".format(HTTPresponse.status_code , itemName))
                DataStatus[itemName] = [HTTPresponse.status_code,itemId]

            else:
                logger.error("HTTP Response of: {0} for layer: {1}".format(HTTPresponse.status_code , dataset))
                DataStatus[itemName] = [HTTPresponse.status_code,itemId]

        #will refresh the download cache
        synch_url = "https://opendata.arcgis.com/api/sites/{0}/groups/synchronize.json?token={1}".format(self.OpenDataSite, self.token)
        requests.put(synch_url, verify=False)

        return DataStatus‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Environments:
Python 3.5.2
arcgis python api v 0.1 (from ArcGIS Pro's Python Package Manager)
Server 10.3.1
AGOL Open Data v1.9

[UPDATE]
I reference the ArcGIS API for Python from pypi under package name 'arcgis' but I'm not sure if it is updated anymore. You could/should reference from ArcGIS Pro's package manager. ArcGIS API for Python | ArcGIS for Developers

Re: Auto Update of Datasets on OpenData

CraigSwadner — Wed, 24 May 2017 17:56:22 GMT

i have the same script and have shared it with others, however sometimes it works and other times it does not. With no changes on our side, it just leads me to one conclusion.....

Re: Auto Update of Datasets on OpenData

PhilLarkin1 — Wed, 24 May 2017 21:02:51 GMT

I'm intrigued. What is your conclusion?

Re: Auto Update of Datasets on OpenData

DanielFenton1 — Wed, 19 Jul 2017 17:34:25 GMT

Hi,

I just wanted to share that this is on our roadmap. We have begun work to implement the next version of our indexers. They will refresh the search index and download cache daily. We currently expect this to hit production during the 4th quarter this year. We'll make sure to update you when this feature launches.

Daniel Fenton

Software Engineer | ArcGIS Hub

Re: Auto Update of Datasets on OpenData

LauriSohl — Tue, 29 Aug 2017 19:24:31 GMT

Hi Daniel, are you able to share any more information about this? I'd be interested in learning more about the timeline of implementation, what "time" the index will run, will the date on each record in open data be updated, etc.? Currently, for our critical datasets that are on open data, I have to manually update the index, and often more than once, in order for all of the records to be updated - the count/date on the data layer, the number of records in the table, and the correct number of records downloaded. I'd appreciate any tips on how to ensure, currently, that my data in open data is reflecting the current data that it is pointing to. Thank you!!

Lauri Sohl

GIS Manager

City of Sioux Falls

Re: Auto Update of Datasets on OpenData

DanielFenton1 — Wed, 06 Sep 2017 19:42:36 GMT

Hi Lauri, we're targeting the 4th quarter of this year for rolling out the new system. More details will be coming as we get closer to rollout.

The best thing you can do to keep your data up to date is to edit the ArcGIS Online item when something changes, for example adding a publish date to metadata. That will tell our systems to index the item and services as well as giving us the exact publish date.

Re: Auto Update of Datasets on OpenData

deleted-user-7SLqM7dlvYNN — Thu, 01 Mar 2018 10:09:27 GMT

Hi Daniel, was this change implemented? We're seeing erratic behaviour for annonymous and authenticated users when it comes to download capability and item details. This is despite updating the index, after updating dataset details which refreshed the update date.

Changelog for hub http://hub.arcgis.com/pages/changelog mentions on 27th Feb there was, "A regression on item-based feature service exports has been addressed". Do you have any further information on this change?

Re: Auto Update of Datasets on OpenData

LeoLadefian5 — Thu, 01 Mar 2018 22:30:28 GMT

any pdate on this? Our users are complaining about old data.

Re: Auto Update of Datasets on OpenData

LukeBehling — Mon, 23 Apr 2018 15:30:03 GMT

This was brought to our attention today as well, and even with re-indexing weekly. ESRI please update the status of this.

Re: Auto Update of Datasets on OpenData

DanielFenton1 — Mon, 23 Apr 2018 15:33:33 GMT

Luke et al,

We are still actively working on this as a top priority. We don't have an exact timeline to share yet.

Daniel Fenton

Software Engineer | ArcGIS Hub

Re: Auto Update of Datasets on OpenData

KyleWikstrom — Fri, 17 Aug 2018 16:43:11 GMT

Our local government clients are still experiencing this issue with their users (i.e., the user will download a dataset and it's not the most recent, updated dataset). The idea of firing up the FTP sites again has been thrown around at these organizations unless this gets resolved soon.

Re: Auto Update of Datasets on OpenData

BrandonLongenberger — Mon, 11 Mar 2019 14:50:07 GMT

Any update on when this issue will be resolved?

Re: Auto Update of Datasets on OpenData

Anonymous User — Mon, 11 Mar 2019 15:52:47 GMT

Hi Craig,

Just wanted to let you know that with our release for search last week we've made a significant update to the indexing system. Here's a bit more info about how the new download system works:

When a download request is made, one of two things happens:

For hosted services, we check for changes before refreshing the cache. If a dataset has been changed or updated since the cache was made, we refresh the cache. Otherwise the download comes from the previous cache
For services hosted elsewhere (e.g. on-premises servers) we refresh the cache if it has been more than 24 hours since the last refresh. If it is within the 24 hour window then the user gets the data as it was last cached

Hopefully the new download system helps resolve the issue, but be sure to let us know if you observe any issues with download freshness going forward.

Best,

Patrick

Re: Auto Update of Datasets on OpenData

Anonymous User — Mon, 11 Mar 2019 15:54:08 GMT

Hi Brandon,

We just released updates to the indexer last week. See reply to OP for more information.

Best,

Patrick

Re: Auto Update of Datasets on OpenData

ZahirIbrahim — Thu, 11 Apr 2019 14:08:38 GMT

Hi Patrick,

This is definately not the case. We have services being used in our Open Data site, where we know the data, when downloaded, is not up to date, or in some cases, does not even download properly (in spreadsheet format for instance).

We also use a script to force update all items used in our Open Data site every night, to ensure that the cache is updated. Even then, we have erratic and inconsistant behaviour. Its only when we manually go in the back end, and update the cache, that in some instances, this may resolve the issue. We seem to find this issue with 2 of our large datasets, ie one with over 12,000 rows, and another with 120,000 rows. We dont consider this big data, therefore are unsure as to why this is occuring.

Like the above post, we are thinking of creating offline versions of our data to ensure our users get current data, which is not ideal.

If you could provide best practice methods for these large datasets, that would be great.