overwriting GeoJSON-based hosted feature service?

[DavidAskov1]

Hi everyone - I have a GeoJSON file that I loaded to ArcGIS Online as a hosted feature layer. The GeoJSON file updates periodically, and I want to automate overwriting the whole AGOL hosted feature layer layer with the latest data. I am using Python to automate this, and I honestly don't care if I use the ArcGIS API for Python or just interact with the REST services directly. I'd like to avoid ArcPy so I can install it on a linux server with no other ArcGIS software. 

Manually, it's really easy to do this (see screenshot). I initially loaded a GeoJSON file to AGOL. I find its hosted feature layer, click on "Update Data", select the overwrite option, select the new file, and it's all taken care of. This is the process I'd like to automate. 

In the ArcGIS API for Python, I see the FeatureLayer object has an append method, which looks like what I want. That method has an "upload_format" input, and "geojson" is one of the options. But where and how do I pass in the GeoJSON file itself? I couldn't find any code samples of this.

Note1: I suppose I could read the GeoJSON and construct a "regular" feature request, but I'd like to find a way to just upload the whole GeoJSON file, like I can do with the AGOL web GUI. 

Note2: I probably need to delete old features before I append, but I know how to do that, if needed.

As I said, I am ok with just interacting with the REST API via Python, if that's any easier. I tried watching Chrome Developer Tools while I did the manual steps above, but I couldn't quite figure out how to replicate that. 

thanks!!!

[RobbieBagby]

Installing the API using ArcGIS Pro Python Package Manager is the easiest way I found to get the environment up and running: Install and set up | ArcGIS for Developers. This is assuming you have already downloaded ArcGIS Pro.

Once you have the package installed navigate to C:\Program Files\ArcGIS Pro\bin\Python\envs\arcgispro-py3\Scripts and place this file anywhere in that folder.

Edit 5/20/2019: The link to the jupyter notebook no longer works. See attached python file. This file can be configured to work with your data. If you want to work in jupyter notebooks, then you will need to copy each input line into a new notebook.

This is a jupyter notebook that I have created for overwriting a feature layer in AGOL from a geojson file. While in this folder find an application called jupyter-notebook and double click to run. At this point you will see something like this:

Click the notebook that you just added to open and change the areas I have bracketed to work with your data. Once you have the notebook working, you can download as a python file and set up a task in Windows Task Scheduler to automate how often the script will run. 

I am working in the C:\test folder but you can change this to wherever your data is stored. You can also ignore the part of the script that is getting the geojson from a url and writing to a file if you already have the data stored somewhere. 

[EricShreve]

Robbie, I followed the example code that you shared to update a feature service that captures remote automated weather station data from MesoWest, which is in GeoJSON format. Everything is running good in the code until I get to the last line "#if there are features then overwrite the layer with the new data. If no features then truncate the database."  Once I perform the function I am prompted with an "Error while analyzing Shapefile 'RAWS.geojson Invalid Shapefile " 

Any clue as to what I am missing?

Thanks,

Eric Shreve

[EricShreve]

Nevermind I was able to successfully do it. I initially uploaded the file type as a zipped shapefile when it should have been a GeoJSON file type. Thanks Robbie.

[JanBurdziej]

Hi Robbie,

Thanks for sharing. We have a similar scenario: we need to regularly update our AGOL Feature Services with new GeoJSON files extracted from a fixed URL ("http" not "https").

Could you share again your Jupyter notebook file - I think it is not available anymore?

Thanks and best regards,

Jan

PS. Ideally I'd like the Jupyter notebook to be hosted on some remote server, not on my own PC though. Any ideas what would be the best approach?

[RobbieBagby]

I attached the python file to my original post. You can host the environment on any windows machine that has task scheduler. We have a couple machines dedicated for ArcGIS Server and I just have task scheduler running the python script every so often on one of them. You have to install ArcGIS Pro as well. 

[JanBurdziej]

Thanks! Works great on my desktop.

PS. Did you try running Jupyter notebooks on Azure Databricks? Just wondering if I can somehow install ArcGIS Pro dependencies in this environment...

[DavidAskov1 · Accepted solution]

Robbie Bagby‌ & Eric Shreve‌ put me on the right trail above.

The key is this method:

https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.features.managers.html#arcgis.features.m...

Connect to your server, get the right layer (named 'layer' in the code snippet below), and then do the following:

collection = arcgis.features.FeatureLayerCollection.fromitem(layer)
overwrite_result = collection.manager.overwrite(file_path)‍‍‍

In my case, I was creating a hosted feature layer from a GeoJSON, but it will work with a shapefile, FGDB, etc. It will create duplicate items with the same name, but one is type "Feature Layer (hosted)" in the web interface (which is callled "Feature Service" in the API) and the other is JSON, FGDB, shapefile, etc. You want to find and use the former layer in the example above.

Also note that when I was doing this last year, it would always overwrite the symbology I set on it every time I updated the data. This seemed like a bug, so maybe it's fixed now, but FYI.

[MelanieWilliams1]

David, 

Your answer was very helpful.  I was wondering if you had found a work around for the overwriting of the symbology issue?  This process will be really beneficial for us if we can figure out how to keep from having to reset the symbology and popup windows every week.  

Thanks for any ideas you or anyone else can share!

Melanie