Automate AGOL publishing

[Roli]

We are using ArcGIS Desktop 10.7.1 and trying  to automate AGOL publishing ( overwriting a service)..

We have a service ( created by publishing an mxd) that needs to be updated on a daily basis. We have automated the local data creation/update process using models.

Once the data is updated, we need to  push the updates to AGOL . One option is to re-publish the MXD and overwrite the service. However ,  there must be a better/easier way to do this!

Found a python script online and it seems to be working but not in a consistent way: sometimes we need to have the mxd  open and be logged in to AGOL... sometimes not...  This doesn't make sense because the script contains the credentials.

Not sure if this is an AGOL limitation/issue or an issue with the  python script ( see attached).

Any help is appreciated.

Thanks in advance!

[jcarlson]

I would advise against overwriting the service. As long as there's no schema change, it's not necessary, and is more likely to result in some error. We have several processes that update hosted layers on a daily basis, and we're able to use the ArcGIS Python API to do so.

A simplified version of it would look like this:

from arcgis import GIS
import pandas as pd

# Log in
gis = GIS('your-portal-url', 'username', 'password')

# Get hosted layer; replace "0" with appropriate index as needed; use ".tables[0]" to get standalone tables from a service
lyr = gis.content.get('itemid of service').layers[0]

# Query source data
df = pd.read_sql('some sql query')

# Truncate existing records
lyr.manager.truncate()

# Apply new features to layer
lyr.edit_features(adds=df)

There's a lot more to it than that, but the specifics will depend a lot upon where the data's coming from.

[MichaelVolz]

Josh:

Are you using the ArcGIS Python API to perform a truncate and append process on the hosted feature layer?

Did you find this source code in an ESRI sample?

[jcarlson]

Yes, I am, and no I didn't. The actual code is quite a bit more involved, and I wrote it up on my own.

[Roli]

Josh,

Thanks for the reply!

In our case, we need to drop all records and append new ones. I guess we could update the layers instead of re-publishing...  🙂

The only issue with our script is that it seems to be dependent on the MXD being open and the line that errors out (when it is not working) is the last one arcpy.UploadServiceDefinition_server()

error 000732: Dataset My Hosted Services does not exist or is not supported.

WARNING 001404: You are not signed in to ArcGIS Online.

Failed to execute (UploadServiceDefinition).

Thanks again!

[jcarlson]

Well, that's one of the benefits of the ArcGIS API. You don't need to rely on having desktop software open, making it possible to run your script on any machine with Python on it, so long as it has access to the source data.

Where's the data coming from? An SDE, or file-based resource?

[Roli]

Normally, the data we publish comes from SDE. However, we are trying to automate the AGOL update and let another department handle it.  

Our first AGOL publishing was done by uploading a shp/gdb (zip).  However, because all our data is in SDE, it made not sense to continue that way and now we have MXD-s we use for publishing.

After some reading on this Python automation , I realized we are approaching  this the wrong way: updating/overwriting the feat. layer by uploading a zip is a better option. The fact that the data used for this automation is not coming from SDE ( it is created by a model and exported to SHP/GDB)  goes along with the zip-based update method ...

While we'll have to read the documentation, having a sample script would help.... Found the ESRI samples, but as always , they use the most simplistic example ( csv) and it is not as helpful as they thought it would be. 🙂

Thanks!

[Roli]

Just tested the concept above ( zip-based publishing & overwrite) and it is working, with one issue: the Py script requires ARCGIS module which is only available on computers with ArcGIS Pro installed.

When attempting to run script on PC with Desktop 10.7 ( Python 2.7 ) I get error: "Import error: No module named arcgis.gis"

I'd hope that installing Python 3.7 ( current version bundled with ArcGIS Pro)  we should be able to add the ARCGIS module. Do you have a computer with Python 3.5 ( or 3.7)  without ArcGIS Pro and are able to execute python scripts updating AGOL?

Thanks!

[jcarlson]

I've run our scripts on Linux CLI environments where the ArcGIS Python API is the only Esri thing on it. You just need to have the arcgis module installed in your Python env.

I highly suggest using a package manager like Anaconda Navigator (or miniconda, if you're comfortable working in the command line). Then you can have a Python env specific to your automated scripts, with all the necessary modules installed. You'll need to add the esri channel to conda so that it can actually see the arcgis module.