Pubilishing to ArcGIS Online via Arcpy

[ThomasIllingworth]

Hello Everyone

Does anyone if it is possible to use ArcPy to publish feature layers up to ArcGIS Online from say an enterprise geodatabase as a source including overwriting the layer?

We are exploring ways to do open data without using our enterprise web services if possible. Including python, distributed collaboration from enterprise to name a few things

If not no worries just curious 🙂

Thanks

Tom

[DavidAnderson_1701]

That is what the Python API was built to do, work with online content.  The GIS module is the one to read up on.

[jcarlson]

To add onto that, if you're using the Python API together with ArcPy, you can load a geodatabase layer into a spatial dataframe, and then publish that directly to a service.

Alternatively, you can use ArcPy to prepare a service definition, then use the Python API to publish it as a new item.

[John_Spence]

Here you go...See the attachment for the code as a text file.

#-------------------------------------------------------------------------------
# Name: Portal Update Service (updateService.py)
# Purpose:
#
# Author: John Spence, Spatial Data Administrator, City of Bellevue
#
#
# Created:
# Modified:
# Modification Purpose:
#
#-------------------------------------------------------------------------------

# 888888888888888888888888888888888888888888888888888888888888888888888888888888
# ------------------------------- Configuration --------------------------------
# To be completed.
#
# ------------------------------- Dependencies ---------------------------------
# 1) Using PIP, install PyODBC if you have not previously.
# 2) This script assumes you are using MS SQL as your RDBMS.
#
# 888888888888888888888888888888888888888888888888888888888888888888888888888888

# Portal Signin Config
portalURL = r'https://www.arcgis.com'
portalUSR = r''
portalPAS = r''

# Data Store Location
data_store_path_Proj = r'\\filestore\ArcGISPro\CrimeAnalysisMaps\TransparencyDashboard.aprx'
outputStorePath = r'D:\Temp\CrimeAnalysisMaps\temp'

# Configure default sharing otpions to the service
# Reference https://pro.arcgis.com/en/pro-app/2.8/tool-reference/server/upload-service-definition.htm

shareConfig = [('ArrestData', r'Open Data (PD)', 'SHARE_ONLINE', 'PRIVATE', 'NO_SHARE_ORGANIZATION', [], ['whoever@yourplace.gov'])]

# Standard Terms of User to be used for all published items.
standardTOU = '''<div style="font-family:&quot;Avenir Next W01&quot;, &quot;Avenir Next W00&quot;, &quot;Avenir Next&quot;, Avenir, &quot;Helvetica Neue&quot;, sans-serif; font-size:16px;"></div>'''

# Error Notification
errorNotify = ['whoever@yourplace.gov']

# Send confirmation of rebuild to
adminNotify = 'whoever@yourplace.gov'

# Configure the e-mail server and other info here.
mail_server = 'smtprelay.yourplace.gov'
mail_from = 'Data Service Sync<noreply@yourplace.gov>'

# ------------------------------------------------------------------------------
# DO NOT UPDATE BELOW THIS LINE OR RISK DOOM AND DISPAIR! Have a nice day!
# ------------------------------------------------------------------------------

# Import Python Libraries
import arcpy
import os
import sys
import datetime
import time
import string
import re
import json
import collections
import urllib
import requests
import smtplib
import base64
import concurrent.futures

#-------------------------------------------------------------------------------
#
#
# Function
#
#
#-------------------------------------------------------------------------------

def main():
#-------------------------------------------------------------------------------
# Name: Function - main
# Purpose: Starts the whole thing.
#-------------------------------------------------------------------------------

starttime = datetime.datetime.now()
signinPortal(starttime)
getProject(data_store_path_Proj, starttime)

return()

def signinPortal(starttime):
#-------------------------------------------------------------------------------
# Name: Function - signinPortal
# Purpose: Signs into Portal
#-------------------------------------------------------------------------------

portalInfo = arcpy.SignInToPortal(portalURL, portalUSR, base64.b64decode(portalPAS))
portalDesc = arcpy.GetPortalDescription()
portalID = portalDesc['id']

print ('\nStartup : {}\n'.format(starttime))
print ('******** Portal Check Completed ******** ')
if portalID == '0123456789ABCDEF':
print (' - Portal connection: Internal Portal')
else:
print (' - Portal connection: ArcGIS Online')
print ('\n\n')

return()

def getProject(data_store_path_Proj, starttime):
#-------------------------------------------------------------------------------
# Name: Function - getProject
# Purpose: Prepares Proj Project for publishing.
#-------------------------------------------------------------------------------

try:
arpx = arcpy.mp.ArcGISProject(data_store_path_Proj)
print ('Found Project: {}\n'.format(data_store_path_Proj))
for maps in arpx.listMaps():
serviceName = maps.name
serviceTitle = maps.metadata.title
serviceSummary = maps.metadata.summary
serviceDescription = maps.metadata.description
serviceCredits = maps.metadata.credits
serviceTags = maps.metadata.tags
serviceConstraints = maps.metadata.accessConstraints

if serviceTitle == '':
serviceTitle = serviceName
if serviceSummary == '':
serviceSummary = 'Pending update.'
if serviceDescription == '':
serviceDescription = 'Pending update.'
if serviceCredits == '':
serviceCredits = 'City of Bellevue'
if serviceTags == '':
serviceTags = 'TBD'
if serviceConstraints == '':
serviceConstraints = standardTOU

print (' - Map Service Name: {}'.format(serviceName))
print (' - Map Title: {}'.format(serviceTitle))
print (' - Map Summary: {}'.format(serviceSummary))
print (' - Map Description: {}'.format(serviceDescription))
print (' - Map Credits: {}'.format(serviceCredits))
print (' - Map Tags: {}'.format(serviceTags))
print (' - Map Constraints: {}\n'.format(serviceConstraints))

pendingMap = arpx.listMaps(serviceName)[0]

print (' Sending to Publishing...')

try:
publishWebService(serviceName, outputStorePath, serviceTitle, serviceSummary,
serviceDescription, serviceCredits, serviceTags, serviceConstraints, pendingMap, starttime)
except Exception as publishWebServiceError:
print ('Failure! -- {}'.format(publishWebServiceError.args[0]))
sendErrorTitle = 'Portal Service Update Failure!'
sendErrorInfo = 'There was an error publishing the map for this project. \nDetails: {}'.format(publishWebServiceError.args[0])

payLoadMessage = ('''The {} service failed to update as expected. Please check the serivce and repair as soon as reasonable.\n\n{}\n\nStarted @ {}\nCompleted @ {}\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceName, sendErrorInfo, starttime, finishtime))
payLoadSubject = 'Failure! -- GIS Data Service Sync'
payloadPriority = '2'
emailContact = '{}'.format(adminNotify)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)

for errorSend in errorNotify:
payLoadMessage = ('''The {} service failed to update as expected. NSS has been notified and will reach out if your support is required.\n\n{}\n\nStarted @ {}\nCompleted @ {}\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceName, sendErrorInfo, starttime, finishtime))
payLoadSubject = 'Failure! -- {} GIS Data Service Sync'.format(serviceName)
payloadPriority = '2'
emailContact = '{}'.format(errorSend)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)

except Exception as arpxReviewError:
finishtime = datetime.datetime.now()
print ('Failure! -- {}'.format(arpxReviewError.args[0]))
sendErrorTitle = 'Portal Service Update Failure!'
sendErrorInfo = 'There was an error in obtaining the map project for publishing. \nDetails: {}'.format(arpxReviewError.args[0])

payLoadMessage = ('''The {} service failed to update as expected. Please check the serivce and repair as soon as reasonable.\n\n{}\n\nStarted @ {}\nCompleted @ {}\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceName, sendErrorInfo, starttime, finishtime))
payLoadSubject = 'Failure! -- GIS Data Service Sync'
payloadPriority = '2'
emailContact = '{}'.format(adminNotify)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)

for errorSend in errorNotify:
payLoadMessage = ('''The {} service failed to update as expected. NSS has been notified and will reach out if your support is required.\n\n{}\n\nStarted @ {}\nCompleted @ {}\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceName, sendErrorInfo, starttime, finishtime))
payLoadSubject = 'Failure! -- {} GIS Data Service Sync'.format(serviceName)
payloadPriority = '2'
emailContact = '{}'.format(errorSend)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)

return()

def publishWebService(serviceName, outputStorePath, serviceTitle, serviceSummary, serviceDescription,
serviceCredits, serviceTags, serviceConstraints, pendingMap, starttime):
#-------------------------------------------------------------------------------
# Name: Function - publishWebService
# Purpose: Publishes the web service.
#-------------------------------------------------------------------------------

clearToPublish = 0
for shareSet in shareConfig:
if shareSet[0] == serviceName:
setPortalFolder = shareSet[1]
setMyContents = shareSet[2]
setPublic = shareSet[3]
setOrganization = shareSet[4]
setGroups = shareSet[5]
mailCustomer = shareSet[6]
clearToPublish = 1

if clearToPublish == 1:
arcpy.env.overwriteOutput = True
# Create Service Draft and set the properties.
sdDraft = pendingMap.getWebLayerSharingDraft("HOSTING_SERVER", "FEATURE", serviceName)
sdDraft.title = '{}'.format(serviceTitle)
sdDraft.summary = '{}'.format(serviceSummary)
sdDraft.description = '{}'.format(serviceDescription)
sdDraft.credits = '{}'.format(serviceCredits)
sdDraft.tags = '{}'.format(serviceTags)
sdDraft.useLimitations = '{}'.format(serviceConstraints)
sdDraft.portalFolder = '{}'.format(setPortalFolder)
sdDraft.overwriteExistingService = 'TRUE'
sdDraftFile = '{}.sddraft'.format(serviceName)

sdDraftFileOuput = os.path.join(outputStorePath, sdDraftFile)

# Create Service Definition Draft file
sdDraft.exportToSDDraft(sdDraftFileOuput)

# Stage Service
sdFileName = '{}.sd'.format(serviceName)
sdOutput = os.path.join(outputStorePath, sdFileName)
arcpy.StageService_server(sdDraftFileOuput, sdOutput)

# Prepping SD variables
inSdFile = sdOutput
inServer = 'HOSTING_SERVER'
inServiceName = serviceName
inCluster = ''
inFolderType = ''
inFolder = ''
inStartup = ''
inOverride = 'OVERRIDE_DEFINITION'
inMyContents = '{}'.format(setMyContents)
inPublic = '{}'.format(setPublic)
inOrganization = '{}'.format(setOrganization)
inGroups = setGroups

# Share to portal
print(' ...Uploading Service Definition')
arcpy.UploadServiceDefinition_server(inSdFile, inServer, inServiceName, inCluster, inFolderType, inFolder,
inStartup, inOverride, inMyContents, inPublic, inOrganization, inGroups)

finishtime = datetime.datetime.now()

if serviceTitle == 'Map' or serviceTitle == 'Map1':
serviceTitle = serviceName

# Prepare Message
for customerNotify in mailCustomer:
payLoadMessage = ('''The {} service has been successfully updated. If there was a schema modification, please check any related views to ensure they are still operating as expected\n\nStarted @ {}\nCompleted @ {}\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceTitle, starttime, finishtime))
payLoadSubject = 'Success! -- GIS Data Service Sync of {}'.format(serviceTitle)
payloadPriority = '4'
emailContact = '{}'.format(customerNotify)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)

payLoadMessage = ('''The {} service has been successfully updated. If there was a schema modification, please check any related views to ensure they are still operating as expected.\n\nStarted @ {}\nCompleted @ {}\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceTitle, starttime, finishtime))
payLoadSubject = 'Success! -- GIS Data Service Sync of {}'.format(serviceTitle)
payloadPriority = '4'
emailContact = '{}'.format(adminNotify)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)
clearToPublish = 0

else:
print (' This map is not configured to be published! Sending notification and moving on to the next map.')

payLoadMessage = ('''{} is not configured to publish. Please contact coordinate with the owner and run automation again.\n\n[SYSTEM AUTO GENERATED MESSAGE]'''.format(serviceName))
payLoadSubject = 'WARNING! -- Unexpected title attempted to publish {}'.format(serviceName)
payloadPriority = '3'
emailContact = '{}'.format(adminNotify)
sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact)

print ('\n\n')

return()

def sendNotification (payLoadMessage, payLoadSubject, payloadPriority, emailContact):
#-------------------------------------------------------------------------------
# Name: Function - main
# Purpose: Starts the whole thing.
#-------------------------------------------------------------------------------

print (' ----- Sending notification to {}'.format(emailContact))

server = smtplib.SMTP(mail_server)
email_target = emailContact
mail_priority = '{}'.format(payloadPriority)
mail_subject = '{}'.format(payLoadSubject)
mail_msg = '{}'.format(payLoadMessage)

send_mail = 'To: {0}\nFrom: {1}\nX-Priority: {2}\nSubject: {3}\n\n{4}'.format(email_target, mail_from, mail_priority, mail_subject, mail_msg)
server.sendmail(mail_from, email_target, send_mail)

server.quit()

print (' !-- Message Sent!')

return

#-------------------------------------------------------------------------------
#
#
# MAIN SCRIPT
#
#
#-------------------------------------------------------------------------------

if __name__ == "__main__":
main()

[John_Spence]

Here you go by way of code.

[John_Spence]

I should probably add a few notes...

The way I handle this kind of publishing is, and this is because it is hosted, I publish a private copy of the data (as shown by way of example in the code) and then create a view of the published service that is consumed by the end user/product. In a way, this provides a abstraction layer to how you are configured allowing a greater degree of control and/or reuse of the same data for multiple products if you need to slice and dice the data different. 1 "service" to rule them all. Thankfully, when you overwrite the service during publishing the view remains connected and configured. The only caveat to that would be if someone changes the schema on the core service. Then you need to recreate the views. 1 downside for a lot of upside.

[Brian_Wilson]

Thanks for sharing your code.