How to publish a feature service to ArcGIS Online from an mxd USING PYTHON?

The following code will publish a hosted feature service. Change the following:

- Line 5, the workspace
- Line 11, the name you want to give the service
- Line 58, the username and password to your AGO account

import arcpy import xml.dom.minidom as DOM  import os  workspace = 'C:/Temp/hosted publishing/'  # Reference map document for CreateSDDraft function. mapDoc = arcpy.mapping.MapDocument(r'C:\Temp\betty4.mxd') # Create service and sddraft variables for CreateSDDraft function. sddraft = workspace + 'HostedMS.sddraft' service = 'FeatureService' arcpy.mapping.CreateMapSDDraft(mapDoc, sddraft, service, 'MY_HOSTED_SERVICES', summary="test", tags='test') con = 'My Hosted Services' sd = workspace + service + '.sd'  doc = DOM.parse(sddraft)  # Read the sddraft xml. doc = DOM.parse(sddraft) # Change service from map service to feature service typeNames = doc.getElementsByTagName('TypeName') for typeName in typeNames:     # Get the TypeName we want to disable.     if typeName.firstChild.data == "MapServer":      typeName.firstChild.data = "FeatureServer"  #Turn off caching configProps = doc.getElementsByTagName('ConfigurationProperties')[0] propArray = configProps.firstChild propSets = propArray.childNodes for propSet in propSets:     keyValues = propSet.childNodes     for keyValue in keyValues:         if keyValue.tagName == 'Key':             if keyValue.firstChild.data == "isCached":                 # turn on caching                 keyValue.nextSibling.firstChild.data = "false"  #Turn on feature access capabilities configProps = doc.getElementsByTagName('Info')[0] propArray = configProps.firstChild propSets = propArray.childNodes for propSet in propSets:     keyValues = propSet.childNodes     for keyValue in keyValues:         if keyValue.tagName == 'Key':             if keyValue.firstChild.data == "WebCapabilities":                 # turn on caching                 keyValue.nextSibling.firstChild.data = "Query,Create,Update,Delete,Uploads,Editing"  outXml = workspace + 'HostedMSNew.sddraft'  f = open(outXml, 'w')      doc.writexml( f )      f.close() analysis = arcpy.mapping.AnalyzeForSD(outXml) #print analysis #arcpy.SignOutFromPortal_server() arcpy.SignInToPortal_server("username","password","http://www.arcgis.com/") arcpy.StageService_server(outXml, sd) arcpy.UploadServiceDefinition_server(sd, con)

All comments

import arcpy import xml.dom.minidom as DOM  import os  workspace = 'C:/Temp/hosted publishing/'  # Reference map document for CreateSDDraft function. mapDoc = arcpy.mapping.MapDocument(r'C:\Temp\betty4.mxd') # Create service and sddraft variables for CreateSDDraft function. sddraft = workspace + 'HostedMS.sddraft' service = 'FeatureService' arcpy.mapping.CreateMapSDDraft(mapDoc, sddraft, service, 'MY_HOSTED_SERVICES', summary="test", tags='test') con = 'My Hosted Services' sd = workspace + service + '.sd'  doc = DOM.parse(sddraft)  # Read the sddraft xml. doc = DOM.parse(sddraft) # Change service from map service to feature service typeNames = doc.getElementsByTagName('TypeName') for typeName in typeNames:     # Get the TypeName we want to disable.     if typeName.firstChild.data == "MapServer":      typeName.firstChild.data = "FeatureServer"  #Turn off caching configProps = doc.getElementsByTagName('ConfigurationProperties')[0] propArray = configProps.firstChild propSets = propArray.childNodes for propSet in propSets:     keyValues = propSet.childNodes     for keyValue in keyValues:         if keyValue.tagName == 'Key':             if keyValue.firstChild.data == "isCached":                 # turn on caching                 keyValue.nextSibling.firstChild.data = "false"  #Turn on feature access capabilities configProps = doc.getElementsByTagName('Info')[0] propArray = configProps.firstChild propSets = propArray.childNodes for propSet in propSets:     keyValues = propSet.childNodes     for keyValue in keyValues:         if keyValue.tagName == 'Key':             if keyValue.firstChild.data == "WebCapabilities":                 # turn on caching                 keyValue.nextSibling.firstChild.data = "Query,Create,Update,Delete,Uploads,Editing"  outXml = workspace + 'HostedMSNew.sddraft'  f = open(outXml, 'w')      doc.writexml( f )      f.close() analysis = arcpy.mapping.AnalyzeForSD(outXml) #print analysis #arcpy.SignOutFromPortal_server() arcpy.SignInToPortal_server("username","password","http://www.arcgis.com/") arcpy.StageService_server(outXml, sd) arcpy.UploadServiceDefinition_server(sd, con)

MaryEllen_Perko

This worked great for me! You saved me a bunch of time. Thanks.

Now, I'm just wondering if anyone knows how to set the sharing during publishing using python. I can see references to it in the documentation for UploadServiceDefinition_server, but I cannot figure out the syntax when it is the only optional setting you need. And, does it work in AGOL?

Thanks,
Mary Ellen

BenoitMetzger

Hi Jeff,

thanks for your help, I will give that a go... but from what i read it seems to be working 🙂

Mary Helen, not too sure if thsi is what you want... here is a line i wrote which worked a little while ago (it was sharing the service on AGOL from memory):
arcpy.UploadServiceDefinition_server("C:\\...", "My Hosted Services", '', '', "NEW", "folder", "STARTED","OVERRIDE_DEFINITION", "SHARE_ONLINE","PRIVATE", "NO_SHARE_ORGANIZATION", "")

Cheers.

KaushalShah

Hi, I have manually created/published the ArcGIS online feature service from arcmap as a one time activity.
The data in the mxd will keep on changing and I have updated the data in the mxd.
Since the data is changing, each time I will have to update the already published feature service. Manually we can publishing the service and select overwrite existing service. The same I want to automatize through python script.

As the script for creating new feature service and also sharing the feature service was provided in the thread, can you please urgently help me by providing the python script for updating the data of the existing feature service. We do not have to change service definition of the already published feature service.

Urgent help would be appreciated.

Thanks and regards,
Kaushal Shah.

Add the following 2 pieces into Jeff's code and it'll allow you to use UploadService (after staging of course) on an existing service (because it overwrites the service).

typeReplace = 'esriServiceDefinitionType_Replacement'
statePublished = 'esriSDState_Published'
           
myTagsType = doc.getElementsByTagName('Type')
for myTagType in myTagsType:
    if myTagType.parentNode.tagName == 'SVCManifest':
        if myTagType.hasChildNodes():
            myTagType.firstChild.data = typeReplace

myTagsState = doc.getElementsByTagName('State')
for myTagState in myTagsState:
    if myTagState.parentNode.tagName == 'SVCManifest':
        if myTagState.hasChildNodes():
            myTagState.firstChild.data = statePublished

KaushalShah

Thanks a ton for the help. This helped me overwrite the existing feature service without deleting it manually. I am saved.

But here I believe that we had to add the new part of the code that you provided BEFORE the staging of the service definition and not after the line arcpy.StageService_server(outXml, sd). You mentioned otherwise. So the new code comes before the "outXml = workspace + 'HostedMSNew.sddraft' of the main code.

Can you also help me provide an insight on the map service being overwritten. If we have already created the Map using the original feature service, and then overwrite the existing feature service, will the previous maps work with the same shared url. Hope I am clear in my requirement.

Lots of Regards,
Kaushal Shah.

TiagoRibeiro

I've been following this thread, which has helped me a lot, so let me start with a big THANK YOU! I already have a script that updates my feature service but... this needs the desktop installed to run right?
For example: If you publish this as gp task on ArcGIS for Server the upload part will throw this error:

ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Server: Dataset My Hosted Services does not exist or is not supported
Failed to execute (UploadServiceDefinition).

Is there a way around this or the way is to have ArcGIS For Desktop installed, and the "ArcGISConnection.exe" process running?

Kaushal Shah:
Right, the code is to modify the XML, thus it must go before stage.
As long as the service name doesn't change, the link should remain the same. The hosted service account id is a set of letters and numbers which to the best of my knowledge doesnt change. So to the best of my knowledge, it should be ok.

tribeiro:
You've made a GP Service to perform this publishing routine?
I'd strongly recommend against that. The Upload and Stage tools (which are required in this workflow) are not designed to be re-published in your own geoprocessing service. I'd suggest making it a scheduled task on one of your desktop machines.

TiagoRibeiro

tribeiro:
You've made a GP Service to perform this publishing routine?
I'd strongly recommend against that. The Upload and Stage tools (which are required in this workflow) are not designed to be re-published in your own geoprocessing service. I'd suggest making it a scheduled task on one of your desktop machines.

Thanks for the reply Kevin, that's what I thought, and the scheduled task approach was (and now will be) my next step.
Any plans of making those tools, Upload and Stage, available through geoprocessing services in the future?

Not in 10.2, and no plans to the best of my knowledge after that.
What these tools do, and how they work is not conducive to being used in a service. I'd say at best they would act like a scheduled task that could be modified slightly at execution time.
I say this as the server will most likely have to have the MXD + Data you want to publish. It would be a pretty involved service to that gets both the MXD and all required data from the client, extracts it somehow, and then publishes. So I'm making the assumption your server already has the data/mxds on it. If thats the case, setting it up as a scheduled task would probably be more straight forward.
I'm not saying these as a service cant be done - I'm just saying its something that we haven't tested and I could imagine a few areas which will be difficult to overcome. As such, I'd take the "easier" approach of just making a scheduled task.

new profile pic (4).jpg

Just for anyone else who comes upon this thread...
Jeff and I posted a blog with the python script to:
-CreateMapSDDraft, Analyze, Stage and Upload for map documents to be pushed to ArcGIS.com as a hosted feature service /w overwrite and shared to arcgis.com turned on.

http://blogs.esri.com/esri/arcgis/2013/04/23/updating-arcgis-com-hosted-feature-services-with-python/

JenniferBorlick_0-1753372348440.png

I've attempted to use this sample and am able to successfully update feature services with 10.1, but not with 10.2. I'm encountering the following error:

ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Server: Dataset My Hosted Services does not exist or is not supported
WARNING 001404: You are not signed in to ArcGIS Online.
Failed to execute (UploadServiceDefinition).

[ATTACH=CONFIG]27306[/ATTACH]

This appears to be an issue with the arcpy SignInToPortal function. I'm able to publish/overwrite the service with 10.2 only when I comment out the SignInToPortal line and sign in separately using ArcGIS Desktop. Is anyone else having this problem with ArcGIS 10.2? Are there any workarounds?

Thanks,
Erik

publish fs error.PNG

DanikBourdeau2

I'm able to publish/overwrite the service with 10.2 only when I comment out the SignInToPortal line and sign in separately using ArcGIS Desktop. Is anyone else having this problem with ArcGIS 10.2? Are there any workarounds?

Thanks,
Erik

Since 10.2, this is how I have had to publish/overwrite services as well. Not that big of a deal but I do wish that somewhere it would have been stated that SignInToPortal doesn't work like it used to.

Danik, Erik,

With the 10.2 release, the authentication method for Portal has changed, as such the SignIntoPortal tool is no longer necessary.

We're currently investigating why the tool is causing these sorts of issues. (It should have just passed through the tool with a warning message indicating you need to Sign In from the ArcMap menu)

To get around the issue for now:

Dont use the SignIn tool in your script/workflows
Any machine you want to run this script from, must first open ArcMap, and use File > Sign In with the automatic option checked

Kevin,

Great, thanks for weighing in on the changes in 10.2 regarding the SignIn function.

Do you have a recommended workflow to update feature services using a scheduled task when a user isn't logged into the machine. Is that no longer possible?

Thanks much,
Erik

From the previous testing I've done, the following should do what you want:

Log on to the machine, open ArcMap, and sign in (make note of the account you logged into Windows with)
Make sure you've checked the automatic sign in on the Portal sign in box.
Create your scheduled task (it must run under the same windows user account from above)

The key here is to log in once, have it be remembered, then have the task run under the user account which had logged in previously.

Of course, make sure the SignInToPortal tool has been commented out.
If I get time later this afternoon I'll give this another go.

I tried the approach you suggested but was unable to successfully publish the service as a scheduled task (or manually while not logged in through Desktop). I'm very interested in any other suggestions you or other folks may have that would enable running the update FS as a scheduled task.

Thanks!
Erik

Erik,

Are you getting an error when running the python script? Or are you having a problem creating a scheduled task?

I get the error when running the python script, and the scheduled task fails (presumably from the same error but I haven't logged the errors yet).

Are you getting a different error now that you implemented the new sign in methodogy? Can you send me the exact error message? Thanks.

Correction on previous post: I attempted to run the publishing script again today from within my python IDE (pyscripter) and it executed correctly without having signed into ArcGIS Online manually through Desktop. I hadn't changed anything in my script from last week before executing it, so it's possible that logging out and back in on the system may account for it working now.

However, I confirmed that I am still unable to get it to work though a scheduled task. I logged the error to a text file, and the following was recorded:

Executing: UploadServiceDefinition C:\Workspace\KYEM_Test\KYEM_TEST3.sd "My Hosted Services" KYEM_TEST3 # FROM_SERVICE_DEFINITION # STARTED OVERRIDE_DEFINITION SHARE_ONLINE PRIVATE NO_SHARE_ORGANIZATION #
Start Time: Mon Sep 16 14:06:05 2013
Failed to execute. Parameters are not valid.
ERROR 000732: Server: Dataset My Hosted Services does not exist or is not supported
WARNING 001404: You are not signed in to ArcGIS Online.
Failed to execute (UploadServiceDefinition).
Failed at Mon Sep 16 14:06:05 2013 (Elapsed Time: 0.02 seconds)

The script (only minor changes from Esri's sample):


import arcpy, os, sys
from datetime import datetime
import xml.dom.minidom as DOM

arcpy.env.overwriteOutput = True

# Define global variables from User Input
mapDoc = 'KYEM.mxd'
serviceName = 'KYEM'
shareLevel = 'PRIVATE'              # Options: PUBLIC or PRIVATE
shareOrg = 'NO_SHARE_ORGANIZATION'  # Options: SHARE_ORGANIZATION and NO_SHARE_ORGANIZATION
shareGroups = ''                    # Options: Valid groups that user is member of

tempPath = sys.path[0]

#SignInToPortal function does not work in 10.2
#for 10.1: uncomment below line to enable sign in
#for 10.2:  (1): Sign in to arcgis desktop and check 'sign in automatically'
#           (2): Schedule task to run publishFS.py script, using same user as in step 1
##arcpy.SignInToPortal_server('','','http://www.arcgis.com/')

sdDraft = tempPath+'/{}.sddraft'.format(serviceName)
newSDdraft = 'updatedDraft.sddraft'

try:
    os.remove(serviceName+'.sd')
    print 'SD already exists, overwriting..'
    SD = os.path.join(tempPath, serviceName + '.sd')
    print 'File removed and overwritten'
except OSError:
    print 'No SD exists, writing one now.'
    SD = os.path.join(tempPath, serviceName + '.sd')

try:

    # create service definition draft
    analysis = arcpy.mapping.CreateMapSDDraft(mapDoc, sdDraft, serviceName, 'MY_HOSTED_SERVICES')


    # Read the contents of the original SDDraft into an xml parser
    doc = DOM.parse(sdDraft)

    # The follow 5 code pieces modify the SDDraft from a new MapService
    # with caching capabilities to a FeatureService with Query,Create,
    # Update,Delete,Uploads,Editing capabilities. The first two code
    # pieces handle overwriting an existing service. The last three pieces
    # change Map to Feature Service, disable caching and set appropriate
    # capabilities. You can customize the capabilities by removing items.
    # Note you cannot disable Query from a Feature Service.
    tagsType = doc.getElementsByTagName('Type')
    for tagType in tagsType:
        if tagType.parentNode.tagName == 'SVCManifest':
            if tagType.hasChildNodes():
                tagType.firstChild.data = 'esriServiceDefinitionType_Replacement'

    tagsState = doc.getElementsByTagName('State')
    for tagState in tagsState:
        if tagState.parentNode.tagName == 'SVCManifest':
            if tagState.hasChildNodes():
                tagState.firstChild.data = 'esriSDState_Published'

    # Change service type from map service to feature service
    typeNames = doc.getElementsByTagName('TypeName')
    for typeName in typeNames:
        if typeName.firstChild.data == 'MapServer':
            typeName.firstChild.data = 'FeatureServer'

    #Turn off caching
    configProps = doc.getElementsByTagName('ConfigurationProperties')[0]
    propArray = configProps.firstChild
    propSets = propArray.childNodes
    for propSet in propSets:
        keyValues = propSet.childNodes
        for keyValue in keyValues:
            if keyValue.tagName == 'Key':
                if keyValue.firstChild.data == 'isCached':
                    keyValue.nextSibling.firstChild.data = 'false'

    #Turn on feature access capabilities
    configProps = doc.getElementsByTagName('Info')[0]
    propArray = configProps.firstChild
    propSets = propArray.childNodes
    for propSet in propSets:
        keyValues = propSet.childNodes
        for keyValue in keyValues:
            if keyValue.tagName == 'Key':
                if keyValue.firstChild.data == 'WebCapabilities':
                    keyValue.nextSibling.firstChild.data = 'Query,Create,Update,Delete,Uploads,Editing'

    # Write the new draft to disk
    f = open(newSDdraft, 'w')
    doc.writexml( f )
    f.close()

     # Analyze the service
    analysis = arcpy.mapping.AnalyzeForSD(newSDdraft)

    if analysis['errors'] == {}:
        # Stage the service
        arcpy.StageService_server(newSDdraft, SD)

        # Upload the service. The OVERRIDE_DEFINITION parameter allows you to override the
        # sharing properties set in the service definition with new values.
        arcpy.UploadServiceDefinition_server(SD, 'My Hosted Services', serviceName, '', '', '', '', 'OVERRIDE_DEFINITION','SHARE_ONLINE', shareLevel, shareOrg, shareGroups)

        print 'Uploaded and overwrote service'

        # Write messages to a Text File
        txtFile = open(tempPath+'/{}-log.txt'.format(serviceName),"a")
        txtFile.write (str(datetime.now()) + " | " + "Uploaded and overwrote service" + "\n")
        txtFile.close()

    else:
        # If the sddraft analysis contained errors, display them and quit.
        print analysis['errors']

        # Write messages to a Text File
        txtFile = open(tempPath+'/{}-log.txt'.format(serviceName),"a")
        txtFile.write (str(datetime.now()) + " | " + analysis['errors'] + "\n")
        txtFile.close()

except:

    print arcpy.GetMessages()
    # Write messages to a Text File
    txtFile = open(tempPath+'/{}-log.txt'.format(serviceName),"a")
    txtFile.write (str(datetime.now()) + " | Last Chance Message:" + arcpy.GetMessages() + "\n")
    txtFile.close()

When you got it to work in an IDE, did you have ArcMap open at the same time? And when you ran it as a task, where you logged out of the machine?

With the new security enhancements introduced at 10.2, it is necessary to have ArcMap open to get the sign-in information, even with "sign me in automatically" checked on. I realize that this is not ideal as it makes nightly auto-updates to hosted features services nearly impossible. A time frame for getting this to work is still being investigated on our end.

We also have another bug where overwriting an existing hosted feature service sometimes fails - usually every second time for most services. We are investigating a fix for this issue as well. Let me know if you are hitting this issue as well - try running your script several times in a row (perhaps 3 times).

I will let you know when I have more information.

Jeff:

Yes, I have successfully run this script while ArcMap was not open, though the ArcGIS Connection globe is present in the taskbar and indicates that i'm signed into ArcGIS Online. I have not yet been able to run the task while logged out,

I've also run into the problem where the script fails when run several consecutive times, I get this error:

Executing: UploadServiceDefinition C:\Workspace\KYEM_Test\KYEM_TEST3.sd "My Hosted Services" KYEM_TEST3 # FROM_SERVICE_DEFINITION # STARTED OVERRIDE_DEFINITION SHARE_ONLINE PRIVATE NO_SHARE_ORGANIZATION #
Start Time: Thu Sep 19 08:41:03 2013
ERROR 001566: Service overwrite error: failed to delete the service.
ERROR: code:400, Item 'e18aae36f2874deba39e380611c59745' does not exist or is inaccessible., Bad syntax in request.
Failed to execute (UploadServiceDefinition).

However, when attempted again it typically works, so this seems like a relatively minor problem that can be solved with a 'try until no error' loop.

I'd definitely appreciate it if you let me know when you have more information, specifically about the sign-in issue.

Thanks,
Erik

Even though you are able to run the script from an IDE with ArcMap closed, you may eventually find that over time, the AGO connection invalidates itself. I dont know exactly how long it will be valid for. For example, if I shut down ArcMap (without my IDE open), the connection goes away after a couple minutes. However, if I have ArcMap open, then run my script from an IDE, then close ArcMap, the connection stays alive for longer until I shut down the IDE - I am trying to get an answer on how this exactly works and how long the token is valid for. A workaround to this in your python script would be to open arcmap and force it to connect, then execute the code to publish the service, then kill arcmap. Something like:

from subprocess import Popen, PIPE
import win32com.client
import time
import os
proc = Popen(["C:\\Program Files (x86)\\ArcGIS\\Desktop10.2\\bin\\ArcMap.exe", "c:\\temp\\states.mxd"], stdout=PIPE) # put in a dummy mxd to get around the start dialog
time.sleep(10)
shell = win32com.client.Dispatch("WScript.Shell")
shell.SendKeys("%f", 0) # sends ALT+F to the UI, which signs you in - just opening ArcMap doesnt sign you in. You have to click the file menu
< code to overwrite a hosted feature service goes here >
os.system("taskkill /im arcmap.exe /f")

And yes, 'Try until no exception is raised' condition should get around the issue of it failing every second time.

It's ugly, but it should work if you are logged in. I havent thouht of a workaround that'll work if you are logged off your machine.

huaweiYin

First, I want to give you guys a big Thank you to provide very rich-full information. I am publishing and updating my stuff on ARCGISONLINE perfectly with 10.1 every week.

However, I got some issues for 10.2. I sign in my account manually, and running the scripts, it gave me the error message below:

ExecuteError: ERROR 001566: Service overwrite error: failed to delete the service.
ERROR: code:400, Item '37a9c329cea54f339b56d142258e387d' does not exist or is inaccessible., Bad syntax in request.
Failed to execute (UploadServiceDefinition).

So i think in 10.2 it can not overwrite in ARCGIS ONLINE with the script in 10.1. But if I delete the files I published on AGO, then republish it, that works well, so just can not overwite.

Anyone has any idea how to solve this overwrite problem in 10.2?

Appropriate!

Howie

MikeVolz

Besides the overwrite issue, has anyone been able to get this automated process to work in v10.2?

Can you get this automated process to work on a brand new service where there is no overwrite to be done?

KerstinAgatz

Hello,

is there a possibilty to turn on "enable attachments" for a feature in a feature service using python code to publish a feature servide?

Thanks Kerstin