Make it Easier to Replicate the ArcGIS Pro Workflow for Publishing with arcpy

[MatthewGeorge]

It'd be great if there was a way to easily replicate the out of the box workflows for publishing a Layer to Portal. I've been trying to replicate the workflow that is performed using the method right-click the Layer, select Sharing > Share As Web Layer, using arcpy.

Looking at the staging folders, there are several artifacts that are created during the publishing process and the SD draft XML looks different - mostly the same but different enough - than using the arcpy.mp.Map.getWebLayerShaingDraft() function. See additional post https://community.esri.com/t5/arcgis-api-for-python-questions/publish-a-map-image-layer-with-feature... that covers one of the other differences that I've noted. There are also other items such as the Document Info that seem to be different and I'm sure there are other things I have missed.

If this isn't that straightforward then at least provide better documentation on the process and properties available in an SD draft file.

Another option would be to allow users to configure default publishing settings so that they don't have to be edited to meet an organisations standards each time.

[RonnieRichards]

When publishing services to ArcGIS Server using arcpy it does not provide the ability to set service parameters. Please expose all service parameters so they can be automated for service deployment. This should include the basic things like "overWriting" a service as well as enabled or disabling capabilities and parameters. 

Provide arcpy classes, modules or methods to allow a complete arcpy publication solution and similar workflow to the Desktop publication workflow.

We should not have to resort to xml or file based hacks to publish services in the same manner as Desktop. Support will not provide examples on modifying the SDDraft and the documentation does not even mention this limitation or ability so a solution needs be provided. 

[AngelaSchirck]

So you're trying to publish layers on a federated server (or hosting server?) using a script rather than actually going through the process for each layer?

I have written a script and made a geoprocessing tool that allows you to select one, or several layers and and specify certain parameters and publishes them to a federated server.  Is this what you're trying to accomplish?

[StaticK]

Yes, I appreciate this, and would ask to expand it to include MapService/FeatureService (all of them) as well.

Having to parse the sddraft XML to change certain settings should be unnecessary. It should be an easy property setting, such as doing sddraft.FeatureService = True and sddraft.FeatureService.maxRecordCount = 10000 instead of iterating through every XML tag for each changed property with the 4++ nested if for ... if for ... if for- using FeatureSharingDraft as an example:

# Find all elements named TypeName
# This is where the additional layers and capabilities are defined
typeNames = doc.getElementsByTagName('TypeName')
for typeName in typeNames:
    # Get the TypeName to enable
    if typeName.firstChild.data == "FeatureServer":
        extension = typeName.parentNode
        for extElement in extension.childNodes:
            if extElement.tagName == 'Definition':
                for propArray in extElement.childNodes:
                    if propArray.tagName == 'Info':
                        for propSet in propArray.childNodes:
                            for prop in propSet.childNodes:
                                for prop1 in prop.childNodes:
                                    if prop1.tagName == "Key":
                                        if prop1.firstChild.data == 'webCapabilities':
                                            if prop1.nextSibling.hasChildNodes():
                                                prop1.nextSibling.firstChild.data = "Create,Sync,Query"
                                            else:
                                                txt = doc.createTextNode("Create,Sync,Query")
                                                prop1.nextSibling.appendChild(txt)

This is ugly and introduces a ton of chances for errors. If the code exists to generate the xml in the exportToSDDraft, why cant it be expanded to take a list of given parameters to generate a complete service without having to touch the XML?

sddraft.credits = "These are credits"
sddraft.description = "This is description"
sddraft.summary = "This is summary"
sddraft.tags = "tag1, tag2"
sddraft.useLimitations = "These are use limitations"
sddraft.serverFolder = "MyServerFolder"

sddraft.FeatureService = True

sddraft.FeatureService.webCapabilities = "Create,Sync,Query"

sddraft.FeatureService.allowTrueCurvesUpdates'= 'false'

sddraft.FeatureService.maxRecordCount = 10000

You can't really debug/ build your script until you save the .sd and .sddraft so it makes it hard to know where certain properties / settings should go, what xml tags they are children of, and makes it harder to incorporate services such as FeatureService. The only reference to FeatureService is in the MapService documentation in the code example accessed via xml. What settings can I set on it? All I see are the webCapabilities or items from the FeatureSharing- does it have the same properties as the MapService?

# Defaults are Query,Create,Update,Delete,Uploads,Editing

[MatthewGeorge]

@AngelaSchirck 

I'm trying to publish layers to a federated server. I'm able to achieve this programmatically via arcpy, however, if I compare the same layer published using arcpy against one published using the standard workflow from Pro - right click the layer in the Contents Pane > Sharing > Share As Web Layer - the outcome is different. A few things I've found:

• If the FeatureServer extension is enabled, when accessing the layer from Pro (go to Portal from the Catalog and right click the Feature Service > Add to Current Map) the single layer will be nested in a group layer of the same name but if the exact same layer is published using the standard workflow, adding the Feature Service to Pro in the same way will add only the single inner layer inside the Feature Service (screenshots in the link in the original post to explain this more clearly).
• It's minor, but the thumbnail isn't included and I can't see where in the SD draft to specify a thumbnail. I understand this can be achieved using the arcgis module after the fact but would prefer to have this as a part of the original publishing process. It probably is but there's no documentation that explains where it might be set in the SD draft file.
• The Default Visibility property that is seen in the REST API for the FeatureServer/MapServer doesn't honour the visibility setting of the layer being published. Not sure if this is somehow related to the first point.

I'm sure there are other properties that I haven't come across yet but generally, it would be good to either have some off-the-shelf functions that make this process simpler, or just have some better documentation on the different properties that can be set in an SD draft and how they impact the published service.

[MatthewGeorge]

In addition to the info I've already provided, I've also attempted to replicate the exact process that executes using the standard workflow from Pro but even copying those artefacts - msd file and sddraft file - then updating them to use the layer being published ends up failing with an ERROR 9999 when arcpy.server.StageService() is run. I've compared the edited files with the originals too many times to count and there is nothing that I can see that would be causing the SD draft to fail to stage. This makes me think that there is something else that Pro does - uses a different method for publishing than the suggested publishing steps via arcpy.

[AngelaSchirck]

Okay,

Well I made most of my changes by editing the XML sddraft file, and my program is tailored to the specific settings we use when publishing to the Federated server (e.g. branch versioning, etc).  I found no "easy way" to do this unfortunately, but I found the settings that needed changed in the sddraft file by creating one first, then looking through it to find where the various settings were.  This was a very tedious process, and it would actually be nice if ESRI would make a geoprocessing tool that would publish multiple layers.  What I have works for us but may not work for another organization.  I have made it so some settings can be changed (layers to publish, map name, portal and server folders, sharing, etc) but the things that we need static are built into the script (e.g. branch version, timezone, schema locks disabled, etc). 

This script took me quite a bit of time to develop, so I totally understand your frustration!

Good luck, and maybe ESRI will listen! 🙂

[MatthewGeorge]

@AngelaSchirck 

Sounds exactly the same as I've done - created a tool that sets the desired settings for our org and publishes to the desired federated server - we have a specific server mapped to each database. This is working but the main issue I can't get past is the nested single layer in the Feature Service after publishing is complete. It took me hours of scouring sd draft files created using the standard workflow to finally get to this point but it shouldn't be that hard in my opinion.

[AngelaSchirck]

@MatthewGeorge 

I agree, it shouldn't be that difficult.

[SusanTran]

In Pro 3.4, the arcpy sharing draft classes were enhanced with additional properties that match the configurable properties available in the user interface.