Popup information not published to ArcGIS Enterprise when using Arcpy?

[JoëlHempenius3]

I've automated sharing my mapservices to ArcGIS Enterprise using arcpy and now I see a difference on the configured popups between the services I share manually from Pro and the services I Share using Arcpy. When I share manually, the popup information is also shared in ArcGIS Enterprise and when I configure the mapservice in a new map in ArcGIS Enterprise the popup configuration  is the same as ArcGIS Pro. When I share the service from Arcpy and add the service to a new map in ArcGIS Enterprise, there is no popup configuration and I need to do the popup configuration again in the webmap. 

Using Ago-assistant, I dug a little deeper and found the service I shared manually has a data section containing the popup configuration and the service I shared using Arcpy doesn't have this.

Screenshots to illustrate this:

The services shared manually from ArcGIS Pro has the data section bottom right with the popup info:

The service shared using Arcpy doesn't show the data item:

From the network tab in dev console I see a /data? request, but this gives a 200OK with 0 content length. 

I tried a very minimal script from this page (MapImageSharingDraft—ArcGIS Pro | Documentation )  and modified it only to publish my pro project to my server, but this gives the same result.

Does anybody know how I can publish the popupinfo with the service from Arcpy?

[MaxSquires]

That was a very well written question and i thought it had put me on the track to finding the answer i'm looking for as well, but alas, i had to work around it.  meanwhile, here are my thoughts on the matter:

It seems to me that given the additional data section you mention here we could use python requests to give the server the data parameter, but that doesn't explain why the arcpy sharing of the service is stripping out this inherent functionality (configure popups) of arcgis pro. 

I have inspected the service definition file that is generated in the arcpy publication process and it has the popupinfo section but it is ignored at the upload service definition stage.

unfortunately i was unable publish the service with the popups reflected in the service as i wanted.

it seems to be a function of the hosted feature layer. i can't seem to add a data element to an existing feature service.  but a hosted feature layer does have this property.  well, i should say, the property is not empty in the case of the hosted feature layer.

the problem with that, as i understand it, is that if you are working with live data, the hosted feature layer resides in the internal portal database and therefore would need to be recreated every time the data updates in the real backend.  

I may have missed something obvious here and am eagerly awaiting someone to point it out to me.

for me the workaround is injecting the popup info in the client application.  this is not ideal because anywhere else you consume this service the popups will also have to be manually configured i.e. anywhere else in the ArcGIS Enterprise. 

import * as somelayer_popup_template from '../../../content/popupTemplates/somelayer.json';

//more stuff

if (featLayer.title.toLowerCase() === "some layer title") {
                                        featLayer.popupTemplate = PopupTemplate.fromJSON(somelayer_popup_template.popupInfo);
                                    }

where

somelayer.json is like:

{"popupInfo": {

	"title": "Some Value: {some_fieldname}",
	"fieldInfos": [{
		"fieldName": "some_field",
		"visible": false,
		"isEditable": false,
		"format": {
			"places": 0,
			"digitSeparator": true
		},
                 ...],
	"description": "<div>some html here</div>"
        }
}

maybe someone knows of a way to post a new data element to an existing portal item?

[JoëlHempenius3]

Posting a new data element to an existing portal item is easy using the python api:

given that you have succesfully logged in to your GIS and have the itemid of the mapservice and a portalLogo File path variable and you also want to set the description in Portal for the mapservice:

portalitem = gis.content.get(itemid)
item_properties ={ 'description' :  description}
item_properties['text'] =data
portalitem.update(item_properties=item_properties, data=None, thumbnail=portalLogo, metadata=None)

and the data object should be structured like this, in this case I have two layers with "Allow assignment of unique numeric IDs for sharing web layers" enabled in the map:

{
    "layers": [
        {
            "id": 0,
            "name": "Adreslocatie",
            "popupInfo": {
                "title": "{namespace}",
                "mediaInfos": [],
                "fieldInfos": [
                    {
                        "fieldName": "OBJECTID",
                        "visible": true,
                        "isEditable": false,
                        "label": "OBJECTID"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 6,
                            "digitSeparator": false
                        },
                        "label": "BAR_ID"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "label": "LAST_MODIFIED"
                    },
                    {
                        "fieldName": "ID",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 6,
                            "digitSeparator": false
                        },
                        "label": "ID"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "label": "Bag Id"
                    },
                    {
                        "fieldName": "BAG_OBJECT",
                        "visible": true,
                        "isEditable": true,
                        "label": "Bag Object"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "label": "GEOMETRIE_AANWEZIG"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 6,
                            "digitSeparator": false
                        },
                        "label": "ORIENTATION"
                    },
                    {
                        "fieldName": "GEMEENTENAAM",
                        "visible": true,
                        "isEditable": true,
                        "label": "GEMEENTENAAM"
                    },
                    {
                        "fieldName": "HUISLETTER",
                        "visible": true,
                        "isEditable": true,
                        "label": "HUISLETTER"
                    },
                    {
                        "fieldName": "HUISNUMMER",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 0,
                            "digitSeparator": false
                        },
                        "label": "HUISNUMMER"
                    },
                    {
                        "fieldName": "HUISNUMMER_TOEVOEGING",
                        "visible": true,
                        "isEditable": true,
                        "label": "Huisnummer Toevoeging"
                    },
                    {
                        "fieldName": "PLAATSNAAM",
                        "visible": true,
                        "isEditable": true,
                        "label": "PLAATSNAAM"
                    },
                    {
                        "fieldName": "POSTCODE",
                        "visible": true,
                        "isEditable": true,
                        "label": "POSTCODE"
                    },
                    {
                        "fieldName": "STRAATNAAM",
                        "visible": true,
                        "isEditable": true,
                        "label": "STRAATNAAM"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "label": "SE_ANNO_CAD_DATA"
                    }
                ],
                "popupElements": [
                    {
                        "fieldInfos": [
                            {
                                "fieldName": "OBJECTID",
                                "visible": true,
                                "isEditable": false,
                                "label": "OBJECTID"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 6,
                                    "digitSeparator": false
                                },
                                "label": "BAR_ID"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "label": "LAST_MODIFIED"
                            },
                            {
                                "fieldName": "ID",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 6,
                                    "digitSeparator": false
                                },
                                "label": "ID"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "label": "Bag Id"
                            },
                            {
                                "fieldName": "BAG_OBJECT",
                                "visible": true,
                                "isEditable": true,
                                "label": "Bag Object"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "label": "GEOMETRIE_AANWEZIG"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 6,
                                    "digitSeparator": false
                                },
                                "label": "ORIENTATION"
                            },
                            {
                                "fieldName": "GEMEENTENAAM",
                                "visible": true,
                                "isEditable": true,
                                "label": "GEMEENTENAAM"
                            },
                            {
                                "fieldName": "HUISLETTER",
                                "visible": true,
                                "isEditable": true,
                                "label": "HUISLETTER"
                            },
                            {
                                "fieldName": "HUISNUMMER",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 0,
                                    "digitSeparator": false
                                },
                                "label": "HUISNUMMER"
                            },
                            {
                                "fieldName": "HUISNUMMER_TOEVOEGING",
                                "visible": true,
                                "isEditable": true,
                                "label": "Huisnummer Toevoeging"
                            },
                            {
                                "fieldName": "PLAATSNAAM",
                                "visible": true,
                                "isEditable": true,
                                "label": "PLAATSNAAM"
                            },
                            {
                                "fieldName": "POSTCODE",
                                "visible": true,
                                "isEditable": true,
                                "label": "POSTCODE"
                            },
                            {
                                "fieldName": "STRAATNAAM",
                                "visible": true,
                                "isEditable": true,
                                "label": "STRAATNAAM"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "label": "SE_ANNO_CAD_DATA"
                            }
                        ],
                        "type": "fields"
                    }
                ],
                "expressionInfos": [],
                "description": "",
                "layerOptions": {
                    "showNoDataRecords": true
                },
                "relatedRecordsInfo": null,
                "showAttachments": true,
                "showLastEditInfo": true
            }
        },
        {
            "id": 1,
            "name": "Markeringsobject (locatie)",
            "popupInfo": {
                "title": "{namespace}",
                "mediaInfos": [],
                "fieldInfos": [
                    {
                        "fieldName": "OBJECTID",
                        "visible": true,
                        "isEditable": false,
                        "label": "OBJECTID"
                    },
                    {
                        "fieldName": "BAR_ID",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 6,
                            "digitSeparator": false
                        },
                        "label": "BAR_ID"
                    },
                    {
                        "fieldName": "LAST_MODIFIED",
                        "visible": true,
                        "isEditable": true,
                        "label": "LAST_MODIFIED"
                    },
                    {
                        "fieldName": "ID",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 6,
                            "digitSeparator": false
                        },
                        "label": "ID"
                    },
                    {
                        "fieldName": "OPGEVOERD_DOOR",
                        "visible": true,
                        "isEditable": true,
                        "label": "Opgevoerd Door"
                    },
                    {
                        "fieldName": "GEWIJZIGD_DOOR",
                        "visible": true,
                        "isEditable": true,
                        "label": "Gewijzigd Door"
                    },
                    {
                        "fieldName": "OPGEVOERD_DATUM",
                        "visible": true,
                        "isEditable": true,
                        "label": "Opgevoerd Datum"
                    },
                    {
                        "fieldName": "GEWIJZIGD_DATUM",
                        "visible": true,
                        "isEditable": true,
                        "label": "Gewijzigd Datum"
                    },
                    {
                        "fieldName": "NAN",
                        "visible": true,
                        "isEditable": true,
                        "label": "NAN"
                    },
                    {
                        "fieldName": "SOORT",
                        "visible": true,
                        "isEditable": true,
                        "label": "SOORT"
                    },
                    {
                        "fieldName": "OPMERKING",
                        "visible": true,
                        "isEditable": true,
                        "label": "OPMERKING"
                    },
                    {
                        "fieldName": "GEOMETRIE_AANWEZIG",
                        "visible": true,
                        "isEditable": true,
                        "label": "GEOMETRIE_AANWEZIG"
                    },
                    {
                        "fieldName": "ORIENTATION",
                        "visible": true,
                        "isEditable": true,
                        "format": {
                            "places": 6,
                            "digitSeparator": false
                        },
                        "label": "ORIENTATION"
                    },
                    {
                        "fieldName": "",
                        "visible": true,
                        "isEditable": true,
                        "label": "SE_ANNO_CAD_DATA"
                    }
                ],
                "popupElements": [
                    {
                        "fieldInfos": [
                            {
                                "fieldName": "OBJECTID",
                                "visible": true,
                                "isEditable": false,
                                "label": "OBJECTID"
                            },
                            {
                                "fieldName": "BAR_ID",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 6,
                                    "digitSeparator": false
                                },
                                "label": "BAR_ID"
                            },
                            {
                                "fieldName": "LAST_MODIFIED",
                                "visible": true,
                                "isEditable": true,
                                "label": "LAST_MODIFIED"
                            },
                            {
                                "fieldName": "ID",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 6,
                                    "digitSeparator": false
                                },
                                "label": "ID"
                            },
                            {
                                "fieldName": "OPGEVOERD_DOOR",
                                "visible": true,
                                "isEditable": true,
                                "label": "Opgevoerd Door"
                            },
                            {
                                "fieldName": "GEWIJZIGD_DOOR",
                                "visible": true,
                                "isEditable": true,
                                "label": "Gewijzigd Door"
                            },
                            {
                                "fieldName": "OPGEVOERD_DATUM",
                                "visible": true,
                                "isEditable": true,
                                "label": "Opgevoerd Datum"
                            },
                            {
                                "fieldName": "GEWIJZIGD_DATUM",
                                "visible": true,
                                "isEditable": true,
                                "label": "Gewijzigd Datum"
                            },
                            {
                                "fieldName": "NAN",
                                "visible": true,
                                "isEditable": true,
                                "label": "NAN"
                            },
                            {
                                "fieldName": "SOORT",
                                "visible": true,
                                "isEditable": true,
                                "label": "SOORT"
                            },
                            {
                                "fieldName": "OPMERKING",
                                "visible": true,
                                "isEditable": true,
                                "label": "OPMERKING"
                            },
                            {
                                "fieldName": "GEOMETRIE_AANWEZIG",
                                "visible": true,
                                "isEditable": true,
                                "label": "GEOMETRIE_AANWEZIG"
                            },
                            {
                                "fieldName": "ORIENTATION",
                                "visible": true,
                                "isEditable": true,
                                "format": {
                                    "places": 6,
                                    "digitSeparator": false
                                },
                                "label": "ORIENTATION"
                            },
                            {
                                "fieldName": "",
                                "visible": true,
                                "isEditable": true,
                                "label": "SE_ANNO_CAD_DATA"
                            }
                        ],
                        "type": "fields"
                    }
                ],
                "expressionInfos": [],
                "description": "",
                "layerOptions": {
                    "showNoDataRecords": true
                },
                "relatedRecordsInfo": null,
                "showAttachments": true,
                "showLastEditInfo": true
            }
        }
    ]
}

This solved the problem for me and I get popups on the mapservice after publishing from python with the code example

[Mostafa_Ghiyasvand]

Very helpful!! Thank you so much. It still took one day for me to change the code a bit to got it to work in my project. Here the changes that I made:

portalitem = gis.content.get(itemid)
item_properties ={ 'description' : description}
item_properties['text'] =data
portalitem.update(item_properties=item_properties)

[Brian_Wilson]

I hit this problem yesterday. After finally getting a script to reliably publish/overwrite map image layers and feature services, and designing beautiful popups in Pro, I found it will not publish the popups.

I see in content/items folder that there is no JSON file when I use arcpy. Is that what you are talking about here? You are adding it with the "update" method?

If so, where do you get the JSON data that you are adding? I can see you opening up the SD file and there is a mapx file in there.

The mapx does indeed have popupInfo.

Your code sample is opaque to me though. What is this doing? How do I get from a CIM popup to one that can be sent to the service?

import * as somelayer_popup_template from '../../../content/popupTemplates/somelayer.json';

//more stuff

if (featLayer.title.toLowerCase() === "some layer title") {
                                        featLayer.popupTemplate = PopupTemplate.fromJSON(somelayer_popup_template.popupInfo);
                                    }

Thanks, Brian

[Mostafa_Ghiyasvand]

Hello Brian,
So, I first set up the pop-ups in ArcGIS Pro and then I got the json file of pop-ups using ArcGIS Assistant tool and saved it separately. Then I tried to use the Update method to leverage the json file.
Please let me know if have more questions.
Best,

[Brian_Wilson]

AH brilliant! You let PRO generate the JSON, then copy it. I had already published from PRO so I should be able to read the PRO service and then update the arcpy server. Got it. Thanks.

[Brian_Wilson]

You showed me how to do something that appears completely opaque but I am okay with it... I don't see any meaning in the documentation for the item_properties['text'] element. It says

text	Optional string. For text based items such as Feature Collections & WebMaps

I don't see why setting 'text' to the data would do anything useful at all. But it works so I will just be grateful press on!

[SimonSchütte_ct]

-> Bug BUG-000139253
Pop-ups do not maintain their configuration in ArcGIS Online when publish from ArcGIS Pro using Python.
https://support.esri.com/en/bugs/nimbus/QlVHLTAwMDEzOTI1Mw==

(same Problem with ArcGIS Enterprise, since this is an ArcGIS Pro Bug)

[Brian_Wilson]

I wrote my way around this bug.

I already read a map from an APRX file and publish a Map Image Layer and Feature Service from it.

Now before I write the services, I read the CIM info for each layer, and build the JSON to configure the popup. Then using the in-memory python dict representation of the popup I use the same update call talked about above.

This way I no longer have to publish from pro to generate the popup. There is one hurdle that I am ignoring for the moment; it seems that the field definitions are only written to the CIM layers when they are redefined in Pro, I do this routinely to control what fields are visible in a map so it was no hardship for me.

If anyone is interested I have most of the code in a github repo already, I could clean it up and release the URL. This is the core code though.

"""
    Read from an arcpy map and
    build JSON for a layer list that can be used to define popups.
"""
import os
import arcpy
import json
import subprocess
from config import Config


def makeLayer(layer: arcpy._mp.Layer) -> dict:
    """ 
    Make a popup layer from an arcpy layer.
    Returns a dict or None. 
    """
        
    cimLayer = layer.getDefinition('V2')
    if cimLayer.layerType != 'Operational':
        return None

    showPopups = cimLayer.showPopups
    id = cimLayer.serviceLayerID
    name = cimLayer.name
    popfile = os.path.join(Config.SCRATCH_WORKSPACE, name + '.html')

    print(f'layer name: "{name}" id:{id}, showPopups={showPopups}')
    
    fieldInfoList = list()

    # There won't be any fieldDescriptions for a layer
    # unless you alter the data view in Pro.
    # If there are no FD's then I guess you have to
    # read the field list in the feature class directly.
    # I don't feel like doing that today.
    if not len(cimLayer.featureTable.fieldDescriptions):
        connection = cimLayer.featureTable.dataConnection.workspaceConnectionString
        print(f"There are no fields described for {id} in your map.")
        return None
    
    for field in cimLayer.featureTable.fieldDescriptions:
        if not field.visible: continue
        d = {
            "fieldName": field.fieldName,
            "label": field.fieldName,
        }
        if field.numberFormat:
            d["format"] = {
                "places": field.numberFormat.roundingValue,
                "digitSeparator": field.numberFormat.useSeparator
            }
        else:
            d["stringFieldOption"] = "textbox"
        
        fieldInfoList.append(d)

    popup_info = cimLayer.popupInfo

    expressionList = list()
    for e in popup_info.expressionInfos:
        expressionList.append({
            'expression': e.expression,
            'title': e.title,
            'name': e.name,
            'returnType': e.returnType
        })

    mediaInfoList = list()
    for minfo in popup_info.mediaInfos:
        html = minfo.text
        # Test the HTML by opening it in a browser.
#        with open(popfile, "w") as fp:
#            fp.write(html)
#        if SHOWPOPUP:
#            print("Opening {popfile} in browser")
#            subprocess.check_call([BROWSER, "file:///"+popfile])

    return {
        "id": id,
        "name": name,
        "popupInfo": {
            "fieldInfos": fieldInfoList,
            "expressionInfos": expressionList,
            'mediaInfos': mediaInfoList,
            'description': html
        }
    }


def makePopup(layers: arcpy._mp.Layer) -> dict:
    """
    Using an arcpy layer list as input,
    a list of layers to define a popup, in a dict.
    """
    popupLayers = list()
    for layer in layers:
        # If the layer is not operational (eg a basemap)
        # or no fields are defined in it, None will be returned.
        #
        # Generally since the way we use this is to update
        # a service in Enterprise, empty layers probably 
        # won't hurt us, they will just disappoint us.
        popupDict = makeLayer(layer)
        if popupDict:
            popupLayers.append(popupDict)
    return {"layers": popupLayers}


# ===============================
if __name__ == "__main__":
    # Unit tests

    mapname = "Taxlot Queries"
    aprx = arcpy.mp.ArcGISProject(Config.TAXMAP_APRX)
    map = aprx.listMaps(mapname)[0]
    popupDict = makePopup(map.listLayers())

    print(json.dumps(popupDict,indent=2))

    # For debugging, dump to a file
    json_file = os.path.join(Config.SCRATCH_WORKSPACE, mapname + ".json")
    with open(json_file, 'w') as fp:
        json.dump(popupDict, fp, indent=2)