How to work with map layers in in-memory ArcGIS Pro project created with ConvertWebMapToArcGISProject?

Arne_Gelfert · ‎09-30-2020

TLDR - Since there may be a lot to unpack here, let me start with a nutshell version of what I'm asking:

What's the suggested use of and what are the limitations of web layers and staged vector layers in a aprx-map created from JSON?
What governs whether a property is supported by a specific layer/layer of either type?
What could be behind perfectly good "staged service layers" not exporting to PDF?

Long Version :

I can't seem to figure out how to work with the in-memory ArcGISProcect that is created using 'ConvertWebMapToArcGISProject'. It's straight forward if all you want to do is print what's in a web map or even use a Layout. Beyond that, it blows my mind.

Background - I've managed to capture webmap as JSON in a custom WAB widget and passed that into a custom Python GP service. But working with the map here, I currently run into issues.

When using 'ConvertWebMapToArcGISProject' with a .pagx file,...

result = arcpy.mp.ConvertWebMapToArcGISProject(Web_Map_as_JSON_new,templatePagx,mapframe_name)
aprx = result.ArcGISProject
myMap = aprx.listMaps('Web Map')[0]‍‍‍‍‍‍

you end up with all one set of layers derived from the web map and one set of layers ESRI refers to as "staged vector layers" - those layers that are referenced in your Layout. That was a surprise at first but is actually included in the documentation (Example 3 on this page).

In my case, the layout comes from a Pro project and references the same map that was used to publish the service feeding the web map. So there is a one-to-one match between web layers. Make sense?

Next, since I now have a corresponding layer in my Pro Layout for each layer in the webmap, I remove all the webmap layers. I am not entirely sure what the role/advantage/use of each of the two types of layers here is, or why I can't or shouldn't work with the web layers. That's part of my question. But I proceed to remove them.

for lyr in myMap.listLayers():
    if lyr.isWebLayer:
        myMap.removeLayer(lyr)‍‍‍‍‍‍‍‍‍

Now when I iterate over all my layers, and check visibility, only those that were turned on in the Pro project's map and the state of the Layout when it was used to file are turned on. Upto this point - from web app => Json => to Python GP logic - no issues.

What I want to do next is: dump some layers in a list, turn them all off and then iterate over the list, turning them on as needed, each time creating a PDF. These layers all belong to the same GroupLayer in Pro and the map service. But I want to create a separate PDF for each.

# First turn them all off
for lyr in myListOfLayers:
   lyr.visible = False

# Then turn them on one by one
for i in range(len(myListOfLayers)):
   myListOfLayers[i].visible = True
   
   # create my PDF

   myListOfLayers[i].visible = False‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

As I do so, I can check that the right layers are turned to visible and others aren't. Runs great. The following puts out one PDF for each layer in my list

layout.exportToPDF(layerPDF, resolution = 300)‍‍

The layout of the file that is generated is the desired one from the Pro project. The legend looks correct and shows all the layers. But the map is blank.

These same layers (based on layer.dataSource and layer.connectionProperties are what they should be based on what's in my Pro project. When I run the same code against my Pro project (referencing an APRX file rather than creating one from JSON) using the familiar syntax...

aprx = arcpy.mp.ArcGISProject(<path_to_APRX_File>)‍

it works flawlessly, generates a series of PDF pages and later bundles them into one file. So what am I doing wrong with APRX created from webmap? Why are my maps blank?

Like I said above, I'm not sure why I should be working with the "stages service layers" that 'ConvertWebMapToArcGISProject' creates for you. If I try to use the weblayers instead, this is what happens.

for lyr in myMap.listLayers():
    if not lyr.isWebLayer:
        myMap.removeLayer(lyr)‍‍‍‍‍‍‍‍‍

You'd think this should work (in light of the fact I haven't found any documentation explaining why it shouldn't). But:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-206-cc7d5dde5fc1> in <module>
      1 for layer in myMap.listLayers():
      2     if not layer.isWebLayer:
----> 3         myMap.removeLayer(layer)
      4 

C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\_mp.py in removeLayer(self, remove_layer)
   1801              remove_layer(Layer):
   1802            A reference to a Layer object representing the layer to be removed."""
-> 1803         return convertArcObjectToPythonObject(self._arc_object.removeLayer(*gp_fixargs((remove_layer,), True)))
   1804 
   1805     def removeTable(self, remove_table):
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So removing staged service layers appears to be behaving quite differently from removing webmap layers. It bombs at trying to remove a layer once the parent GroupLayer has been removed. Instead I can do this:

for layer in myMap.listLayers():
    if not layer.isWebLayer:
        try:
            myMap.removeLayer(layer)
        except:
            print("can't remove this one... ")‍‍‍‍‍‍‍‍‍‍‍‍

This removes all "staged vector layers" and leaves you with only web layers. It appears that web layers don't know about GroupLayers but "staged service layers". Hmm. Why?

So now, I try the same as before, I iterate over my list of layers to create PDFs. This is what happens. The code fails after 3 loops with this error:

During handling of the above exception, another exception occurred:

NameError                                 Traceback (most recent call last)
<ipython-input-308-9a972bcf26b6> in <module>
      2     
      3 
----> 4     myListOfLayers.visible = True
      5     
      6     

C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\arcobjects\_base.py in _set(self, val)
    113                     gp.getIDMessage(89013,
    114                         "The attribute %r is not supported on this instance of %s.") %
--> 115                     (attr_name, self.__class__.__name__))
    116     def _del(self, val):
    117         return delattr(self._arc_object, val)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It runs through two layers in the list with no problem and then fails on an almost identical layer #3. Oh my! The first three PDF's that are created look good as far as the map goes. But I've lost my legend. Apparently removing the Layout referenced layers has wiped out the legend. So that suggests I need to work with staged vector layers after all. But there is no obvious reason that layer #3 is any different from 1 or 2 except maybe in def queries.

So what would prevent me from turning off a particular layer or why would a particular layer not support "VISIBLE".
What should I be using web layers for versus "staged services layers"?
What would keep the "staged service layers" from plotting in the PDF when the same code works inside a Pro project?
Does layer visibility (turning on/off) depend on any other layer properties? What am I looking for when comparing as to why one turns of and the other doesn't?
Are there any known limitations to this when using GroupLayers? I've had better luck on another project when not using them. But I don't always have full control over the services I consume.

Don't expect anyone to have answers for all of these. But I'll take any answers or suggestions. This has been very frustrating. If there is better documentation, please point me to it and I will absorb every line.

Arne_Gelfert · ‎10-02-2020

Wow - this one has been a beating. Finally got it to work though. Lessons learned are:

Those web layers behave differently from the staged service layers. They also begin their lives in the new aprx map with different settings, e.g. visibility. They do not work the same for Group Layers. Whether that's by design, I don't know. I finally did get it to work by removing off web layers.
What did the trick for me was when I realized the following lines

legend_element.syncLayerVisibility = True
legend_element.syncLayerOrder = True‍‍

don't mean "keep map and legend synch'd" but they mean "synch map and legend now and only once". So when you're iterating over a list of layers, turning various things on/off, you always have to invoke those lines again to make sure legend and map show the same things.

Now, I'm just running into another ESRI Bug because wanting to turn off Group Layer names in a Pro layout's legend only works for the top entry. BUG-000128688: The Group layer name option in the Format Legend Ite.. So stupid! Looks like maybe it hasn't been fixed in 2.6. None of the work-arounds suggested here.

Well at least I get my PDF maps now and have the ability to rename layers in the legend. Nice side-effect has been that I've dug around the world of CIM as I've been exploring and will be using that a lot more going forward.

Arne_Gelfert · ‎10-02-2020

Turns out hiding the Group Layer labels in the legend can be accomplished by setting them to an empty string.

lyr.name = ""‍

Well... that was easy.