add a SwissTopo WMS layer to an existing arcgis online map using arcgis API for python

3647
9
03-09-2021 02:08 AM
JKaelin
New Contributor II

I have been trying to add (or move) a SwissTopo WMS layer to an existing arcgis Online map using the arcgis API for python.

Using the API, I can add a set a selected set of SwissTopo layers using WMS as a new map to my arcgis Online portal. (I am happy to share code for this if anyone is interested.)

I cannot add (or move) the WMS layer to another map. 

I cannot find any API documentation addressing this, nor a useful answer after web searching.,

I can do the above manually on arcgis Online, but my requirements are to script this using python.

Are there examples or tutorials available for handling SwissTopo WMS layers with the arcgis API for python? Perhaps a Jupyter Notebook?

Thanks in advance for replies.

0 Kudos
9 Replies
JoëlHempenius3
Occasional Contributor

You should register the WMS (layers) to your portal first. From there it's possible to search for the registered layers and add them to any map you want.

This code should help you registering the WMS layers, the script has a lot of options, and I'm not sure whether they all work for your WMS. 

 

from arcgis import GIS
import requests 
from xml.etree import ElementTree
gis = GIS("pro") #using the very easy arcgis pro login, run this script from arcgis pro python console and make sure you're logged in

#inputs
wmsurl = 'https://the.wms.server/geoserver/namespace/wms?' #should end with ?
mapurl = 'https://the.wms.server/geoserver/namespace/ows?' #should end with ?
featureinfourl = 'https://the.wms.server/geoserver/namespace/ows?' #should end with ?
capabilitiesurl = wmsurl + 'SERVICE=WMS&request=getcapabilities' #should end with ?
xmlfile = r'path/to/yourcapabilitiesdocumentlocation\wms.xml'
portalproxy = None #r'https://your.portal.server/portal/sharing/proxy?' change this to you portal proxy when you want to want to proxy all requests through portal, 

#layerdefautls and prefixes and postfixes
thumbnailfile = r'path/to/your/logofile'
usethumbnailfromWMS = True
downloadcapabilities = True # set this to False and save the getcapabilities xml to specified location yourself. could be helpful if your struggling with a corporate proxy
layerdescriptionpostfix =  'generic text about this layer'
defaultaccessInformation = 'CopyrightHolder'
snippetprefix = 'text to insert before layertitle in layer summary'
snippetpostfix = 'text to insert after layertitle in layer summary'
layerlicenseinfo = 'licenseinfo and use constraints go here'
defaulttags = 'Tags,Go,here'
useWMSkeywordsAsTags = True #set this to False if you only want the default tags
folder = None #'yourfolder' change to a string with existing folder name in your portal content

if downloadcapabilities:
    resp = requests.get(wmsurl + 'service=wms&request=getcapabilities&version=1.3.0')
    with open(xmlfile, 'w') as f:
        f.write(resp.text)

if portalproxy is not None:
    wmsurl = portalproxy + wmsurl
    mapurl = portalproxy + mapurl
    featureinfourl = portalproxy + featureinfourl
    


tree = ElementTree.parse(xmlfile) 
root = tree.getroot()
for capability in root.findall('{http://www.opengis.net/wms}Capability'): 
    for rootlayer in capability.findall('{http://www.opengis.net/wms}Layer'): #root layer
        #here we get all supported crs 
        crsresult = []
        for crs in rootlayer.findall('{http://www.opengis.net/wms}CRS'): 
            crstext = crs.text
            if crstext.startswith("EPSG:"):
                crsresult.append(crstext.lstrip('EPSG:'))
        for layer in rootlayer.findall('{http://www.opengis.net/wms}Layer'): 
            name = layer.find('{http://www.opengis.net/wms}Name').text
            title = layer.find('{http://www.opengis.net/wms}Title').text
            abstract = layer.find('{http://www.opengis.net/wms}Abstract').text
            if abstract is None:
                abstract = '' 
            bboxelement = layer.find('{http://www.opengis.net/wms}EX_GeographicBoundingBox')
            xmin = bboxelement.find('{http://www.opengis.net/wms}westBoundLongitude').text
            xmax = bboxelement.find('{http://www.opengis.net/wms}eastBoundLongitude').text
            ymin = bboxelement.find('{http://www.opengis.net/wms}southBoundLatitude').text
            ymax = bboxelement.find('{http://www.opengis.net/wms}northBoundLatitude').text
            bbox = '{},{},{},{}'.format(xmin,ymin,xmax,ymax)
            layertags = defaulttags
            keywordlist = layer.find('{http://www.opengis.net/wms}KeywordList')
            if keywordlist is not None:
                for keyword in keywordlist.findall('{http://www.opengis.net/wms}Keyword'):
                    layertags = layertags+"," + keyword.text
            attributionelement = layer.find('{http://www.opengis.net/wms}Attribution')
            accessInformation = defaultaccessInformation
            if attributionelement is not None:
                accessInformation = defaultaccessInformation + ',' + attributionelement.find('{http://www.opengis.net/wms}Title').text
            
            properties['title'] = title
            properties['type'] = 'WMS'
            properties['typeKeywords'] =  'Data, Service, Web Map Service, OGC'
            properties['tags'] =  layertags
            properties['url'] =  wmsurl
            properties['description'] = abstract + layerdescriptionpostfix
            properties['accessInformation'] = accessInformation
            properties['snippet'] = snippetprefix + title  + snippetpostfix
            properties['licenseInfo'] = layerlicenseinfo
            properties['text'] = '{"title":"WMS","url":"'+wmsurl+'","mapUrl":"'+mapurl+'","version":"1.3.0","layers":[{"name":"'+ name+'","title":"'+title+'","legendURL":"'+ mapurl + 'service=WMS&request=GetLegendGraphic&format=image%2Fpng&width=20&height=20&layer='+name+'","queryable":true}],"copyright":"none","maxHeight":5000,"maxWidth":5000,"spatialReferences":['+','.join(crsresult)+'],"format":null,"featureInfoUrl":"' +featureinfourl+'","featureInfoFormat":"text/html"}'
            properties['extent'] = bbox
            properties['thumbnailURL'] = mapurl + 'SERVICE=WMS&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=TRUE&STYLES=&VERSION=1.3.0&LAYERS='+name+'&WIDTH=800&HEIGHT=532&CRS=EPSG:4326&BBOX=50.720978888973335,3.238626318098767,53.59715706128808,7.262998577472529'
            if usethumbnailfromWMS:
                thumbnailfile = properties['thumbnailURL']
            gis.content.add(properties,thumbnail = thumbnailfile, folder=folder)
-Joël Hempenius.

Languages: JavaScript, Python and Dunglish
JKaelin
New Contributor II

Thanks a lot for your reply.

The gis.content.add() method is what I have been using. This puts the retrieved WMS layer(s) into a new map, which has the name given by the item title.

Now if I do this several times, I end up with a set of "WMS" maps, each of which has one set of retrieved WMS layers.

What I want is one map in arcgis online that has all the retrieved WMS layer sets as layers. This can be done manually. I want to script it.

As far as I can see, your method is a very nice elaboration of what I am doing (and I will probably use it to harden my procedure assuming that is ok with you). However, it doesn't solve my issues as described above.

Any more ideas?
Thanks again :) 


 

 

 

0 Kudos
JKaelin
New Contributor II

Just for clarification:

> You should register the WMS (layers) to your portal first.
Yes, this is what I am doing. This works for me. The WMS layers are on my portal.

>From there it's possible to search for the registered layers and add them to any >map you want.
This is where I am having difficulty with the arcgis API. Any hints would be appreciated. 

 

0 Kudos
JoëlHempenius3
Occasional Contributor

this should probably work, haven't tested it though

first I get the Portal registered WMS, I retrieve the url and create a new WMSlayer with that url and then I add the first layer. 

from arcgis.gis import GIS
from arcgis.mapping.ogc import WMSLayer
from arcgis.mapping import WebMap

wmslayers = gis.content.search("DataCentraWarmte", item_type="WMS", max_items=5)
print (wmslayers[0].itemid)
layer = WMSLayer(wmslayers[0].url)
wm = WebMap()  
wm.add_layer(layer.layers[0])

 

Would be much easier when you could just add the item to the map and let the Python api find out what kind of layer the item contains

-Joël Hempenius.

Languages: JavaScript, Python and Dunglish
JKaelin
New Contributor II

Once more, many thanks.

Unfortunately using this procedure still doesn't achieve what I need, as described above.

First I fixed a typo in your code:

 

wmslayers = gis.content.search("DataCentraWarmte", item_type="WMS", max_items=5)
print (wmslayers[0].itemid)
layer = WMSLayer(wmslayers[0].url)
wm = WebMap()  
wm.add_layer(layer)  # FIXED

 

Then I see that the output does not match the WMS 'layers' that I select for gis.content.add(). What happens (for me, might be different for your WMS) is that WMSLayer(wmslayers[0].url just gets the SwissTopo WMS URL with no layers selected, resulting in my case with hundreds of layers being put into the map.

So still no joy.

I have been going through various methods systematically and all the methods I have tried have issues with a WMS layer :(

I am starting to think this may be an ESRI feature  ;)
Hopefully I am wrong.

0 Kudos
JoëlHempenius3
Occasional Contributor

 (and I will probably use it to harden my procedure assuming that is ok with you)

Yes, please steal my code :)

 

My last code example wasn't my best one. I knew I had to do something with the layers object within the WMS layer, but not that way.

If you want a subset of all the layers in the WMS you should use this. This time I tested my code:

This WMS had three layers. I only want the first one visible in my map

JoëlHempenius3_3-1615395688395.png

 

JoëlHempenius3_1-1615395341942.png

using this line of code,I overwrite the layer list in my WMS layer (wm.layers[0]) with a new list containing only the first layer

wm.layers[0].layers = [wm.layers[0].layers[0]]

Which gives me this:

JoëlHempenius3_0-1615395267831.png

 

Now, when I run:

wm.update()

The map is saved and only the remaining layer is visible in the Map viewer

 

JoëlHempenius3_2-1615395550360.png

Hope this helps

 

 

-Joël Hempenius.

Languages: JavaScript, Python and Dunglish
JKaelin
New Contributor II

Thanks again for your tips. Using WMSlayer.layers property is a potential workaround, but apparently there are layer indexes in webmap that also need to be set (?). WMSlayer however does not have any methods associated with it.

I have now done a fair amount of testing for handling WMS layers with the arcgis API for Python for arcgis Online. My conclusions so far are:
- WMS layers can be created as content
- a practical way to add WMS layers to a webmap was not found
- the documentation, API methods and exception handling for WMS layers are rudimentary.

ESRI developers present in this forum (hopefully) are welcome to respond.

A code snippet of my efforts follows.

 

# test retrieving WMS layers and putting these onto a webmap on arcgis online portal

from arcgis.gis import GIS
from arcgis.mapping import WebMap
from arcgis.mapping.ogc import WMSLayer
import json

gis = GIS("https://poyry.maps.arcgis.com/", USERNAME, PASSWD)


# retrieve WMS layers
spatialReferences = [2056, 2400, 3006, 3007, 3008, 3009, 3010, 3011, 3012, 3013, 3014, 3015, 3016, 3017, 3018, 
                     3021, 3031, 3032, 3033, 3035, 3785, 3857, 4326, 30061, 30062, 900913, 3152, 30063, 30064]
# above list works -> need to research exactly what is needed
layers= [
        {
        "name": "ch.swisstopo.geologie-eiszeit-lgm",
        "title": "Letzteiszeitl. Max. (Vektor) 500",
        "queryable": True
        },
        {
        "name": "ch.swisstopo-karto.skitouren",
        "title": "Skirouten",
        "queryable": True
        }
        ]
text_layers = {
                "title": "SwissTopo",
                "url": "http://wms.geo.admin.ch/",
                "version": "1.3.0",
                "layers": layers,
                "spatialReferences": spatialReferences ,
                "format": None,
                "featureInfoFormat": "text/html"
              }
item={
        'title':'SwissTopo_WMS_testing',
        'description':'SwissTopo_testing2',
        'tags':'wms',
        'type':'WMS',
        'url':'http://wms.geo.admin.ch/',
        'extent': {'xmin': 5.83581, 'ymin': 45.65916, 'xmax': 10.97931, 'ymax': 47.86991},  # not working
        'text': json.dumps(text_layers)
     }
swisstopo_testing_wms = gis.content.add(item_properties=item)
print("swisstopo_testing_wms: ", swisstopo_testing_wms)
display(swisstopo_testing_wms)  # results in 2 WMS layers grouped under a single webmap layer, OK


# put WMS layers onto webmap
# attempt #1 using .add_layer() -> NG since swisstopo_testing_wms.url is general to all SwissTopo layers
layer = WMSLayer(swisstopo_testing_wms.url)
wm = WebMap()  
wm.add_layer(layer)
wm_item_properties = {'title':'WMS_test',
                      'snippet':'Map created using Python API',
                      'tags':['python'],
                     }
wm.save(wm_item_properties)  # puts all 705 SwissTopo layers (!) onto a webamp on portal

# attempt #2 using .publish() -> NG as content.publish() method does not wor for WMS
#swisstopo_htesting_feaure_layer_item = swisstopo_testing_wms.publish(
#                                                            publish_parameters={'file_type':"WMS"})
#   ValueError: A file_type must be provide, data format not recognized

# attempt #3 using WMSlayer.layers property -> NG as wrong layer is displayed in Map Viewer
wm.layers[0].layers = wm.layers[0].layers[501]  # replace 705 SwissTopo layers with a single layer
#wm.update()
# RuntimeError: Item object missing, you should use `save()` method if you are creating a new web map item  ??
wm.save(wm_item_properties)  # results in a single webmap layer entry, which is correctly shown in Overview,
                             # however the Map Viewer displays the wrong layer !

 

 

0 Kudos
JoëlHempenius3
Occasional Contributor

I totally agree with your conclusions. Maybe somebody from Esri can take a look at them? @AndrewChapkowski 

 

 

-Joël Hempenius.

Languages: JavaScript, Python and Dunglish
0 Kudos
achapapkowski
New Contributor II

I'll add it to the backlog of issues to get it on our board. 

Thank you for all the code and the description of the problem.  The ArcGIS API team will take a look at this Post 1.8.5 release, which will occur next month.