Automating Release Process for Experience Builder App

[fdeters]

How can I automate the Dev>Production promotion process for a map-centric Experience Builder app where the Production version uses different data from the Dev version?

---

Using Python API 2.3.0, AGOL Map Viewer, and AGOL Experience Builder (not Developer Edition)

---

I'm building an Experience Builder app that will be used to view and edit a high-value dataset for my organization. We have an ecosystem in AGOL that includes both Dev and Prod versions of the data. There are multiple views and related tables of the dataset, so there are several items used in the app that have both Dev and Prod versions in our organization.

My strategy so far is to have two versions (Dev and Prod) of both the EB app and the Web Map that it consumes. I make updates in the Dev map and app, then use ArcGIS Assistant to

1. Literally copy the JSON from the Dev versions to the Prod versions ("Data" for the web map; and "Data", "config/config.json", and any other files such as images for the EB app)
2. Update the `url` and `itemId` of the data resources (such as operationalLayers or tables) to the Prod values in the Prod version of the app.

This ArcGIS Assistant process has worked surprisingly well. I especially like that it doesn't entail creating a new Production item every time (I don't want the EB app URL to change).

However, as the app will soon have more environment-specific data sources, I'm looking for a way to automate the process, probably using the Python API. I've spent some time trying already, but haven't found a way to do it yet. In particular, I tried

• The `arcgis.gis.Item.update()` method on an Item representing the Web Map, trying to replace the JSON data on the Prod map with the data from the Dev map. I got this error: "ValueError: The `update` method requires a user to pass a `fileName` value in the `item_properties` if the file name is not defined on the Item"
• The `arcgis.mapping.WebMap.update_layer()` method. Well, I didn't actual try this, but the documentation explicitly says that changing the `itemId` of a Feature Layer using that method will not work.

Does anyone have any suggestions on how to make this work using the Python API, or even a REST API? Or any other automation method, really.

[EarlMedina]

Can you elaborate on your attempt using the arcgis.gis.Item.update() method ? How were you passing in the JSON? Were you reading from a file?

[fdeters]

@EarlMedina Sure, here's the gist of my attempt:

agol = GIS(username=USER, password=PASS)
qa_map_item = agol.content.get(QA_MAP_ID)
prod_map_item = agol.content.get(PROD_MAP_ID)

qa_map_data = io.StringIO(json.dumps(qa_map_item.get_data()))
prod_map_item.update(item_properties={}, data=qa_map_data)

 And that last line is where I get the `fileName` error

[EarlMedina]

Ah, for map to map this is a simple fix:

import json
from arcgis import GIS

gis = GIS("https://www.arcgis.com", "user", "pass")

map_a_id = "iusdhjfuihjsdiojmg23423432klkklj"
map_b_id = "opo9o0pokszxcvbnmopjkwer87923459"

map_a = gis.content.get(map_a_id)
map_b = gis.content.get(map_b_id)

data = map_a.get_data()
item_properties = {"text": json.dumps(data)}
map_b.update(item_properties)

[fdeters]

@EarlMedina This worked beautifully, thanks! I came up with a solution for the experience builder app too. I'll add an update to this post later explaining my approach.

[fdeters · Accepted solution]

I was able to find a solution! It's very much a work in progress, and I'm a little apprehensive that it will stand the test of time, but here's the outline of my approach:

Note: Proceed with caution - none of this is guaranteed to work. Be sure to make backups before trying.

Another note: This entire process relies on the underlying data sources (feature layers, related tables, etc.) having both a "QA" or "Dev" version and a "Prod" version with settings and schema that are as identical as possible, so the Prod version can be a drop-in replacement for the QA version but with actual live data.

Step 1 - Set up

1. Import all the packages you need, like `arcgis`, `copy`, and `json`.
2. Define settings, such as the IDs and URLs of all the objects you want to work with.
3. Log into AGOL and create a GIS object to work with.

Step 2 - Update Prod map

1. Get web map items and retrieve the QA map configuration

qa_map_item = gis.content.get(QA_MAP_ID)
prod_map_item = gis.content.get(PROD_MAP_ID)

qa_map_data = qa_map_item.get_data()
prod_map_data = copy.deepcopy(qa_map_data)

The second-to-last line gets the JSON/dict that defines the web map, exactly as if you retrieved the "Data" JSON from ArcGIS Assistant. The last line makes a copy of the QA data, which we will update and eventually "paste" on top of the Prod map.

2. Update resources in the `qa_map_data` dictionary to reference their Prod versions. Here's a simplified version of the way I do this:

for layer in prod_map_data['operationalLayers']:
    layer['itemId'] = PROD_LAYER_ID
    layer['url'] = PROD_LAYER_URL

map_item_properties: {"text": json.dumps(prod_map_data)}
prod_map_item.update(map_item_properties)

The last two lines send the new map config to the Prod map item.

Eventually, I will also do this for tables, not just `operationalLayers`. This seems to work pretty well, as long as the Prod and QA versions have the same schema and settings.

Step 3 - Update Prod Experience Builder app

Be warned! My approach for this step seems to work, but uses a protected, undocumented property of the `WebExperience` object from the Python API.

1. Get app items and retrieve the QA app configuration

qa_app_item = agol.content.get(QA_APP_ID)
prod_app_item = agol.content.get(PROD_APP_ID)

qa_app = WebExperience(qa_app_item)
prod_app = WebExperience(prod_app_item)

prod_app_data = copy.deepcopy(qa_app._draft)

This should look pretty similar to Step 2.1 from above - we get the QA and Prod items, then make a copy of the QA configuration and prepare to make changes to it. This time, however, we use an undocumented property: `_draft`.

Best I can tell, this `WebExperience._draft` property seems to mirror the `config/config.json` file you may be familiar with if you've used EB Developer Edition or explored in ArcGIS Assistant, but it only applies to the draft of the Experience. Making changes to this is sort of like editing the experience in the GUI without publishing - changes to it won't show up in the published version of the app. It's a dictionary that has all the properties that control configuration of the Experience. We can update the keys of the dictionary just like we did with the web map earlier.

2. Update resources in the `prod_app_data` dictionary to reference their Prod versions. Again, here's a simplified version of how I do this:

for data_source_id in prod_app_data['dataSources']:
    # get the actual object with info about the data source
    source = prod_app_data['dataSources'][data_source_id]

    source['itemId'] = PROD_DATASOURCE_ID
    source['sourceLabel'] = PROD_DATASOURCE_LABEL
    
    # Delete the auto-generated config for the data 
    # source's child data sources.
    # Best I can tell, this will be auto-regenerated, 
    # and it's not strictly needed for the app to work
    del source['childDataSourceJsons']

# overwrite the app's config with the new data
prod_app._draft = prod_app_data
# save, but don't publish the new config
prod_app.save()

3. Go to Experience Builder and check out the draft preview of the Experience to make sure nothing is majorly broken.

4. If everything looks good, publish it

prod_app.save(publish=True)

Footnote

I welcome any comments on this approach, how it might be improved, questions, grave warnings, etc. There are a lot of things that could potentially go wrong here, but so far it's better than the alternative of manually updating and keeping in sync two separate versions of the web map and EB app.

[James_Whitacre_PGC]

@fdeters really like your overall approach and I am very glad to have stumbled upon this post as I have been thinking about this topic for some time ('._draft' is a lifesaver!!). I sometimes think the need for retaining a Web Mapping Application's URL is not something that is emphasized enough, so your attention to this is admirable. Thank you for you efforts here! Below is my (probably way too long) contribution to the discussion...

I especially appreciated this note from your solution, but I changed it slightly for you 😉...you'll see why!):

Another note: This entire process relies on the underlying data sources (feature layers, related tables, etc.) having both a "QA" or "Dev" version and a "Prod" version with settings and schema that are as exactly identical as possible, so the Prod version can be a drop-in replacement for the QA version but with actual live data.

I have actually independently developed (i.e. before I even read this post!) some code (shared below) to help create a new 'dev environment' for the content items, but just the the Web Experience and all Web Maps sourced (though I haven't actually tested multiple web maps, but I am pretty sure my code accounts for it!). I tried using the .clone_items() method, but unfortunately, it just falls short with (my) Web Experiences. Things would not load properly for my use case...not sure why, but the cloned Web Map caused an error. So my method is more of a 'shallow copy' if you will.

In my code and scenario, for example, layers remain sourced to the original layers of the source Web Map, and this is due to my slightly different way of thinking (or set of assumptions) and due to some data limitations. Those assumptions are, if everything is in a 'production' state, that should mean that the underlying data (i.e., feature services) and schema (i.e., data design...field names, relationships, etc.) will remain static most of the time, but the Web Map and Web Experience will likely get the bulk of the enhancements/design changes.

Of course, data schemas do change and may need updated over time, but my focus isn't on that (yet!!!). This is mainly because for my use case (link to my public production app), the data includes nearly 30 GB of attachments, and I have employed a few View and Join View layers that make everything more wonderful, but also more difficult, at least from an automation standpoint. By not having 'dev' versions of the layers, I can just retain the source layers as they are, and my changes to the Web Map and Web Experience won't affect the data anyway.

Hope that makes sense. I am not saying you (or anyone) shouldn't bother with dev layers, we all definitely should (I have plans to add some new fields to a feature layer eventually)! Just that my use case makes things more difficult, so I am not dealing with them at this time. What would be ideal is to develop a way to copy the schema and append a subset of the data, that way the 'dev' data doesn't take up as much space (and use more AGO credits!!).

So, finally, here is my code! It was developed in an ArcGIS Online Notebook, so it is written for that environment. Also, the URLs printed at the end go directly to the new 'dev' published Web Experience and Web Maps, not the content item URL. Also, I consider this code to still be in development, so any feedback is appreciated and there are no warranties! Thanks!

from arcgis.gis import GIS
from arcgis.apps import expbuilder
gis = GIS("home")

''' Parameters '''
# User should edit these variables

# Web Experience Content Item ID to be copied
web_exp_id = WEB_EXPERIENCE_CONTENT_ITEM_ID

# Suffix for New Items Name (i.e. Title). A space + the sufffix will be added to the end of the original name.
suffix = 'Dev'

# Folder (None or '' if same folder)
fldr = 'Project Name Dev'

''' Script '''

# Get Web Experience as Web Experience object
web_exp_app = expbuilder.WebExperience(web_exp_id)

# Make a copy of the Web Experience
web_exp_copy = web_exp_app.save(title=f'{web_exp_app.item.title} {suffix}', duplicate=True, include_private=True)

# Move the Web Experience to the folder if one is given
if fldr:
    web_exp_copy.item.move(fldr)

# list of Item URLs to check after the code is ran. New Web Experience is alreay populated
copy_items_urls = [web_exp_copy.item.url]

# Get the source Web Experience data sources
web_exp_srcs = web_exp_app.datasources

# For each Web Map, copy, move to folder, and replace in new Web Expereience
for src in web_exp_srcs:
    if web_exp_srcs[src]['type'] =='WEB_MAP':
        
        # Get Web Map Item
        web_map = gis.content.get(web_exp_srcs[src]['itemId'])
        
        # Copy Web Map Item
        web_map_copy = web_map.copy(title=f'{web_map.title} {suffix}')
        
        # Move to folder if one is given
        if fldr:
            web_map_copy.move(fldr)

        # Replace data source elements with copied Web Map
        web_exp_copy.datasources[src]['itemId'] = web_map_copy.id
        web_exp_copy.datasources[src]['sourceLabel'] = web_map_copy.title
        
        copy_items_urls += [f'https://pagame.maps.arcgis.com/apps/mapviewer/index.html?webmap={web_map_copy.id}']

# Save the copy of the Web Experience
web_exp_copy.save(publish=True)

print(copy_items_urls)

[fdeters]

@James_Whitacre_PGC Thanks for sharing your code! That's super handy for creating the items themselves. I was also running into issues using `.clone_items()` and ended up just using ArcGIS Assistant for copying instead.

I appreciate your edit of my comment above—indeed, the schema and settings being exactly the same will cut risk down by a lot. In my particular case, the feature layers in question are highly sensitive, editable internal datasets, so we want to be able to test editing those feature layers in a QA/dev environment before opening the door to interacting with production data. So in my case, being able to swap out versions of layers is crucial.

Your approach above seems targeted toward setting up the initial Dev environment. I'm curious, what's your process right now for making sure changes made in Dev are copied over exactly to the production items?

Thanks again for contributing to the discussion 🙂

--

For other future readers, I also want to point out that my approach (in the Solution) has grown significantly more complex over time to account for related tables and Experience "resource" files (like images and icons). I've found that ArcGIS Assistant is an extremely valuable tool for exploring AGOL items' JSON configuration files and using those to discover how my script needs to modify QA/dev object data before pasting it onto the Production versions. 

Also, use a staging version. Create a throwaway "test copy" of each item that you can test the script on before applying the results to a real production item. Otherwise, you risk the script ruining your production items if it makes a mistake.

[James_Whitacre_PGC]

@fdeters thanks for replying! I have used ArcGIS Assistant also, but I prefer automated workflows whenever possible. And we have similar editing workflows as you describe. And in my use case, it contains editable data as well, it is just stable at the moment, so I don't need the layers to be in a separate dev service. Eventually, I expect to incorporate that. Just not needed yet!

My next steps are to work on the process of pushing the edits to the production. I haven't started developing that code yet, which is why it was great to come by your post! I will likely adapt to what you have posted. I will be sure to share an update when that comes. I hope that will be soon, but the project has a deadline, and if I can't get the code to work for me, I may need to do it manually. I'd love to learn more about how you have dealt with 'Experience "resource" files,' as this will be part of my use case as well.

---

As an aside, and if any Esri staff read this, I think that this idea that you (@fdeters) have brought up is an important workflow and I would love to see some added tools in the ArcGIS API for Python. Being able to push updates so we can keep consistent URLs/Item IDs is very important for end users. Another way to help implement this is to enable Custom short URLS for applications that can also be updated with a new URL when an updated application needs to be swapped. With this method, all this copying/cloning/staging/pushing may not be as necessary.

Also, the '.clone_items()' method isn't cutting it with Web Experiences. I think more testing needs to be done to ensure that applications don't break.

I would love to see other add to this conversation!!

[fdeters]

@James_Whitacre_PGC I'd be happy to share what I have for updating the resource files; here's the code. Note that it references two variables `qa_app_item` and `prod_app_item` from an outer scope. Those are just the Item objects for the Dev/QA Experience and the Prod Experience from the ArcGIS API for Python (as you may have guessed).

def update_resources(resource_type: str):
    """Copies new icons or images to Prod from QA"""
    # validate the resource_type
    valid_types = ['icon', 'image']
    if resource_type not in valid_types:
        raise ValueError(f"Invalid resource type: {resource_type}")
    
    # get the lists of resources for qa and prod
    resource_list_filepath = f"images/{resource_type}-resources-list.json"
    qa_resource_list = qa_app_item.resources.get(resource_list_filepath)
    prod_resource_list = prod_app_item.resources.get(resource_list_filepath)
    
    # copy individual files from qa to prod
    for resource in qa_resource_list:
        if resource not in prod_resource_list:
            # copy it over to prod
            properties = qa_resource_list[resource]['properties']
            
            path = properties['path'].replace('${appResourceUrl}/', '')
            folder = path[:path.rfind('/')]
            name = properties['originalName']

            file = qa_app_item.resources.get(path)
            try:
                prod_app_item.resources.update(file=file, folder_name=folder)
            except:
                prod_app_item.resources.add(file=file, folder_name=folder)

            print(f">> Copied new {resource_type} '{name}'")
    
    # copy list file from qa to prod
    qa_resource_list_file = qa_app_item.resources.get(
        file=resource_list_filepath,
        try_json=False
    )
    prod_app_item.resources.update(
        file=qa_resource_list_file,
        folder_name='images'
    )
    print(f">> Updated {resource_type} resources list")


print('Checking icons for updates...')
update_resources('icon')
print('Checking images for updates...')
update_resources('image')
print('Finished updating resources.')

It doesn't handle deleting unused resources from the Prod app, but that seemed like more trouble than it was worth for my purposes.