Automating Release Process for Experience Builder App

[fdeters]

How can I automate the Dev>Production promotion process for a map-centric Experience Builder app where the Production version uses different data from the Dev version?

---

Using Python API 2.3.0, AGOL Map Viewer, and AGOL Experience Builder (not Developer Edition)

---

I'm building an Experience Builder app that will be used to view and edit a high-value dataset for my organization. We have an ecosystem in AGOL that includes both Dev and Prod versions of the data. There are multiple views and related tables of the dataset, so there are several items used in the app that have both Dev and Prod versions in our organization.

My strategy so far is to have two versions (Dev and Prod) of both the EB app and the Web Map that it consumes. I make updates in the Dev map and app, then use ArcGIS Assistant to

1. Literally copy the JSON from the Dev versions to the Prod versions ("Data" for the web map; and "Data", "config/config.json", and any other files such as images for the EB app)
2. Update the `url` and `itemId` of the data resources (such as operationalLayers or tables) to the Prod values in the Prod version of the app.

This ArcGIS Assistant process has worked surprisingly well. I especially like that it doesn't entail creating a new Production item every time (I don't want the EB app URL to change).

However, as the app will soon have more environment-specific data sources, I'm looking for a way to automate the process, probably using the Python API. I've spent some time trying already, but haven't found a way to do it yet. In particular, I tried

• The `arcgis.gis.Item.update()` method on an Item representing the Web Map, trying to replace the JSON data on the Prod map with the data from the Dev map. I got this error: "ValueError: The `update` method requires a user to pass a `fileName` value in the `item_properties` if the file name is not defined on the Item"
• The `arcgis.mapping.WebMap.update_layer()` method. Well, I didn't actual try this, but the documentation explicitly says that changing the `itemId` of a Feature Layer using that method will not work.

Does anyone have any suggestions on how to make this work using the Python API, or even a REST API? Or any other automation method, really.

[EarlMedina]

Can you elaborate on your attempt using the arcgis.gis.Item.update() method ? How were you passing in the JSON? Were you reading from a file?

[fdeters]

@EarlMedina Sure, here's the gist of my attempt:

agol = GIS(username=USER, password=PASS)
qa_map_item = agol.content.get(QA_MAP_ID)
prod_map_item = agol.content.get(PROD_MAP_ID)

qa_map_data = io.StringIO(json.dumps(qa_map_item.get_data()))
prod_map_item.update(item_properties={}, data=qa_map_data)

 And that last line is where I get the `fileName` error

[EarlMedina]

Ah, for map to map this is a simple fix:

import json
from arcgis import GIS

gis = GIS("https://www.arcgis.com", "user", "pass")

map_a_id = "iusdhjfuihjsdiojmg23423432klkklj"
map_b_id = "opo9o0pokszxcvbnmopjkwer87923459"

map_a = gis.content.get(map_a_id)
map_b = gis.content.get(map_b_id)

data = map_a.get_data()
item_properties = {"text": json.dumps(data)}
map_b.update(item_properties)

[fdeters]

@EarlMedina This worked beautifully, thanks! I came up with a solution for the experience builder app too. I'll add an update to this post later explaining my approach.

[fdeters · Accepted solution]

I was able to find a solution! It's very much a work in progress, and I'm a little apprehensive that it will stand the test of time, but here's the outline of my approach:

Note: Proceed with caution - none of this is guaranteed to work. Be sure to make backups before trying.

Another note: This entire process relies on the underlying data sources (feature layers, related tables, etc.) having both a "QA" or "Dev" version and a "Prod" version with settings and schema that are as identical as possible, so the Prod version can be a drop-in replacement for the QA version but with actual live data.

Step 1 - Set up

1. Import all the packages you need, like `arcgis`, `copy`, and `json`.
2. Define settings, such as the IDs and URLs of all the objects you want to work with.
3. Log into AGOL and create a GIS object to work with.

Step 2 - Update Prod map

1. Get web map items and retrieve the QA map configuration

qa_map_item = gis.content.get(QA_MAP_ID)
prod_map_item = gis.content.get(PROD_MAP_ID)

qa_map_data = qa_map_item.get_data()
prod_map_data = copy.deepcopy(qa_map_data)

The second-to-last line gets the JSON/dict that defines the web map, exactly as if you retrieved the "Data" JSON from ArcGIS Assistant. The last line makes a copy of the QA data, which we will update and eventually "paste" on top of the Prod map.

2. Update resources in the `qa_map_data` dictionary to reference their Prod versions. Here's a simplified version of the way I do this:

for layer in prod_map_data['operationalLayers']:
    layer['itemId'] = PROD_LAYER_ID
    layer['url'] = PROD_LAYER_URL

map_item_properties: {"text": json.dumps(prod_map_data)}
prod_map_item.update(map_item_properties)

The last two lines send the new map config to the Prod map item.

Eventually, I will also do this for tables, not just `operationalLayers`. This seems to work pretty well, as long as the Prod and QA versions have the same schema and settings.

Step 3 - Update Prod Experience Builder app

Be warned! My approach for this step seems to work, but uses a protected, undocumented property of the `WebExperience` object from the Python API.

1. Get app items and retrieve the QA app configuration

qa_app_item = agol.content.get(QA_APP_ID)
prod_app_item = agol.content.get(PROD_APP_ID)

qa_app = WebExperience(qa_app_item)
prod_app = WebExperience(prod_app_item)

prod_app_data = copy.deepcopy(qa_app._draft)

This should look pretty similar to Step 2.1 from above - we get the QA and Prod items, then make a copy of the QA configuration and prepare to make changes to it. This time, however, we use an undocumented property: `_draft`.

Best I can tell, this `WebExperience._draft` property seems to mirror the `config/config.json` file you may be familiar with if you've used EB Developer Edition or explored in ArcGIS Assistant, but it only applies to the draft of the Experience. Making changes to this is sort of like editing the experience in the GUI without publishing - changes to it won't show up in the published version of the app. It's a dictionary that has all the properties that control configuration of the Experience. We can update the keys of the dictionary just like we did with the web map earlier.

2. Update resources in the `prod_app_data` dictionary to reference their Prod versions. Again, here's a simplified version of how I do this:

for data_source_id in prod_app_data['dataSources']:
    # get the actual object with info about the data source
    source = prod_app_data['dataSources'][data_source_id]

    source['itemId'] = PROD_DATASOURCE_ID
    source['sourceLabel'] = PROD_DATASOURCE_LABEL
    
    # Delete the auto-generated config for the data 
    # source's child data sources.
    # Best I can tell, this will be auto-regenerated, 
    # and it's not strictly needed for the app to work
    del source['childDataSourceJsons']

# overwrite the app's config with the new data
prod_app._draft = prod_app_data
# save, but don't publish the new config
prod_app.save()

3. Go to Experience Builder and check out the draft preview of the Experience to make sure nothing is majorly broken.

4. If everything looks good, publish it

prod_app.save(publish=True)

Footnote

I welcome any comments on this approach, how it might be improved, questions, grave warnings, etc. There are a lot of things that could potentially go wrong here, but so far it's better than the alternative of manually updating and keeping in sync two separate versions of the web map and EB app.