Migrate Content for ArcGIS Enterprise/ArcGIS Online

[#1]

It's always a good practice to have a DEV/STAGING environment along with a PRODUCTION environment for ArcGIS Enterprise. I frequently get asked how to migrate content from one to the other.  Hosted services can be straight forward, and can easily be scripted (expect an update later to this toolset for this functionality), but referenced services can be rather difficult. For example, there are numerous prerequisites that are required for a service to be published from an Enterprise Geodatabase, such as:

• does the ArcGIS Service Account have necessary privileges
• is the geodatabase registered with the ArcGIS Server instance
• does the feature class in fact exist within the new geodatabase
• etc

These tools pick up once your services have been published to the other environment and you would like to begin migrating web maps, apps, and dashboards. Take a look at the video below on how to execute these tools.

These tools should only be used to migrate web maps/apps where the environments are the same version (i.e. 11.1).  Migrating content between two ArcGIS Online organizations is also supported.

Currently, these tools have only been tested in ArcGIS Enterprise 11.1 and ArcGIS Online. If there are any issues, please report in the comment section below.

 

Updates

1/16/24:  Added tools to copy hosted feature services and file based items.

1/29/24:  Added option to run when portal has Windows Authentication enabled.  Also, updated the Copy Dashboard tool to specify stand-alone layers.

2/12/24:  Web Map item IDs now update

3/28/24:  Added tool to copy Story Maps

6/30/24:  Group layers and tables are now supported when copying web maps

7/10/24: Experience Builder Templates are now supported

Known Issues

• If hosted feature services exist in Enterprise, they will not successfully overwrite when executing the Copy Hosted Feature Services tool.  Will successfully overwrite in AGOL.  Possible bug with ArcGIS API for Python
• After Story Map is copied to Enterprise, the Story Map must first be opened in Edit mode and published before it can be successfully viewed.  This is not the case for AGOL.  Possible bug with ArcGIS API for Python

[DarinaTchountcheva]

Hello Jake,

Thank you so much for this solution! You are a life saver! 

I was just going to tackle the task of deploying dashboards from dev to prod and your post popped up. It saved me so much effort and time. Thank you!

Here are my 2 cents:

I have tried the Copy Web Map and Copy Dashboard tools. 

We use Integrated Windows Authentication and an error popped up while filling out the parameters that connection to the portal cannot be made. 

I changed the validation code to not provide user and password as arguments, as they are assumed from the user currently logged in and it worked: 

source = GIS(self.params[0].value)

target = GIS(self.params[6].value)

I made the same changes in all .py scripts:

source = GIS(sourcePortal)

target = GIS(targetPortal)

And I was able to copy a web map and a dashboard that has a map widget. 

A lot of our dashboards don't have a map widget, and the tool cannot be used, but I was able to use your script to get the JSON of the original dashboard, change the item IDs of the source to the target ones and then use the second part of the script to copy the dashboard to the target portal. And it worked.

Thank you again for your priceless contribution!

Darina

[JakeSkinner]

@DarinaTchountcheva thank you for this valuable feedback.  I've updated the tools to include options for when Windows Authentication is enabled for Portal, and also updated the Copy Dashboard tool to include updating the item IDs of stand-alone layers.

[DarinaTchountcheva]

@Jake you are truly amazing! 

Thank you so much for taking the time to make the modifications. I and a lot of other users will benefit greatly from it and will be more efficient. 

I will let you know how it worked as soon as I get a chance to test it. 

Best regards,

Darina

[Eliot_Reid]

@JakeSkinner 

I'm trying to use these tools with an IWA integrated portal and an AGOL. I'm getting the following error when I use my portal in the Source URL and check 'Source Portal Windows Authentication Enabled'. I've tried with my Pro signed into my portal and out of my portal, as well as with different Python environments. My user does have admin rights to the Portal. Do you have any thoughts? Thanks!

Error 032659 updateParameters Error: Traceback (most recent call last):
  File "Q:\admin\Template\expressions\ESRI_ScriptsToolboxs\Migrate Content\Migrate Content.atbx\CopyWebMap.tool\tool.script.validate.py", line 203, in <module>
  File "Q:\admin\Template\expressions\ESRI_ScriptsToolboxs\Migrate Content\Migrate Content.atbx\CopyWebMap.tool\tool.script.validate.py", line 109, in updateParameters
  File "Q:\admin\Template\expressions\ESRI_ScriptsToolboxs\Migrate Content\Migrate Content.atbx\CopyWebMap.tool\tool.script.validate.py", line 40, in usersDropDown
  File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\__init__.py", line 4943, in search
    users = self._portal.get_org_users(
  File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 1244, in get_org_users
    resp = self._org_users_page(
  File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 2922, in _org_users_page
    return self.con.post("portals/self/users", postdata)
  File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1524, in post
    return self._handle_response(
  File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1000, in _handle_response
    self._handle_json_error(data["error"], errorcode)
  File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1023, in _handle_json_error
    raise Exception(errormessage)
Exception: You do not have permissions to access this resource or perform this operation.
(Error Code: 403)

[JakeSkinner]

@Eliot_Reid 

Open the python console, or new Notebook in Pro by going to Analysis tab > Python dropdown:

Enter the following code, replacing https://gis.esri.com/portal with your URL to portal:

from arcgis.gis import GIS
gis = GIS('https://gis.esri.com/portal')
source_users = gis.users.search(max_users=10000)
source_users

 Ex:

Does this execute successfully?

[Eliot_Reid]

It does not, same error. 

I just double checked and our Portal security actually says SAML is set up, not IWA. I assumed they were the same thing. I'm not sure if they work the same.

[JakeSkinner]

@Eliot_Reid I had a feeling this may be the case.  SAML is not supported.  If you can, create a built-in Administrator account and use that instead.

[anonymous55]

@JakeSkinner 
I am getting same error as @Eliot_Reid could you help me how can I fix this error? How can I create  a built-in Administrator account? I am admin in our portal org

Best,

[JakeSkinner]

@anonymous55 go to Organization > Members > Add Members:

There you will have an option to create a Built-in account:

[MichaelWestbrook]

@JakeSkinner  I'm trying to use these tools but when I run it I get this error :

Any ideas?

[JakeSkinner]

@MichaelWestbrook right-click on the tool you are trying to execute and click on the Execution tab on the left.  It will list the path to the python script.  Make sure it exists in the path referenced here.  Ex:

[MichaelWestbrook]

@JakeSkinner 

Thank you. I had to create all of the folders in the the and copy the data there. It works perfectly now and does exactly what I need it to do. Are you going to make it so you can copy EXB templates as well? 

[JakeSkinner]

@MichaelWestbrook EXB templates are now supported.  You will need to re-download the tool.

[MichaelWestbrook]

@JakeSkinner Thank you

[CodyPatterson]

Hey @JakeSkinner 

Doing a staging to production migration now and just found your tool, seemingly I just found the blog sections of Community! Great tools you have here and great work you've done, thank you for providing!

Cody

[JakeSkinner]

Thank you @CodyPatterson, I appreciate the feedback.

[DavidGisvision]

Hi Jake

Great work.  I have sendt you a suggestion for a updated version of the "hosted feature service" script on mail.

Best regards

David

[MichaelWestbrook]

@JakeSkinner  Good morning, I am trying to copy web maps and as soon as I select the web, PortalUrl gets an error

[JakeSkinner]

@MichaelWestbrook I found the issue, the validation code checks the web map for tables, if none exists, it produces this error.  Redownload the tools, and it should work successfully now.

[MichaelWestbrook]

@JakeSkinner  Works perfect now! Thank you!

[MichaelWestbrook]

@JakeSkinner I copied the web maps over successfully. When I copied the Experience Builder it ran successfully but out of the 7 web maps in it, it only mapped one successfully. 

[JakeSkinner]

@MichaelWestbrook I've seen sometimes if you don't tab out of each Web Map ID box, the GP tool won't pick up the IDs correctly.  Try tabbing out of the New Web Map ID and re-run the tool again to see if it works.

[JakeHanson-eaglenz]

Hi @JakeSkinner have you considered adding this to Github? Potentially as a Python Toolbox to enable easier contribution to the validation rules?

Keen to provide some contributions to this useful resource.

[jdcayetano]

The transfer worked but I was wondering if this error meant something that I need to be wary of:

[CommunityMapsEsriColombia]

Hi @JakeSkinner , I'm trying to migrate a dashboard from my ArcGIS Online account to an ArcGIS Portal 11.1 account, apparently the tool works fine but when I check the item it is empty. I even checked the .json through assitant and it has the dashboard structure copied. Do you know how I could solve this problem?

PD: I am the administrator of both accounts.

[JakeSkinner]

@jdcayetano no, this should not be a concern.  The script did not delete a temporary zip file mentioned in the path above.

[JakeSkinner]

@JakeHanson-eaglenz once I learn GitHub, I'll post it there.

[JakeSkinner]

@CommunityMapsEsriColombia migrating from ArcGIS Online to Enterprise will not be supported.  ArcGIS Online is always at a later release than Enterprise, so applications will fail to open.  What should work is migrating Web Maps and Hosted Feature services from AGOL to Enterprise, but I have not extensively tested this.  Ideally, you should only be migrating content between the same environments (i.e. AGOL --> AGOL, Enterprise 11.1 --> Enterprise 11.1).

[RejeanLabbe]

@JakeSkinner Thanks for your great work.

I need to copy non hosted Feature services but it does not seem to be possible.

I tried many things without success like "content.add" and "content.clone_items".

There must be a way to copy a non hosted Feature service. Do you have any clue?

[JakeSkinner]

@RejeanLabbe copying referenced services is very difficult, especially if the data source will change.  Due to the many possible parameters when publishing referenced services, this is not easily scripted.

[RejeanLabbe]

@JakeSkinner Thanks for your answer. If anything changes about this in the future, I would be glad to ear about that.

[morglarson]

@JakeSkinner Do you have any plans to create a tool to copy ArcGIS Hub sites? Thank you!

[stevetaylor_perth]

Thanks so much for these tools Jake,
Just a quick question about ArcGIS Enterprise Versions being the same.
We have 10.9.1 and will be migrating to ArcGIS Enterprise 11.3
We will manually publish our registered map services to the new environment.
But we would like to use your tool to migrate the Web Maps - will the version discrepancy be an issue?

We are not worried about the apps or dashboards as we are currently using WAB app which will be rebuilt in Experience Builder apps.

[JakeSkinner]

Hi @stevetaylor_perth,

No, the version discrepancy should not be an issue for web maps.

[stevetaylor_perth]

Thanks for quick reply Jake.

[VigneshGopalakrishnan]

string indices must be integers, not 'str'

File "D:\Git\migrate_content__final\MigrationScript.py", line 334, in _migrate_exp_app for val in resourceJSON['dataSources']:

I'm getting this error with experience builder apps

[VigneshGopalakrishnan]

@JakeSkinner  Thank you for this post.

I have been using this to migrate AGE 10.8 to 11.3 but ran into issues with Experience builder Apps. I tried executing the script independently and also in ArcGIS pro but encountering this error in both the places. Hope you can help.

The issue I'm facing is for the copy experience builder app file 

string indices must be integers, not 'str'

for val in resourceJSON['dataSources']:

line 148 in the validation file 

Furthemore the zip is not downloadable anymore. looks like it is disabled

[JakeSkinner]

@VigneshGopalakrishnan the tools may not work for migrating Experience Builder applications between different environments.  This should work for hosted feature services, and web maps, but there could be significant changes in applications that are not supported between different versions.  

Make sure you click the arrow to download the zip file:

[VigneshGopalakrishnan]

Thank you for that @JakeSkinner .

But just wanted to bring into your attention that the point of failure is even before that here.

 

[JakeSkinner]

@VigneshGopalakrishnan it looks like Enterprise 10.8.1 handles the config.json slightly different.  To fix the error you're receiving, perform the following steps:

1.  Right-click on the Copy Experience Builder App script tool > Properties

2.  Select the Validation tab

3.  Delete everything and paste in the below:

import arcpy, json
from arcgis.gis import GIS
from arcgis.gis import User


class ToolValidator:
  # Class to add custom behavior and properties to the tool and tool parameters.

    def __init__(self):
        # set self.params for use in other function
        self.params = arcpy.GetParameterInfo()

    def initializeParameters(self):
        # Customize parameter properties. 
        # This gets called when the tool is opened.
        return


    def disableIWA(self, paramIndex1, paramIndex2):
        '''Disable IWA option if connecting to AGOL'''

        if 'arcgis.com' in self.params[paramIndex1].value:
            self.params[paramIndex2].enabled = False
        else:
            self.params[paramIndex2].enabled = True


    def disableUsernamePassword(self, paramIndex1, paramIndex2, paramIndex3):
        '''Disable username/password if using Windows Authentication'''

        if self.params[paramIndex1].value == True:
            self.params[paramIndex2].enabled = False
            self.params[paramIndex3].enabled = False
        if self.params[paramIndex1].value == False:
            self.params[paramIndex2].enabled = True
            self.params[paramIndex3].enabled = True


    def usersDropDown(self, source, paramIndex1, paramIndex2):
        '''Populate User Dropdown'''

        sourceUserList = []
        source_users = source.users.search(max_users=100000)
        for user in source_users:
            if user.user_types()['name'] not in ('Viewer', 'Editor', 'Field Worker'):
                sourceUserList.append(user.username)

        sourceUserFilter = self.params[paramIndex2].filter
        sourceUserFilter.list = sourceUserList


    def getUserContent(self, source, paramIndex1, paramIndex2):
        '''Get User Content'''

        contentList = []
        username = str(self.params[paramIndex1].value)
        username = username.replace("'", '')
        user = User(source, username)
        user_content = user.items(max_items=100000)
        for item in user_content:
            if item.type == 'Web Experience':
                contentList.append(
                    str(item.title) + ' - ' + str(item.type) + ' - ' + str(item.id))
        folders = user.folders
        for folder in folders:
            user_content = user.items(folder=folder['title'], max_items=100000)
            for item in user_content:
                if item.type == 'Web Experience':
                    contentList.append(
                        str(item.title) + ' - ' + str(item.type) + ' - ' + str(item.id))
        contentList.sort()
        contentFilter = self.params[paramIndex2].filter
        contentFilter.list = contentList


    def getTargetUserFolder(self, target, paramIndex1, paramIndex2):
        '''Get Target User Folder'''

        targetUserFolderList = ['ROOT']
        user = User(target, self.params[paramIndex1].value)
        for folder in user.folders:
            targetUserFolderList.append(folder['title'])

        targetUserFolderListFilter = self.params[paramIndex2].filter
        targetUserFolderListFilter.list = targetUserFolderList


    def updateParameters(self):
        # Modify parameter values and properties.
        # This gets called each time a parameter is modified, before
        # standard validation.

        # Remove Windows Authentication option for source if arcgis.com in URL
        if not self.params[0].hasBeenValidated:
            self.disableIWA(0, 1)

        # Disable username/password for source if Windows Authentication option is True
        for i in range(0, 2):
            if not self.params[i].hasBeenValidated:
                self.disableUsernamePassword(1, 2, 3)

        # Connect to portal using username/password to populate dropdown for source users
        if self.params[1].value == False:
            for i in (0, 2, 3):
                if not self.params[i].hasBeenValidated:
                    if self.params[0].value and self.params[2].value and self.params[3].value:
                        source = GIS(self.params[0].value, self.params[2].value, self.params[3].value,
                                     verify_cert=False)
                        self.usersDropDown(source, 0, 4)

        # Connect to portal using Windows Authentication to populate dropdown for source users
        elif self.params[1].value == True:
            if not self.params[1].hasBeenValidated:
                if self.params[0].value:
                    source = GIS(self.params[0].value, verify_cert=False)
                    self.usersDropDown(source, 0, 4)

        # Dropdown for source user's content
        if self.params[1].value == False:
            for i in (0, 2, 3, 4):
                if not self.params[i].hasBeenValidated:
                    if self.params[0].value and self.params[4].value:
                        source = GIS(self.params[0].value, self.params[2].value, self.params[3].value,
                                     verify_cert=False)
                        self.getUserContent(source, 4, 5)
        elif self.params[1].value == True:
            for i in (0, 1, 4):
                if not self.params[i].hasBeenValidated:
                    if self.params[0].value and self.params[4].value:
                        source = GIS(self.params[0].value, verify_cert=False)
                        self.getUserContent(source, 4, 5)

        # Get Web Map IDs in Experience Builder App
        webMapVtab = self.params[6]
        embedVtab = self.params[7]
        if self.params[5].altered and not self.params[5].hasBeenValidated:
            if self.params[1].value == False:
                source = GIS(self.params[0].value, self.params[2].value, self.params[3].value, verify_cert=False)
            elif self.params[1].value == True:
                source = GIS(self.params[0].value, verify_cert=False)
            webMapIdList = []
            embededURLList = []
            expAppID = self.params[5].value.split(' - ')[-1]
            expApp = source.content.get(expAppID)
            expAppJSON = expApp.get_data(try_json=True)
            jsonFile = expApp.resources.get(file='config/config.json', try_json=True)
            with open(jsonFile, 'r') as file:
                resourceJSON = json.load(file)
            # Create list to store data sources from Config.JSON file
            for val in resourceJSON['dataSources']:
                webmaps = []
                if resourceJSON['dataSources'][val]['type'] == 'WEB_MAP':
                   webMapID = (resourceJSON['dataSources'][val]['itemId'])
                   webmaps.append(webMapID)
                   webmaps.append('') 
                if len(webmaps) > 0:
                    webMapIdList.append(webmaps)
                
            for widget in expAppJSON['widgets']:
                embededURLs = []
                try:
                    embedURL = expAppJSON['widgets'][widget]['config']['expression']
                    embedURL = embedURL.split('>')[1]
                    embedURL = embedURL.split('<')[0]
                    embededURLs.append(embedURL)
                    embededURLs.append('')
                except:
                    pass
                if len(embededURLs) > 0:
                    embededURLList.append(embededURLs)
            
            for val in resourceJSON['dataSources']:
                otherDataSources = []
                try:
                    serviceURL = resourceJSON['dataSources'][val]['url']
                    if serviceURL not in otherDataSources:
                        otherDataSources.append(serviceURL)
                        otherDataSources.append('')
                except:
                    pass
                if len(otherDataSources) > 0:
                    embededURLList.append(otherDataSources)
                
            # Remove empty lists
            for webmap in webMapIdList:
                if len(webmap) == 0:
                    webMapIdList.remove(webmap)
            for embed in embededURLList:
                if len(embed) == 0:
                    embededURLList.remove(embed)
            
            # Remove duplicates from list                        
            embededURLList = [i for n, i in enumerate(embededURLList) if i not in embededURLList[:n]]
            
            webMapVtab.values = webMapIdList
            embedVtab.values = embededURLList

        # Remove Windows Authentication option for target if arcgis.com in URL
        if not self.params[8].hasBeenValidated:
            self.disableIWA(8, 9)

        # Disable username/password for target if Windows Authentication option is True
        for i in range(8, 10):
            if not self.params[i].hasBeenValidated:
                self.disableUsernamePassword(9, 10, 11)

        # Connect to portal using username/password to populate dropdown for target users
        if self.params[9].value == False:
            for i in (8, 10, 11):
                if not self.params[i].hasBeenValidated:
                    if self.params[8].value and self.params[10].value and self.params[11].value:
                        target = GIS(self.params[8].value, self.params[10].value, self.params[11].value,
                                     verify_cert=False)
                        self.usersDropDown(target, 8, 12)

        # Connect to portal using Windows Authentication to populate dropdown for target users
        elif self.params[9].value == True:
            if not self.params[9].hasBeenValidated:
                if self.params[8].value:
                    target = GIS(self.params[8].value, verify_cert=False)
                    self.usersDropDown(target, 8, 12)

        # Connect to portal using username/password to populate dropdown for Target User's folder
        if self.params[9].value == False:
            for i in (8, 10, 11, 12):
                if not self.params[i].hasBeenValidated:
                    if self.params[12].value:
                        target = GIS(self.params[8].value, self.params[10].value, self.params[11].value,
                                     verify_cert=False)
                        self.getTargetUserFolder(target, 12, 13)

        # Connect to portal using Windows Authentication to populate dropdown for Target User's folder
        elif self.params[9].value == True:
            for i in (8, 9, 12):
                if not self.params[i].hasBeenValidated:
                    if self.params[12].value:
                        target = GIS(self.params[8].value, verify_cert=False)
                        self.getTargetUserFolder(target, 12, 13)

        return

    def updateMessages(self):
        # Customize messages for the parameters.
        # This gets called after standard validation.
        return

    # def isLicensed(self):
    #     # set tool isLicensed.
    # return True

    # def postExecute(self):
    #     # This method takes place after outputs are processed and
    #     # added to the display.
    # return

[VigneshGopalakrishnan]

Thank you @JakeSkinner 

This one has the same issue. I believe the issue here could be that the Exp Builder that I'm trying to migrate is unpublished

[NickMiller2]

Hi @JakeSkinner 

I am trying to use your GP tool to copy a hosted feature layer from one 11.3 Portal to another.

I am getting this error using latest version of PRO.

Traceback (most recent call last):
File "C:\Testing\Migrate Content\Copy Hosted Services.py", line 57, in <module>
result_item = item.export(export_name, 'File Geodatabase', wait=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\nicholas.miller\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\__init__.py", line 14095, in export
raise Exception("Could not export item: %s" % self.itemid)
Exception: Could not export item: 7e34b118159246ec95be1a2b97350c08

[JakeSkinner]

@NickMiller2 if you search for item 7e34b118159246ec95be1a2b97350c08 in Portal, are you able to successfully export this hosted feature service to a File Geodatabase?

[Teo]

Hello everyone,

is there a way to migrate map image services?

@JakeSkinner 

Thanks in advance.

[JakeSkinner]

@Teo unfortunately, not.  This would be a referenced service.  See the notes above:

but referenced services can be rather difficult. For example, there are numerous prerequisites that are required for a service to be published from an Enterprise Geodatabase, such as:

• does the ArcGIS Service Account have necessary privileges

• is the geodatabase registered with the ArcGIS Server instance

• does the feature class in fact exist within the new geodatabase

• etc

[NickXu]

Hi,

I'm wandering if you could simplify the process just with a copy function to 

1. down item from source 

2. down item/data from source 

3. download thumbnail, metadata from source 

4. create an item in target using item, item/data thumbnail and metadata

5. download resources from source and copy to target