It's always a good practice to have a DEV/STAGING environment along with a PRODUCTION environment for ArcGIS Enterprise. I frequently get asked how to migrate content from one to the other. Hosted services can be straight forward, and can easily be scripted (expect an update later to this toolset for this functionality), but referenced services can be rather difficult. For example, there are numerous prerequisites that are required for a service to be published from an Enterprise Geodatabase, such as:
These tools pick up once your services have been published to the other environment and you would like to begin migrating web maps, apps, and dashboards. Take a look at the video below on how to execute these tools.
These tools should only be used to migrate web maps/apps where the environments are the same version (i.e. 11.1). Migrating content between two ArcGIS Online organizations is also supported.
Currently, these tools have only been tested in ArcGIS Enterprise 11.1 and ArcGIS Online. If there are any issues, please report in the comment section below.
1/16/24: Added tools to copy hosted feature services and file based items.
1/29/24: Added option to run when portal has Windows Authentication enabled. Also, updated the Copy Dashboard tool to specify stand-alone layers.
2/12/24: Web Map item IDs now update
3/28/24: Added tool to copy Story Maps
6/30/24: Group layers and tables are now supported when copying web maps
7/10/24: Experience Builder Templates are now supported
Hello Jake,
Thank you so much for this solution! You are a life saver!
I was just going to tackle the task of deploying dashboards from dev to prod and your post popped up. It saved me so much effort and time. Thank you!
Here are my 2 cents:
I have tried the Copy Web Map and Copy Dashboard tools.
We use Integrated Windows Authentication and an error popped up while filling out the parameters that connection to the portal cannot be made.
I changed the validation code to not provide user and password as arguments, as they are assumed from the user currently logged in and it worked:
source = GIS(self.params[0].value)
target = GIS(self.params[6].value)
I made the same changes in all .py scripts:
source = GIS(sourcePortal)
target = GIS(targetPortal)
And I was able to copy a web map and a dashboard that has a map widget.
A lot of our dashboards don't have a map widget, and the tool cannot be used, but I was able to use your script to get the JSON of the original dashboard, change the item IDs of the source to the target ones and then use the second part of the script to copy the dashboard to the target portal. And it worked.
Thank you again for your priceless contribution!
Darina
@DarinaTchountcheva thank you for this valuable feedback. I've updated the tools to include options for when Windows Authentication is enabled for Portal, and also updated the Copy Dashboard tool to include updating the item IDs of stand-alone layers.
@Jake you are truly amazing!
Thank you so much for taking the time to make the modifications. I and a lot of other users will benefit greatly from it and will be more efficient.
I will let you know how it worked as soon as I get a chance to test it.
Best regards,
Darina
I'm trying to use these tools with an IWA integrated portal and an AGOL. I'm getting the following error when I use my portal in the Source URL and check 'Source Portal Windows Authentication Enabled'. I've tried with my Pro signed into my portal and out of my portal, as well as with different Python environments. My user does have admin rights to the Portal. Do you have any thoughts? Thanks!
Error 032659 updateParameters Error: Traceback (most recent call last):
File "Q:\admin\Template\expressions\ESRI_ScriptsToolboxs\Migrate Content\Migrate Content.atbx\CopyWebMap.tool\tool.script.validate.py", line 203, in <module>
File "Q:\admin\Template\expressions\ESRI_ScriptsToolboxs\Migrate Content\Migrate Content.atbx\CopyWebMap.tool\tool.script.validate.py", line 109, in updateParameters
File "Q:\admin\Template\expressions\ESRI_ScriptsToolboxs\Migrate Content\Migrate Content.atbx\CopyWebMap.tool\tool.script.validate.py", line 40, in usersDropDown
File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\__init__.py", line 4943, in search
users = self._portal.get_org_users(
File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 1244, in get_org_users
resp = self._org_users_page(
File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 2922, in _org_users_page
return self.con.post("portals/self/users", postdata)
File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1524, in post
return self._handle_response(
File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1000, in _handle_response
self._handle_json_error(data["error"], errorcode)
File "C:\Users\EReid\AppData\Local\ESRI\conda\envs\arcgispro-py3-ereid\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1023, in _handle_json_error
raise Exception(errormessage)
Exception: You do not have permissions to access this resource or perform this operation.
(Error Code: 403)
Open the python console, or new Notebook in Pro by going to Analysis tab > Python dropdown:
Enter the following code, replacing https://gis.esri.com/portal with your URL to portal:
from arcgis.gis import GIS
gis = GIS('https://gis.esri.com/portal')
source_users = gis.users.search(max_users=10000)
source_users
Ex:
Does this execute successfully?
It does not, same error.
I just double checked and our Portal security actually says SAML is set up, not IWA. I assumed they were the same thing. I'm not sure if they work the same.
@Eliot_Reid I had a feeling this may be the case. SAML is not supported. If you can, create a built-in Administrator account and use that instead.
@JakeSkinner
I am getting same error as @Eliot_Reid could you help me how can I fix this error? How can I create a built-in Administrator account? I am admin in our portal org
Best,
@anonymous55 go to Organization > Members > Add Members:
There you will have an option to create a Built-in account:
@MichaelWestbrook right-click on the tool you are trying to execute and click on the Execution tab on the left. It will list the path to the python script. Make sure it exists in the path referenced here. Ex:
Thank you. I had to create all of the folders in the the and copy the data there. It works perfectly now and does exactly what I need it to do. Are you going to make it so you can copy EXB templates as well?
@MichaelWestbrook EXB templates are now supported. You will need to re-download the tool.
@JakeSkinner Thank you
Hey @JakeSkinner
Doing a staging to production migration now and just found your tool, seemingly I just found the blog sections of Community! Great tools you have here and great work you've done, thank you for providing!
Cody
Thank you @CodyPatterson, I appreciate the feedback.
Hi Jake
Great work. I have sendt you a suggestion for a updated version of the "hosted feature service" script on mail.
Best regards
David
@JakeSkinner Good morning, I am trying to copy web maps and as soon as I select the web, PortalUrl gets an error
@MichaelWestbrook I found the issue, the validation code checks the web map for tables, if none exists, it produces this error. Redownload the tools, and it should work successfully now.
@JakeSkinner Works perfect now! Thank you!
@JakeSkinner I copied the web maps over successfully. When I copied the Experience Builder it ran successfully but out of the 7 web maps in it, it only mapped one successfully.
@MichaelWestbrook I've seen sometimes if you don't tab out of each Web Map ID box, the GP tool won't pick up the IDs correctly. Try tabbing out of the New Web Map ID and re-run the tool again to see if it works.
Hi @JakeSkinner have you considered adding this to Github? Potentially as a Python Toolbox to enable easier contribution to the validation rules?
Keen to provide some contributions to this useful resource.
The transfer worked but I was wondering if this error meant something that I need to be wary of:
Hi @JakeSkinner , I'm trying to migrate a dashboard from my ArcGIS Online account to an ArcGIS Portal 11.1 account, apparently the tool works fine but when I check the item it is empty. I even checked the .json through assitant and it has the dashboard structure copied. Do you know how I could solve this problem?
PD: I am the administrator of both accounts.
@jdcayetano no, this should not be a concern. The script did not delete a temporary zip file mentioned in the path above.
@JakeHanson-eaglenz once I learn GitHub, I'll post it there.
@CommunityMapsEsriColombia migrating from ArcGIS Online to Enterprise will not be supported. ArcGIS Online is always at a later release than Enterprise, so applications will fail to open. What should work is migrating Web Maps and Hosted Feature services from AGOL to Enterprise, but I have not extensively tested this. Ideally, you should only be migrating content between the same environments (i.e. AGOL --> AGOL, Enterprise 11.1 --> Enterprise 11.1).
@JakeSkinner Thanks for your great work.
I need to copy non hosted Feature services but it does not seem to be possible.
I tried many things without success like "content.add" and "content.clone_items".
There must be a way to copy a non hosted Feature service. Do you have any clue?
@RejeanLabbe copying referenced services is very difficult, especially if the data source will change. Due to the many possible parameters when publishing referenced services, this is not easily scripted.
@JakeSkinner Thanks for your answer. If anything changes about this in the future, I would be glad to ear about that.
@JakeSkinner Do you have any plans to create a tool to copy ArcGIS Hub sites? Thank you!
Thanks so much for these tools Jake,
Just a quick question about ArcGIS Enterprise Versions being the same.
We have 10.9.1 and will be migrating to ArcGIS Enterprise 11.3
We will manually publish our registered map services to the new environment.
But we would like to use your tool to migrate the Web Maps - will the version discrepancy be an issue?
We are not worried about the apps or dashboards as we are currently using WAB app which will be rebuilt in Experience Builder apps.
No, the version discrepancy should not be an issue for web maps.
Thanks for quick reply Jake.
@JakeSkinner Thank you for this post.
I have been using this to migrate AGE 10.8 to 11.3 but ran into issues with Experience builder Apps. I tried executing the script independently and also in ArcGIS pro but encountering this error in both the places. Hope you can help.
The issue I'm facing is for the copy experience builder app file
Furthemore the zip is not downloadable anymore. looks like it is disabled
@VigneshGopalakrishnan the tools may not work for migrating Experience Builder applications between different environments. This should work for hosted feature services, and web maps, but there could be significant changes in applications that are not supported between different versions.
Make sure you click the arrow to download the zip file:
Thank you for that @JakeSkinner .
But just wanted to bring into your attention that the point of failure is even before that here.
@VigneshGopalakrishnan it looks like Enterprise 10.8.1 handles the config.json slightly different. To fix the error you're receiving, perform the following steps:
1. Right-click on the Copy Experience Builder App script tool > Properties
2. Select the Validation tab
3. Delete everything and paste in the below:
import arcpy, json
from arcgis.gis import GIS
from arcgis.gis import User
class ToolValidator:
# Class to add custom behavior and properties to the tool and tool parameters.
def __init__(self):
# set self.params for use in other function
self.params = arcpy.GetParameterInfo()
def initializeParameters(self):
# Customize parameter properties.
# This gets called when the tool is opened.
return
def disableIWA(self, paramIndex1, paramIndex2):
'''Disable IWA option if connecting to AGOL'''
if 'arcgis.com' in self.params[paramIndex1].value:
self.params[paramIndex2].enabled = False
else:
self.params[paramIndex2].enabled = True
def disableUsernamePassword(self, paramIndex1, paramIndex2, paramIndex3):
'''Disable username/password if using Windows Authentication'''
if self.params[paramIndex1].value == True:
self.params[paramIndex2].enabled = False
self.params[paramIndex3].enabled = False
if self.params[paramIndex1].value == False:
self.params[paramIndex2].enabled = True
self.params[paramIndex3].enabled = True
def usersDropDown(self, source, paramIndex1, paramIndex2):
'''Populate User Dropdown'''
sourceUserList = []
source_users = source.users.search(max_users=100000)
for user in source_users:
if user.user_types()['name'] not in ('Viewer', 'Editor', 'Field Worker'):
sourceUserList.append(user.username)
sourceUserFilter = self.params[paramIndex2].filter
sourceUserFilter.list = sourceUserList
def getUserContent(self, source, paramIndex1, paramIndex2):
'''Get User Content'''
contentList = []
username = str(self.params[paramIndex1].value)
username = username.replace("'", '')
user = User(source, username)
user_content = user.items(max_items=100000)
for item in user_content:
if item.type == 'Web Experience':
contentList.append(
str(item.title) + ' - ' + str(item.type) + ' - ' + str(item.id))
folders = user.folders
for folder in folders:
user_content = user.items(folder=folder['title'], max_items=100000)
for item in user_content:
if item.type == 'Web Experience':
contentList.append(
str(item.title) + ' - ' + str(item.type) + ' - ' + str(item.id))
contentList.sort()
contentFilter = self.params[paramIndex2].filter
contentFilter.list = contentList
def getTargetUserFolder(self, target, paramIndex1, paramIndex2):
'''Get Target User Folder'''
targetUserFolderList = ['ROOT']
user = User(target, self.params[paramIndex1].value)
for folder in user.folders:
targetUserFolderList.append(folder['title'])
targetUserFolderListFilter = self.params[paramIndex2].filter
targetUserFolderListFilter.list = targetUserFolderList
def updateParameters(self):
# Modify parameter values and properties.
# This gets called each time a parameter is modified, before
# standard validation.
# Remove Windows Authentication option for source if arcgis.com in URL
if not self.params[0].hasBeenValidated:
self.disableIWA(0, 1)
# Disable username/password for source if Windows Authentication option is True
for i in range(0, 2):
if not self.params[i].hasBeenValidated:
self.disableUsernamePassword(1, 2, 3)
# Connect to portal using username/password to populate dropdown for source users
if self.params[1].value == False:
for i in (0, 2, 3):
if not self.params[i].hasBeenValidated:
if self.params[0].value and self.params[2].value and self.params[3].value:
source = GIS(self.params[0].value, self.params[2].value, self.params[3].value,
verify_cert=False)
self.usersDropDown(source, 0, 4)
# Connect to portal using Windows Authentication to populate dropdown for source users
elif self.params[1].value == True:
if not self.params[1].hasBeenValidated:
if self.params[0].value:
source = GIS(self.params[0].value, verify_cert=False)
self.usersDropDown(source, 0, 4)
# Dropdown for source user's content
if self.params[1].value == False:
for i in (0, 2, 3, 4):
if not self.params[i].hasBeenValidated:
if self.params[0].value and self.params[4].value:
source = GIS(self.params[0].value, self.params[2].value, self.params[3].value,
verify_cert=False)
self.getUserContent(source, 4, 5)
elif self.params[1].value == True:
for i in (0, 1, 4):
if not self.params[i].hasBeenValidated:
if self.params[0].value and self.params[4].value:
source = GIS(self.params[0].value, verify_cert=False)
self.getUserContent(source, 4, 5)
# Get Web Map IDs in Experience Builder App
webMapVtab = self.params[6]
embedVtab = self.params[7]
if self.params[5].altered and not self.params[5].hasBeenValidated:
if self.params[1].value == False:
source = GIS(self.params[0].value, self.params[2].value, self.params[3].value, verify_cert=False)
elif self.params[1].value == True:
source = GIS(self.params[0].value, verify_cert=False)
webMapIdList = []
embededURLList = []
expAppID = self.params[5].value.split(' - ')[-1]
expApp = source.content.get(expAppID)
expAppJSON = expApp.get_data(try_json=True)
jsonFile = expApp.resources.get(file='config/config.json', try_json=True)
with open(jsonFile, 'r') as file:
resourceJSON = json.load(file)
# Create list to store data sources from Config.JSON file
for val in resourceJSON['dataSources']:
webmaps = []
if resourceJSON['dataSources'][val]['type'] == 'WEB_MAP':
webMapID = (resourceJSON['dataSources'][val]['itemId'])
webmaps.append(webMapID)
webmaps.append('')
if len(webmaps) > 0:
webMapIdList.append(webmaps)
for widget in expAppJSON['widgets']:
embededURLs = []
try:
embedURL = expAppJSON['widgets'][widget]['config']['expression']
embedURL = embedURL.split('>')[1]
embedURL = embedURL.split('<')[0]
embededURLs.append(embedURL)
embededURLs.append('')
except:
pass
if len(embededURLs) > 0:
embededURLList.append(embededURLs)
for val in resourceJSON['dataSources']:
otherDataSources = []
try:
serviceURL = resourceJSON['dataSources'][val]['url']
if serviceURL not in otherDataSources:
otherDataSources.append(serviceURL)
otherDataSources.append('')
except:
pass
if len(otherDataSources) > 0:
embededURLList.append(otherDataSources)
# Remove empty lists
for webmap in webMapIdList:
if len(webmap) == 0:
webMapIdList.remove(webmap)
for embed in embededURLList:
if len(embed) == 0:
embededURLList.remove(embed)
# Remove duplicates from list
embededURLList = [i for n, i in enumerate(embededURLList) if i not in embededURLList[:n]]
webMapVtab.values = webMapIdList
embedVtab.values = embededURLList
# Remove Windows Authentication option for target if arcgis.com in URL
if not self.params[8].hasBeenValidated:
self.disableIWA(8, 9)
# Disable username/password for target if Windows Authentication option is True
for i in range(8, 10):
if not self.params[i].hasBeenValidated:
self.disableUsernamePassword(9, 10, 11)
# Connect to portal using username/password to populate dropdown for target users
if self.params[9].value == False:
for i in (8, 10, 11):
if not self.params[i].hasBeenValidated:
if self.params[8].value and self.params[10].value and self.params[11].value:
target = GIS(self.params[8].value, self.params[10].value, self.params[11].value,
verify_cert=False)
self.usersDropDown(target, 8, 12)
# Connect to portal using Windows Authentication to populate dropdown for target users
elif self.params[9].value == True:
if not self.params[9].hasBeenValidated:
if self.params[8].value:
target = GIS(self.params[8].value, verify_cert=False)
self.usersDropDown(target, 8, 12)
# Connect to portal using username/password to populate dropdown for Target User's folder
if self.params[9].value == False:
for i in (8, 10, 11, 12):
if not self.params[i].hasBeenValidated:
if self.params[12].value:
target = GIS(self.params[8].value, self.params[10].value, self.params[11].value,
verify_cert=False)
self.getTargetUserFolder(target, 12, 13)
# Connect to portal using Windows Authentication to populate dropdown for Target User's folder
elif self.params[9].value == True:
for i in (8, 9, 12):
if not self.params[i].hasBeenValidated:
if self.params[12].value:
target = GIS(self.params[8].value, verify_cert=False)
self.getTargetUserFolder(target, 12, 13)
return
def updateMessages(self):
# Customize messages for the parameters.
# This gets called after standard validation.
return
# def isLicensed(self):
# # set tool isLicensed.
# return True
# def postExecute(self):
# # This method takes place after outputs are processed and
# # added to the display.
# return
Thank you @JakeSkinner
This one has the same issue. I believe the issue here could be that the Exp Builder that I'm trying to migrate is unpublished
Hi @JakeSkinner
I am trying to use your GP tool to copy a hosted feature layer from one 11.3 Portal to another.
I am getting this error using latest version of PRO.
Traceback (most recent call last):
File "C:\Testing\Migrate Content\Copy Hosted Services.py", line 57, in <module>
result_item = item.export(export_name, 'File Geodatabase', wait=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\nicholas.miller\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\__init__.py", line 14095, in export
raise Exception("Could not export item: %s" % self.itemid)
Exception: Could not export item: 7e34b118159246ec95be1a2b97350c08
@NickMiller2 if you search for item 7e34b118159246ec95be1a2b97350c08 in Portal, are you able to successfully export this hosted feature service to a File Geodatabase?
@Teo unfortunately, not. This would be a referenced service. See the notes above:
but referenced services can be rather difficult. For example, there are numerous prerequisites that are required for a service to be published from an Enterprise Geodatabase, such as:
does the ArcGIS Service Account have necessary privileges
is the geodatabase registered with the ArcGIS Server instance
does the feature class in fact exist within the new geodatabase
etc
Hi,
I'm wandering if you could simplify the process just with a copy function to
1. down item from source
2. down item/data from source
3. download thumbnail, metadata from source
4. create an item in target using item, item/data thumbnail and metadata
5. download resources from source and copy to target
@JakeSkinner , My organization is moving all our Portal content from an on-prem Enterprise setup to a brand new one in Azure. We have hundreds of users with thousands of items of content so want to make this as painless for our end users as possible so will be scripting the move as much as possible. We've got a good handle on cloning all hosted layers, WebMaps, etc. So, going back to the idea of cloning non-hosted services ---
In our scripts (using Admin Credentials) we plan to iterate through items and, if it is a referenced feature layer --
1. Open the .mapx file associated with the service (located on PortalHost -- arcgisserver\directories\arcgissystem\arcgisinput) and find all the data sources. For each of the data sources, we will register the data source on the new Portal.
2. Publish the service to the new Portal -- This is the tricky part. The advantage of cloning is that it maintains all the properties that the user originally specified when publishing (edit rights, pooling, capabilities, etc). It would take a ton of time for users to do this by hand for each and every service. I was thinking I'd query the original item to determine all the properties needed for the copy and then publish to the new Portal.
If you knew of a better way to go about it I'd be interested to hear!
@ChristopherPouliot I would recommend migrating your entire Enterprise using the webgisdr instead:
If the DNS is going to change in the new instance, starting at 11.4, there are built-in tools to update the Organization URL:
https://enterprise.arcgis.com/en/portal/latest/administer/windows/update-the-organization-url.htm
Thanks @JakeSkinner . Sounds like what we need.
Hi @JakeSkinner,
I do have the same problem as @EliotReid (3/20/2024). We have SAML on our portal, but I am using a built in account (admin account). It is the same on staging and prod. Any help will be appreciated because this tool seems to be the one I need.
I am working with enterprise 11.1 and AGP 3.1.7