Migrate Content for ArcGIS Enterprise/ArcGIS Online

[#1]

It's always a good practice to have a DEV/STAGING environment along with a PRODUCTION environment for ArcGIS Enterprise. I frequently get asked how to migrate content from one to the other.  Hosted services can be straight forward, and can easily be scripted (expect an update later to this toolset for this functionality), but referenced services can be rather difficult. For example, there are numerous prerequisites that are required for a service to be published from an Enterprise Geodatabase, such as:

• does the ArcGIS Service Account have necessary privileges
• is the geodatabase registered with the ArcGIS Server instance
• does the feature class in fact exist within the new geodatabase
• etc

These tools pick up once your services have been published to the other environment and you would like to begin migrating web maps, apps, and dashboards. Take a look at the video below on how to execute these tools.

These tools should only be used to migrate web maps/apps where the environments are the same version (i.e. 11.1).  Migrating content between two ArcGIS Online organizations is also supported.

Currently, these tools have only been tested in ArcGIS Enterprise 11.1 and ArcGIS Online. If there are any issues, please report in the comment section below.

Migrating Referenced Services:

Updates

1/16/24:  Added tools to copy hosted feature services and file based items.

1/29/24:  Added option to run when portal has Windows Authentication enabled.  Also, updated the Copy Dashboard tool to specify stand-alone layers.

2/12/24:  Web Map item IDs now update

3/28/24:  Added tool to copy Story Maps

6/30/24:  Group layers and tables are now supported when copying web maps

7/10/24: Experience Builder Templates are now supported

5/1/25:  Web Scenes are now supported

6/14/25:  Updated for ArcGIS API 2.4.1

10/6/25:  Referenced services are now supported

10/28/25:  Hosted Views and Surveys are now supported

Known Issues

• If hosted feature services exist in Enterprise, they will not successfully overwrite when executing the Copy Hosted Feature Services tool.  Will successfully overwrite in AGOL.  Possible bug with ArcGIS API for Python
• After Story Map is copied to Enterprise, the Story Map must first be opened in Edit mode and published before it can be successfully viewed.  This is not the case for AGOL.  Possible bug with ArcGIS API for Python

[JakeSkinner]

@denisCallens from you screenshot, it appears you are connecting without specifying a username/password, so you wouldn't be using a built-in account.  Can you send a screenshot of your Logins by going to Portal > Organization > Settings > Logins:

[denisCallens]

Here are my login settings 

[JakeSkinner]

@denisCallens can you send a screenshot of how you are authenticating using the tool in ArcGIS Pro?

[denisCallens]

Here it is. My active portal in AGP is the source portal

Here is the full error message

Error 032659 updateParameters Error: Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 802, in token
self._init_response_type_token()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 551, in _init_response_type_token
raise Exception("Unable to generate oauth token")
Exception: Unable to generate oauth token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 644, in _init_token_auth_handshake
oauth_info = json.loads(js_object)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 829, in get
resp = self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\api.py", line 437, in get
return self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 573, in request
prep = self.prepare_request(req)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 484, in prepare_request
p.prepare(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 372, in prepare
self.prepare_auth(auth, url)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 603, in prepare_auth
r = auth(self)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 818, in __call__
r.headers["X-Esri-Authorization"] = f"Bearer {self.token}"
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 808, in token
self._init_token_auth_handshake()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 646, in _init_token_auth_handshake
oauth_info = json.loads(js_object + "}")
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1948, in _check_product
res = self.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 861, in get
raise Exception("A general error occurred: %s" % e)
Exception: A general error occurred: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 802, in token
self._init_response_type_token()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 551, in _init_response_type_token
raise Exception("Unable to generate oauth token")
Exception: Unable to generate oauth token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 644, in _init_token_auth_handshake
oauth_info = json.loads(js_object)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 829, in get
resp = self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\api.py", line 437, in get
return self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 573, in request
prep = self.prepare_request(req)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 484, in prepare_request
p.prepare(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 372, in prepare
self.prepare_auth(auth, url)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 603, in prepare_auth
r = auth(self)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 818, in __call__
r.headers["X-Esri-Authorization"] = f"Bearer {self.token}"
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 808, in token
self._init_token_auth_handshake()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 646, in _init_token_auth_handshake
oauth_info = json.loads(js_object + "}")
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:************\terrain\Migrate_Content\Migrate Content.atbx#CopyWebMap_NewToolbox.UpdateParameters.py", line 219, in <module>
File "D:************\terrain\Migrate_Content\Migrate Content.atbx#CopyWebMap_NewToolbox.UpdateParameters.py", line 102, in updateParameters
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\__init__.py", line 585, in __init__
raise e
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\__init__.py", line 524, in __init__
self._portal = _portalpy.Portal(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 205, in __init__
self.con = Connection(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 354, in __init__
self._product = self._check_product()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1954, in _check_product
res = self.get(baseurl + "info", params={"f": "json"})
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 861, in get
raise Exception("A general error occurred: %s" % e)
Exception: A general error occurred: Expecting ',' delimiter: line 1 column 615 (char 614)

[JakeSkinner]

@denisCallens did you change the validation code?  It looks like you're receiving a different error (i.e. Unable to generate Oauth token).  This error should not be generated when using built-in credentials.  Below is the error I'm receiving when specifying invalid credentials:

[denisCallens]

Hi @JakeSkinner, 

I didn't change the code but to be sure I downloaded the toolbox again.

No more success but i notice something odd:

In the "copy web map" tool,

• I insert my portal url, my source admin username and source admin password
• When I clicked in "source user" field, the "source Admin Username" disappears.

Then, if i fill my username again, i am back to the error message I sent last week.

[JakeSkinner]

@denisCallens in the Python 3 command prompt, ArcGIS Pro Notebook, or Python IDE, try executing the below code:

from arcgis.gis import GIS

gis = GIS('https://dns.domain.com/portal', 'built-in-user', 'password', verify_cert=False)

Does this work successfully?

[chris_del101]

I keep getting error when I try to login to the tool. I am copy pasting my credentials so they are 100% correct. I've also tried manual entry so it's not due to extra characters, etc.

It was working for a period before this occurred so this makes it even more baffling - that it stopped working

Exception: A general error occurred: Too many invalid attempts. Please try again later.

Obv the creds in the photo are phony, but the error output is the same.

[JakeSkinner]

@chris_del101 what type of security are you attempting to sign in with (i.e. Built-in, Active Directory, SAML)?  Only built-in and active directory are supported.  Check the comment above your last one that has a python snippet.  Does this work for you?

[chris_del101]

@JakeSkinner Thanks for response.

It fails with the same error, but not always.

If I am getting this failure, why is it not failing 100% of the time? I closed and reopened and it connected again. Open/close doesn't look like silver bullet though since it doesn't always fix it. 

[JakeSkinner]

@chris_del101 

1.  Can you send a screen shot of Portal > Organization > Settings > Security > Logins:

2.  Is there an '@' symbol in your username?  For example, jskinner@ESRI.

3.  Check the casing of your username.

4.  What version of ArcGIS Pro and ArcGIS Enterprise are you running?

[denisCallens]

Hi @JakeSkinner,

I have the feeling that your tool is affected by a more general issue explained here : https://support.esri.com/en-us/knowledge-base/error-json-decoder-jsondecodererror-expecting-value-li...

My IT guy is currently investigating our reverse proxy or IIS to allow FQDN path. I will let you know if we have any success with the workaround.

[JakeSkinner]

@denisCallens do you have Windows Authentication enabled on Portal's web adaptor?  If so, you need to check the box highlighted below:

This will remove the username/password parameters and attempt to authenticate using your windows credentials.

[denisCallens]

Hi @JakeSkinner,

I have single sign on via OpenID Connect login (see screenshot below). I tried with the box checked. Same error. 

Earlier you mention that portal with SAML authentication has to use built in user. This why I try this way.

[JakeSkinner]

@denisCallens in the Python 3 command prompt, ArcGIS Pro Notebook, or Python IDE, try executing the below code:

from arcgis.gis import GIS

gis = GIS('https://dns.domain.com/portal', 'built-in-user', 'password', verify_cert=False)

Does this work successfully?

[GIS-KompetenzzentrumBern]

When copying an Experience Builder App with an embedded feature layer (not a web map) the references do not match. The toolbox does not change the item IDs. 

[JakeSkinner]

@GIS-KompetenzzentrumBern can you walk me through your workflow of how you're embedding the Feature Layer?

[GIS-KompetenzzentrumBern]

Hi @JakeSkinner 

I add directly a Feature Layer instead of a web map to a table widget, because i don't need a map in this specific case. After copying the experience the table widget does not find its data. Everything else works perfectly. 

[JakeSkinner]

@GIS-KompetenzzentrumBern this is now resolved.  Download the latest version of the tools.  If there are any issues, please report back.

[denisCallens]

Hi @JakeSkinner,

I come back regarding the issue I report last month.

I couldn't be authenticated with built-in user. There is a webtier authentication via SAML on my portal. 

I get the following error: 

Exception: A general error occurred: Expecting ',' delimiter: line 1 column 627 (char 626)

 After a lot of searching I found on a GitHub page (https://github.com/Esri/arcgis-python-api/issues/1712) the following suggestion : "Adding "use_gen_token=True" parameter fixed this issue for me"

I added this parameter every time your script call GIS class. See below for example.

This does the trick, and your tool works with my portals.

Thank you for this great piece of work.

[JakeSkinner]

@denisCallens awesome, thanks for reporting back what resolved this issue for you.  I was unaware of that parameter.

[chris_del101]

Hey @JakeSkinner, 

I'm replying to a post above from 12/17. 
Here is the screenshot of the login panel

My username has no special chars and is being input correctly. 

Enterprise 11.3 and Pro 3.4

This the latest. I input the info and it just keeps loading with dots across the top. This is the error and screen. I can't even get the data to load. It just does this until I close it

So I close it and restart Pro and I get a different error. 

[JakeSkinner]

@chris_del101 do you have a different admin account that you can try?  Preferably one that does not begin with numbers?

[chris_del101]

Hi @JakeSkinner,

So I've been able to get it to work with said admin account, but then I need the source and target to be the same portal. This would be okay for testing, but then I get this error when I try to clone.

Exception: {'message': "Service name 'chris_copy_Xact_Polygon_Map' already exists for '0123456789ABCDEF'"} ...

targetService = fgd.publish(overwrite=True) ...

Exception: Analyze Service error: Server tool execution failed : ERROR 000800: The value is not a member of SHAPEFILE | CSV | EXCEL. Failed.
(Error Code: 0)

 I only have another admin account with numbers at the start. If that's the only problem here then I can try to get another account.

[JakeSkinner]

@chris_del101   There is a known issue with these tools:

But, I'm not sure I'm following.  Are you trying to copy a hosted feature service to the same portal with the same name?

[LeeHwang]

Hi @JakeSkinner ,

Thank you so much for developing this tool!  It has really sped things up and alleviated a lot of headache migrating servers.  I was able to use the Copy Web Map and Copy Experience Builder App successfully and the experience site worked on the new machine.  However, once I shut down the original machine and tried to open the experience site on the new machine, I was no longer able to open the site or web map.  On the site page, I would receive a "Creating Map Failed" message.  Seems the data is still referencing the old machine.  Any thoughts?

Thanks,

Lee

[JakeSkinner]

@LeeHwang first start with the web map.  Can you verify the data sources in the web map were mapped correctly to the new data sources in the target environment?  For example, go to the web map details and click on the layers:

Is the URL at the bottom right of the page pointing to the correct URL?

[LeeHwang]

@JakeSkinner Yes, the layer found in the web map details page takes me to the details page of the map image layer being referenced.  I'm able to view the map image layer in Map Viewer but not the web map in the Map Viewer.  The domain of the URL of the Map Image Layers is correct and referencing the new machine.  

[JakeSkinner]

@LeeHwang can you upload a screenshot of the error you are receiving when trying to view the web map?

[LeeHwang]

@JakeSkinner 

[LeeHwang]

@JakeSkinner Please disregard that last post with the screenshot as it was from a different site.  I'll post a screenshot as soon as I able as I'm having licensing issues now so am not able to open Pro to recreate the layers and issue.  

[chris_del101]

Hi @JakeSkinner , 

So I am cloning to two different portals now, not one that starts with a number in the username anymore, and copying within the same portal does not work I guess - but I was able to get a feature to clone.

Next question: I have written a clone tool like yours but it does not work because the export function fails to export all the sublayers. I have tried playing with py export and rest export alone, and even alone these do not work. I even exported direct from portal and this also does not export the sublayers.

Yet, once I was finally able to get your tool run without error, it DID copy the sublayers. This is miraculous. How did you do it? 🙂 How did you get export to work? 
I copied alot for your code and also use the same line, but for me it does not work:

result_item = item.export(export_name, 'File Geodatabase', wait=True)

 The only differences are the py api version, and that I not running this inside Pro. I guess I can try these two things in the meantime to see if they make any diff.

[SerjStol]

Unfortunately copying Experience Builder Apps does not seem to work properly for us. It does not pickup all the itemids and urls that need to be swapped over. I had error opening app "Item does not exit CONT001" in the target ArcGIS Enterprise. Upon inspecting the config json i found that lots was still referring to the source Portal. For instance Basemap Gallery widget was referring to the old web map item ids and thumbnail.png. Additionally all the utility services like print and custom GP tool that I had configured werent picked up. It only picked up the web map and one feature service that was added in ExB Builder. Upon swapping all itemids and urls in PS AGO Assistant, we managed to get the app working in the target environment.

[JakeSkinner]

@SerjStol sometimes if you don't tab out of each URL or web map ID section, the GP tool does not pick up a value has been entered.  After entering a URL or web map ID, I always click the Tab key on my keyboard.  If I get the error you were mentioning, simply tab out of each section and execute the tool again.

[SerjStol]

@JakeSkinner I meant that the tool doesnt auto populate all the source URLs and item Ids that need to be remapped. You have to manually search the config.json and find all the items that need to be remapped. But in anyway it migrated the app and then some manual tweaks using AGOL assistant find and replace made the app working so its still a useful tool!

[JakeSkinner]

@SerjStol if the exp app is in AGOL, feel free to invite my account (jskinner_rats) to a Group that it's shared to and I can take a look.

[GIS_Limburg_Province_Belgium]

@JakeSkinner, useful tool, but no luck for me. I want tot copy a web map, fill in the form without any problem, but if I click Run there is this message: 

ERROR 000576: Script associated with this tool does not exist.

Strange, because the script certainly does exist! I'm still using 10.9.1.

[JakeSkinner]

@GIS_Limburg_Province_Belgium right-click on the Copy Web Map tool > Properties > Execution tab.  Check the path of the script location there and ensure it is correct.

[GIS_Limburg_Province_Belgium]

Thanks, my web map has been migrated without problem now. But migrating the web app gives troubles. Your tool ‘Copy web app’ runs without error messages, but the resulting app does not work (message: ‘Item does not exist or is inaccessible’).

Strange is that the url above (see picture) is correct, it refers to the target portal, but at the bottom right the url refers to the source portal, and the ids are the same.

I repeated the process several times, but no luck: always this wrong result.
Any ideas what’s going on? As said, I work with ArcGIS enterprise 10.9.1. Thanks!

[JakeSkinner]

@GIS_Limburg_Province_Belgium,

1.  Are you migrating ArcGIS Enterprise 10.9.1 to Enterprise 10.9.1?

2.  What type of application (i.e. Web AppBuilder, Instant App)?

3.  Can you send a screen shot of how you have the tool configured?

[GIS_Limburg_Province_Belgium]

1. Positive.

2. I do not know (did not develop the app myself), but the url (source portal) says:

https://<xyzxyzx>/portal/apps/instant/minimalist/index.html?appid=b70529dd0e794008aefb2650224557b7

3. See below how I filled the form. I left the Environments tab untouched.

[JakeSkinner]

@GIS_Limburg_Province_Belgium try removing the /home/ from the Source AGOL/Portal and Target AGOL/Portal parameters.  Also, I've seen some issues when you do not click the Tab key after specifying the New Web Map ID.  Place your cursor in this box and click Tab on your keyboard.

[GIS_Limburg_Province_Belgium]

I did what you say. Now, the url of the target app is correct (referring to my remark and picture yesterday). But the app gives a blank page if opened.

[JakeSkinner]

@GIS_Limburg_Province_Belgium use the AGOL Assistant and check if the new web map ID is being referenced in the app's JSON.

[GIS_Limburg_Province_Belgium]

@JakeSkinner : yes, the new web map ID is being referenced in the app's JSON. Checked it.

[GIS_Limburg_Province_Belgium]

Problem solved. App is running now. The show stopper was the picture that appears when the app starts: it did not migrate. I did not realize that, thought it also migrates automatically.

Thanks Jake for your useful tool and your assistance!

[RafaelFernandesdeSouza]

Do you have plans to add Survey123 copy?

[JakeSkinner]

@RafaelFernandesdeSouza take a look at this blog:

https://community.esri.com/t5/arcgis-survey123-blog/clone-surveys-from-one-organization-to-another/b...

[RafaelFernandesdeSouza]

@JakeSkinner, Thank you!

I have an error, about the webmap script.
I am trying to clone webmaps, but for some maps I am receiving  an error message:
Error 032659 updateParameters Error: Traceback (most recent call last):

the file is CopyWebMap.tool\tool_script.validate.py, line 218 AND line 143.

[JakeSkinner]

@RafaelFernandesdeSouza can you send a screenshot of how you have the tool setup?