Migrate Content for ArcGIS Enterprise/ArcGIS Online

[#1]

It's always a good practice to have a DEV/STAGING environment along with a PRODUCTION environment for ArcGIS Enterprise. I frequently get asked how to migrate content from one to the other.  Hosted services can be straight forward, and can easily be scripted (expect an update later to this toolset for this functionality), but referenced services can be rather difficult. For example, there are numerous prerequisites that are required for a service to be published from an Enterprise Geodatabase, such as:

• does the ArcGIS Service Account have necessary privileges
• is the geodatabase registered with the ArcGIS Server instance
• does the feature class in fact exist within the new geodatabase
• etc

These tools pick up once your services have been published to the other environment and you would like to begin migrating web maps, apps, and dashboards. Take a look at the video below on how to execute these tools.

These tools should only be used to migrate web maps/apps where the environments are the same version (i.e. 11.1).  Migrating content between two ArcGIS Online organizations is also supported.

Currently, these tools have only been tested in ArcGIS Enterprise 11.1 and ArcGIS Online. If there are any issues, please report in the comment section below.

 

Updates

1/16/24:  Added tools to copy hosted feature services and file based items.

1/29/24:  Added option to run when portal has Windows Authentication enabled.  Also, updated the Copy Dashboard tool to specify stand-alone layers.

2/12/24:  Web Map item IDs now update

3/28/24:  Added tool to copy Story Maps

6/30/24:  Group layers and tables are now supported when copying web maps

7/10/24: Experience Builder Templates are now supported

Known Issues

• If hosted feature services exist in Enterprise, they will not successfully overwrite when executing the Copy Hosted Feature Services tool.  Will successfully overwrite in AGOL.  Possible bug with ArcGIS API for Python
• After Story Map is copied to Enterprise, the Story Map must first be opened in Edit mode and published before it can be successfully viewed.  This is not the case for AGOL.  Possible bug with ArcGIS API for Python

[JakeSkinner]

@denisCallens from you screenshot, it appears you are connecting without specifying a username/password, so you wouldn't be using a built-in account.  Can you send a screenshot of your Logins by going to Portal > Organization > Settings > Logins:

[denisCallens]

Here are my login settings 

[JakeSkinner]

@denisCallens can you send a screenshot of how you are authenticating using the tool in ArcGIS Pro?

[denisCallens]

Here it is. My active portal in AGP is the source portal

Here is the full error message

Error 032659 updateParameters Error: Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 802, in token
self._init_response_type_token()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 551, in _init_response_type_token
raise Exception("Unable to generate oauth token")
Exception: Unable to generate oauth token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 644, in _init_token_auth_handshake
oauth_info = json.loads(js_object)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 829, in get
resp = self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\api.py", line 437, in get
return self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 573, in request
prep = self.prepare_request(req)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 484, in prepare_request
p.prepare(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 372, in prepare
self.prepare_auth(auth, url)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 603, in prepare_auth
r = auth(self)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 818, in __call__
r.headers["X-Esri-Authorization"] = f"Bearer {self.token}"
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 808, in token
self._init_token_auth_handshake()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 646, in _init_token_auth_handshake
oauth_info = json.loads(js_object + "}")
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1948, in _check_product
res = self.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 861, in get
raise Exception("A general error occurred: %s" % e)
Exception: A general error occurred: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 802, in token
self._init_response_type_token()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 551, in _init_response_type_token
raise Exception("Unable to generate oauth token")
Exception: Unable to generate oauth token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 644, in _init_token_auth_handshake
oauth_info = json.loads(js_object)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 829, in get
resp = self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\api.py", line 437, in get
return self._session.get(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 573, in request
prep = self.prepare_request(req)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\sessions.py", line 484, in prepare_request
p.prepare(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 372, in prepare
self.prepare_auth(auth, url)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\requests\models.py", line 603, in prepare_auth
r = auth(self)
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 818, in __call__
r.headers["X-Esri-Authorization"] = f"Bearer {self.token}"
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 808, in token
self._init_token_auth_handshake()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\auth\_auth\_token.py", line 646, in _init_token_auth_handshake
oauth_info = json.loads(js_object + "}")
File "D:\ARCGIS\cloneEnvi\240620\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\ARCGIS\cloneEnvi\240620\lib\json\decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 615 (char 614)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:************\terrain\Migrate_Content\Migrate Content.atbx#CopyWebMap_NewToolbox.UpdateParameters.py", line 219, in <module>
File "D:************\terrain\Migrate_Content\Migrate Content.atbx#CopyWebMap_NewToolbox.UpdateParameters.py", line 102, in updateParameters
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\__init__.py", line 585, in __init__
raise e
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\__init__.py", line 524, in __init__
self._portal = _portalpy.Portal(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 205, in __init__
self.con = Connection(
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 354, in __init__
self._product = self._check_product()
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1954, in _check_product
res = self.get(baseurl + "info", params={"f": "json"})
File "D:\ARCGIS\cloneEnvi\240620\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 861, in get
raise Exception("A general error occurred: %s" % e)
Exception: A general error occurred: Expecting ',' delimiter: line 1 column 615 (char 614)

[JakeSkinner]

@denisCallens did you change the validation code?  It looks like you're receiving a different error (i.e. Unable to generate Oauth token).  This error should not be generated when using built-in credentials.  Below is the error I'm receiving when specifying invalid credentials:

[denisCallens]

Hi @JakeSkinner, 

I didn't change the code but to be sure I downloaded the toolbox again.

No more success but i notice something odd:

In the "copy web map" tool,

• I insert my portal url, my source admin username and source admin password
• When I clicked in "source user" field, the "source Admin Username" disappears.

Then, if i fill my username again, i am back to the error message I sent last week.

[JakeSkinner]

@denisCallens in the Python 3 command prompt, ArcGIS Pro Notebook, or Python IDE, try executing the below code:

from arcgis.gis import GIS

gis = GIS('https://dns.domain.com/portal', 'built-in-user', 'password', verify_cert=False)

Does this work successfully?

[chris_del101]

I keep getting error when I try to login to the tool. I am copy pasting my credentials so they are 100% correct. I've also tried manual entry so it's not due to extra characters, etc.

It was working for a period before this occurred so this makes it even more baffling - that it stopped working

Exception: A general error occurred: Too many invalid attempts. Please try again later.

Obv the creds in the photo are phony, but the error output is the same.

[JakeSkinner]

@chris_del101 what type of security are you attempting to sign in with (i.e. Built-in, Active Directory, SAML)?  Only built-in and active directory are supported.  Check the comment above your last one that has a python snippet.  Does this work for you?

[chris_del101]

@JakeSkinner Thanks for response.

It fails with the same error, but not always.

If I am getting this failure, why is it not failing 100% of the time? I closed and reopened and it connected again. Open/close doesn't look like silver bullet though since it doesn't always fix it. 

[JakeSkinner]

@chris_del101 

1.  Can you send a screen shot of Portal > Organization > Settings > Security > Logins:

2.  Is there an '@' symbol in your username?  For example, jskinner@ESRI.

3.  Check the casing of your username.

4.  What version of ArcGIS Pro and ArcGIS Enterprise are you running?

[denisCallens]

Hi @JakeSkinner,

I have the feeling that your tool is affected by a more general issue explained here : https://support.esri.com/en-us/knowledge-base/error-json-decoder-jsondecodererror-expecting-value-li...

My IT guy is currently investigating our reverse proxy or IIS to allow FQDN path. I will let you know if we have any success with the workaround.

[JakeSkinner]

@denisCallens do you have Windows Authentication enabled on Portal's web adaptor?  If so, you need to check the box highlighted below:

This will remove the username/password parameters and attempt to authenticate using your windows credentials.

[denisCallens]

Hi @JakeSkinner,

I have single sign on via OpenID Connect login (see screenshot below). I tried with the box checked. Same error. 

Earlier you mention that portal with SAML authentication has to use built in user. This why I try this way.

[JakeSkinner]

@denisCallens in the Python 3 command prompt, ArcGIS Pro Notebook, or Python IDE, try executing the below code:

from arcgis.gis import GIS

gis = GIS('https://dns.domain.com/portal', 'built-in-user', 'password', verify_cert=False)

Does this work successfully?

[GIS-KompetenzzentrumBern]

When copying an Experience Builder App with an embedded feature layer (not a web map) the references do not match. The toolbox does not change the item IDs. 

[JakeSkinner]

@GIS-KompetenzzentrumBern can you walk me through your workflow of how you're embedding the Feature Layer?

[GIS-KompetenzzentrumBern]

Hi @JakeSkinner 

I add directly a Feature Layer instead of a web map to a table widget, because i don't need a map in this specific case. After copying the experience the table widget does not find its data. Everything else works perfectly. 

[JakeSkinner]

@GIS-KompetenzzentrumBern this is now resolved.  Download the latest version of the tools.  If there are any issues, please report back.

[denisCallens]

Hi @JakeSkinner,

I come back regarding the issue I report last month.

I couldn't be authenticated with built-in user. There is a webtier authentication via SAML on my portal. 

I get the following error: 

Exception: A general error occurred: Expecting ',' delimiter: line 1 column 627 (char 626)

 After a lot of searching I found on a GitHub page (https://github.com/Esri/arcgis-python-api/issues/1712) the following suggestion : "Adding "use_gen_token=True" parameter fixed this issue for me"

I added this parameter every time your script call GIS class. See below for example.

This does the trick, and your tool works with my portals.

Thank you for this great piece of work.

[JakeSkinner]

@denisCallens awesome, thanks for reporting back what resolved this issue for you.  I was unaware of that parameter.

[chris_del101]

Hey @JakeSkinner, 

I'm replying to a post above from 12/17. 
Here is the screenshot of the login panel

My username has no special chars and is being input correctly. 

Enterprise 11.3 and Pro 3.4

This the latest. I input the info and it just keeps loading with dots across the top. This is the error and screen. I can't even get the data to load. It just does this until I close it

So I close it and restart Pro and I get a different error. 

[JakeSkinner]

@chris_del101 do you have a different admin account that you can try?  Preferably one that does not begin with numbers?

[chris_del101]

Hi @JakeSkinner,

So I've been able to get it to work with said admin account, but then I need the source and target to be the same portal. This would be okay for testing, but then I get this error when I try to clone.

Exception: {'message': "Service name 'chris_copy_Xact_Polygon_Map' already exists for '0123456789ABCDEF'"} ...

targetService = fgd.publish(overwrite=True) ...

Exception: Analyze Service error: Server tool execution failed : ERROR 000800: The value is not a member of SHAPEFILE | CSV | EXCEL. Failed.
(Error Code: 0)

 I only have another admin account with numbers at the start. If that's the only problem here then I can try to get another account.

[JakeSkinner]

@chris_del101   There is a known issue with these tools:

But, I'm not sure I'm following.  Are you trying to copy a hosted feature service to the same portal with the same name?

[LeeHwang]

Hi @JakeSkinner ,

Thank you so much for developing this tool!  It has really sped things up and alleviated a lot of headache migrating servers.  I was able to use the Copy Web Map and Copy Experience Builder App successfully and the experience site worked on the new machine.  However, once I shut down the original machine and tried to open the experience site on the new machine, I was no longer able to open the site or web map.  On the site page, I would receive a "Creating Map Failed" message.  Seems the data is still referencing the old machine.  Any thoughts?

Thanks,

Lee

[JakeSkinner]

@LeeHwang first start with the web map.  Can you verify the data sources in the web map were mapped correctly to the new data sources in the target environment?  For example, go to the web map details and click on the layers:

Is the URL at the bottom right of the page pointing to the correct URL?

[LeeHwang]

@JakeSkinner Yes, the layer found in the web map details page takes me to the details page of the map image layer being referenced.  I'm able to view the map image layer in Map Viewer but not the web map in the Map Viewer.  The domain of the URL of the Map Image Layers is correct and referencing the new machine.  

[JakeSkinner]

@LeeHwang can you upload a screenshot of the error you are receiving when trying to view the web map?

[LeeHwang]

@JakeSkinner 

[LeeHwang]

@JakeSkinner Please disregard that last post with the screenshot as it was from a different site.  I'll post a screenshot as soon as I able as I'm having licensing issues now so am not able to open Pro to recreate the layers and issue.  

[chris_del101]

Hi @JakeSkinner , 

So I am cloning to two different portals now, not one that starts with a number in the username anymore, and copying within the same portal does not work I guess - but I was able to get a feature to clone.

Next question: I have written a clone tool like yours but it does not work because the export function fails to export all the sublayers. I have tried playing with py export and rest export alone, and even alone these do not work. I even exported direct from portal and this also does not export the sublayers.

Yet, once I was finally able to get your tool run without error, it DID copy the sublayers. This is miraculous. How did you do it? 🙂 How did you get export to work? 
I copied alot for your code and also use the same line, but for me it does not work:

result_item = item.export(export_name, 'File Geodatabase', wait=True)

 The only differences are the py api version, and that I not running this inside Pro. I guess I can try these two things in the meantime to see if they make any diff.