The attached toolbox contains 3 tools for copying content between Portals.
File Geodatabase,CSV, Image, KML, Locator Package, Map Document, Shapefile, Microsoft Word, PDF, Microsoft Powerpoint, Microsoft Excel, Layer Package, Mobile Map Package, Geoprocessing Package, Service Definition, Scene Package, Tile Package, Vector Tile Package
For the tool to successfully update the web map, web application, and dashboard IDs the client machine executing this tool must be able to reach the Target Portal's server where Portal for ArcGIS (not the Web Adaptor) is installed (i.e. https://portal.esri.com:7443/arcgis). It uses this URL to make the post requests if Windows Authentication is enabled on Portal's web adaptor.
Each tool will share the content with Everyone, and the ArcGIS Enterprise Organization (if it is shared with these in the Source Portal). It will also share to each Group in the Target Portal if it has the same name as the Source Portal Group.
Note: This tool is in it's early development and has not been extensively tested. So far I've tested transferring content where both Portals are version 10.7.1. Looking to have any issues or recommendations reported in the comments below.
I can confirm that these tools work to copy Vector Tile Packages between Portal versions 10.6 to 10.7.1
noting this for reference:
Clone Portal users, groups and content | ArcGIS for Developers
Andres Castillo just a note on the above web link, those scripts will not copy any of the services. See the Conclusion section:
Note, this notebook did not copy over the services that power the service based items. Such items continue to point to the same URL as the ones in source portal did
For the Copy Map and Applications tool, when I attempt to specify mulitiple contents, the script tool loads endlessly.
Any suggestions?
I haven’t seen that issue before, sounds like something not right with the script. I suggest reaching out to Esri Support.
E.J. McNaughton
Andres Castillo What version of ArcGIS Pro are you running?
2.5.0
When I try to run outside of pro in vs code, I notice the token is not being stored:
Processing item: u
WARNING: ------------------
Error occurred processing item u
Error: Token Required
(Error Code: 499)
Deleting Connections
some modifications/observations I made to the script:
Andres Castillo if Windows Authentication is not enabled on Portal's web adaptor, you will not have to update the :7443/arcgis to /webadaptor. I just updated a note about this in the tool's description above. Also, thank you for catching the misspelling of Exception. I updated the code and re-uploaded.
I haven't tested going from 10.7.1 to 10.5.1. Are you able to go from 10.5.1 to 10.7.1? It could possibly be a version issue.
For the copy services script,
I can successfully copy over hosted feature layers from 10.7.1 to 10.5.1.
Windows Authentication is enabled on the wa.
Are your comment and the document description above in alignment?
The token error I experienced above was due to the fact that I didn't explicitly store my credentials in the GIS() object.
I sometimes get away with not putting my credentials because I've noticed that in other cases, portal auto-reads them using Windows Authentication and my AD credentials.
Testing transfer on Pro 2.5.1 with both AGS Enterprise systems at 10.8. So far, keep getting 'error occurred processing items' for all items. I love the concept. In the future, it might be good to have an 'all' where we can transfer all content from source regardless of user into one folder or the choice to copy content into default folder schema from source. This will help in staging and transferring AWS containers as well as solution transfers from Dev, Test and Prod.
Jake Skinner This script seems like exactly what we need. I just tried a single web map test with Pro 2.5.1 with the origin portal at 10.6.1 and the destination portal at 10.8.
I also got errors similar to the guy above: "Error occurred processing item" <class 'keyerror'> 402
Any ideas on how to get this running correctly would be greatly appreciated!
verify you are an admin on the portals
verify you are an admin on the portals
Yes Admin and verified.
Evan Marshall do you receive this error for all web maps, or just one in particular? Are you able to copy feature services successfully?
Luke Savage are you attempting to copy feature services, or web maps/apps?
Any content.
Hey Jake. I had not tried it with any services, as we are just looking to move some apps and maps over and will likely republish some services manually since we will be moving some of them to hosted layers in the portal. The map I tried was a simple web map with only our parcel layer in it.
Evan Marshall how did you add the parcel layer to the web map? If you add the layer with the URL (i.e. Add > Layer From Web), you will receive an error when trying to copy the web map. Instead, add the parcel as an Item in Portal (i.e. Content tab > Add Item). Then, add the parcel layer to the map by going to Add > Search for Layers.
By any chance, can this be used with copying over services from AGO to portal? I put in our AGO URL and it did not accept it.
I tend to use AGO Assist, but copying over feature services is unreliable. Please share if you know of other tools that can accomplish this.
Reza Tehranifar I've had sporadic results going from AGOL to Portal. I've written the below code to do this and it is much more reliable. It was originally written to copy all services from all users. You can specify an individual user to copy services from on line 117:
Removing this line will copy all services from all users.
import arcpy, requests, json, sys, os, time
from arcgis.gis import GIS
from arcgis.gis import User
arcpy.env.overwriteOutput = 1
startTime = time.clock()
# Variables
sourcePortal = 'https://www.arcgis.com' # AGOL URL
source_username = 'jskinner_CountySandbox' # AGOL Admin Username
source_password = '********' # AGOL Admin Password
targetPortal = 'https://jake2.esri.com/portal' # Portal URL
target_username = 'jskinner@AVWORLD' # Portal Admin Username
target_password = '*************' # Portal Admin Password
targetOwner = 'jskinner@AVWORLD' # Portal User to copy services to
targetFolder = 'Copy' # Portal User's folder to copy services to
# Create GIS objects for source and target portals
source = GIS(sourcePortal, source_username, source_password)
target = GIS(targetPortal, target_username, target_password)
export_folder = arcpy.env.scratchFolder
# Function to log errors
def errorLog(msg, e):
print(msg)
print("Error: {0}".format(e))
# Function to Copy Hosted Feature Services
def copyContent(item):
# Export hosted feature service to FGD, and downlaod to scracth folder
print("Processing {0}".format(item.title))
export_name = "{0}".format(item.title)
try:
print("Exporting {0}".format(item.title))
result_item = item.export(export_name, 'File Geodatabase', wait=True)
except Exception as e:
errorLog("Error exporting {0} File Geodatabase".format(item.title), e)
return
try:
print("Saving File Geodatabase to: {}".format(export_folder))
download_result = result_item.download(export_folder)
except Exception as e:
errorLog("Error saving {0} File Geodatabase to {1}".format(item.title, export_folder), e)
result_item.delete()
return
try:
print("Deleted {0} File Geodatabase from source portal".format(item.title))
result_item.delete()
except Exception as e:
errorLog("Error deleting {0} File Geodatabase from source portal".format(item.title), e)
return
# Set Item Properties
item_properties = {
'title':item.title,
'type':'File Geodatabase',
'description':item.description,
'snippet': item.snippet,
'tags':item.tags,
'extent': item.extent,
'accessInformation': item.accessInformation,
'licenseInfo': item.licenseInfo
}
# Get thumbnail and metadata
try:
thumbnail_file = item.download_thumbnail(arcpy.env.scratchFolder)
except Exception as e:
errorLog("Error getting thumbnail", e)
return
try:
metadata_file = item.download_metadata(arcpy.env.scratchFolder)
except Exception as e:
errorLog("Error getting metadata", e)
return
# Add File Geodatabase to portal
try:
print("Adding {0} File Geodatabase to target portal".format(item.title))
if targetFolder == 'ROOT':
fgd = target.content.add(item_properties=item_properties, owner=targetOwner, data=download_result, thumbnail=thumbnail_file, metadata=metadata_file)
else:
fgd = target.content.add(item_properties=item_properties, owner=targetOwner, folder=targetFolder, data=download_result, thumbnail=thumbnail_file, metadata=metadata_file)
except Exception as e:
errorLog("Error adding {0}".format(item.title), e)
return
# Publish File Geodatabase
try:
print("Publishing {0} File Geodatabase".format(item.title))
published_service = fgd.publish()
except Exception as e:
errorLog("Error publishing {0} File Geodatabase".format(item.title), e)
fgd.delete()
return
# Delete File Geodatabase from target portal and scratch folder
try:
print("Deleting {0} File Geodatabase in target portal".format(item.title))
fgd.delete()
except Exception as e:
errorLog("Error deleting {0} File Geodatabase in target portal".format(item.title), e)
return
# Main
source_users = source.users.search('!esri_ & !system_publisher', max_users=10000)
for user in source_users:
if user.user_types()['name'] not in ('Viewer', 'Editor', 'Field Worker'):
if user.username == 'jskinner_CountySandbox':
print("----------------------------------")
print("User: {0}".format(user.username))
user_content = user.items()
folders = user.folders
for folder in folders:
for item in user.items(folder=folder['title']):
user_content.append(item)
for item in user_content:
if 'Hosted Service' in item.typeKeywords:
if 'Map Service' not in item.typeKeywords and 'View Service' not in item.typeKeywords and 'Scene Service' not in item.typeKeywords:
copyContent(item)
# Clean up downloaded File Geodatabases
print("Cleaning up Scrath Workspace")
for (path, dirs, files) in os.walk(arcpy.env.scratchFolder):
for file in files:
os.remove(os.path.join(path, file))
endTime = time.clock()
elapsedTime = round((endTime - startTime) / 60, 2)
print("Script finished in {0} minutes".format(elapsedTime))
Thanks Jake Skinner. I will test this out!
Ideally instead of user, I would prefer to pass item ID's. But there is enough there for me to just change username to item ID.
Jake Skinner
Hi, the code looks very promising for some of my current work. I've had to hardcode the parameters again as I can't see any script tools in the zipped toolbox, so i'm struggling with some of the parameters and my trial-runs don't do anything... Has anyone got a sample of the parameters that have worked for them please?
Many thanks for creating this.
David Pike what version of ArcGIS Pro are you using? In the zip file there is a Copy Content.tbx file that will show up as a toolbox in ArcGIS Pro.
I'm using ArcMap, that may be the issue!
I tried to get it to load in my ArcGIS Pro 2.6.2 but I am not seeing it under tool boxes?
What about transferring 'Site Application' and 'Site Page' types?
Hello @JakeSkinner ,
I'm looking to use your script to move web maps and apps from one portal (10.6.1) to another(10.8.1) both federated.
Both portals have windows authentication enabled. I have tried the clone portal content script, but it only references the services used in the first portal. We plan to publish all the services to the new underlying hosting server but want to move the web maps and apps. When I loaded this script on the ArcGISPro (2.7). the source portal parameter gives me an error parameter invalid and I have entered the portal address as https:// myportal.com/webadaptor name. IS there something that I need to change in the script to get it to work?
@BPriyaKcan you send a screen shot of how you have the tool setup and the error message? Since you have Windows Authentication enabled, you will need to specify an active directory account for the Source/Target Admin Username.
Hello @JakeSkinner ,
I'm using the python script posted in this thread last August above to copy hosted feature services from one portal to another. The copyContent(item) routine fails on the item.export statement.
Exception: Could not export item: 378ccb291e7b4e8f89d444d0fbc8d691
The weird thing is, when I go to my source portal account, the file geodatabase item is there. Do you have any suggestions as to why I get an error before I can download it? Using Pycharm with Python 3.6 that comes with Pro.
@ToddMcNeilinstead of using the script, have you tried executing the GP tool attached with this document?
@JakeSkinner I get an error when I put in my source portal URL. ERROR 032659 updateParameters. Looks like a certificate error....
This is why I was using the "verify_cert=False" as a parameter in the code.
# Create GIS objects for source and target portals
source = GIS(sourcePortal, source_username, source_password, verify_cert=False)
target = GIS(targetPortal, target_username, target_password, verify_cert=False)
@ToddMcNeilI updated the validation to include verify_cert=False. Try re-downloading the tool and see if you get the same error.
@JakeSkinner I downloaded it and did not see the change. I added it in to all of the scripts and things started happening... I copied a hosted feature service, its service definition file and an excel document successfully.
Adding some notes for the CopyContent.py script that gives a walk-through of the logic of this code:
define global variables:
for source and target portals,
the content the user wants to process as a semicolon-separated string is what is meant by AGP [string] multiple values
# '"item title - item type";"item title - item type"' aka
content = '"u - Feature Service (Hosted)";"v - Feature Service (Hosted)"'
targetOwner string
targetFolder string 'ROOT' or other
Enter main:
define target portal token
define fqdn for later use in rest end point post requests to get and update target web map json data
# Create dictionary for old and new layer Ids:
oldIDnewIDdict = {}
get all of the source user's content in all folders by iterating and appending to an empty list, user_content
split apart the content to get the input content title, as contentData
if str(item.title) == contentData and 'Service' in str(item.typeKeywords):
# removes duplicate items to process from duplicate contentData item titles
if item not in processItemList:
processItemList.append(item)
processItem(item)
Enter processItem function:
# Get the source portal Groups the item is shared with
ie.:
groupDict = {'Custom Symbols': 'cc5734aba54b4fe08a99a561fb22b072', 'everyone': False, 'org': True}
if 'Hosted Service':
start processing the source portal item
Check if the source item exists in Target Portal in a try except block...if exists, delete it
Clone Item depending on the folder specified
# Share Item
if item.access == 'shared' or item.access == 'public' or item.access == 'org':
shareItemWithGroup(item, targetOwner, groupDict)
Since the clone already occured, search the source item in the target portal, and store the first returned item in the targetItem variable.
iterate over groupDict dictionary using a try/except block that checks:
the iterable variable (group title) as part of the search query for the target portal's groups, and store the first returned group in the target_group variable.
use the .share method of the target portal's item to share with the target_group and everyone, or org
# Get new ID of Item and update all web maps that referenced previous ID
build the oldIDnewIDdict source portal item as key, and the cloned destination portal item (newTargetItem.id) as value.
search the target portal's users
If user type == level 2 or creator,
iterate each user.items()
if item.type == 'Web Map',
run the checkWebMaps() function.
repeat for every user.folders and for every item in user.items(folder=folder['title'])
checkWebMaps():
# If a target Web Map's dependent layer id's are == to the old Item ID key (layerID) from the oldIDnewIDdict dictionary,
run the updateWebMaps() function
1
# get the target web map json (similar to item.get_data()):
make a post request (response) to the ArcGIS Rest API (sharing/rest/content/items/{1}/data) to get the target web map json.
2
# prepare the target web map item id dictionary by overwriting the layer dictionary values with the desired values:
for every response["operationalLayers"]
if old Item ID key (layerID)
build an dictionary (dict) with an incrementing key and newTargetItem.id
else
increment key anyway
for every key in the built dict
reset the response item id with the new newTargetItem.id, as such: response["operationalLayers"][val]['itemId'] = dict[val]
3
# update the target web map (similar to item.update(item_properties=desired target web map dictionary)):
make a post request (r) to the ArcGIS Rest API (sharing/rest/content/users/{1}/items/{2}/update) to update the target web map using the updated response ({'text': json.dumps(response)}) with the newTargetItem.id
if r.status_code == 200
web map was updated with newTargetItem.id
The following piece of functionality was commented out in the original code.
Since the map and feature services are not inherently portal items, some of the workflows applied to the Hosted Service above are not applicable.
For example, the map/feature service might not have an item id.
As per the API guide:
https://developers.arcgis.com/python/guide/cloning-content/
clone_items() will not clone map services and image services.
Since these services can be published to servers other than the hosted server in a configuration,
it's impossible for the function to determine where to publish them in the target.
As a result, these items will copy over, but will continue to point back to the original source URL."
Elif 'Map Service' in str(item.typeKeywords) and 'MapServer' or 'FeatureServer' in str(item.url)
Run the copyMapFeatureServices() function, which uploads the maps/feature service as a gis portal item
Get the properties of the item (as a dictionary) from source and applies it as value to the target item_properties key
Get thumbnail and metadata
Add the service to the gis portal
Worked well for me...thank you.
From Portal 10.8 to 10.8.1
Spoke too soon. Errors with the Copy Hosted Services. Any trouble shooting options please. I ran it from ArcGIS Pro 2.7 via your scripts.
Is there a specifc location where the toolbox and associated python scripts need to be downloaded to on your computer in order for the toolbox to work as it is empty for me and co-worker?
Hi @MichaelVolz , make sure you are opening the toolbox in ArcGIS Pro. The tools were built with Python 3 and the ArcGIS API for Python, which are only supported in Pro.
Thanks for the tip Jake as I had went through all the Posts in the thread too quickly in my initial read when first trying to add the toolbox to ArcMap.
I tried copying a hosted feature layer between two 10.7.1 Portals, but the script threw the following error:
Failed to create feature service JSONObject["globalid"] not found Error code: 500.
Any idea what would cause this error with the script?
@MichaelVolzdo you receive this error for all hosted feature services, or is it just one in particular?
Jake:
It has occurred for every hosted feature layer that a co-worker and myself have tried.
The error is vague, so it's difficult to trouble shoot without taking a look at the environment. I would check to see if the Target environment is working as expected. A few things to verify:
1.) We are able to publish hosted feature services in the target environment.
2.) The target relational Data Store validates successfully in AGS Manager.
So the add globalid tool was run on 2 different feature classes and then they were published up as hosted feature layers. The smaller less complicated feature class with fewer fields was successfully copied, but the more complex feature class with more fields failed.
As far as you know from your testing, does a globalid need to be part of the feature class for the copy tool to work?
@MichaelVolzno, a globalid field is not required.
That's weird because the tool failed when the feature classes did not have a globalid field.
@JakeSkinner Good morning.
I am trying to use this tool and had to make a couple of changes so it wouldn't require usernames and passwords as I am on a PKI certificate enabled portal, so I made those optional in the tool rather than required.
I enter the Source Portal as our older (version 10.7) portal and the Target portal (version 10.8), enter myself as the Source User, leave the Content field blank, then add myself again as Target Owner and use the drop down that is then created to enter Target Folder (I assume it is reading everything fine since it recognizes the folders on my account). I run the tool and it says completed but nothing is copied over. I am using Copy Services. ArcGIS PRO version 2.6.1.
I did add the verify_cert=False to the end of source and target lines:
source = GIS(sourcePortal, sourcePortal, source_username, source_password, verify_cert=False)
target = GIS(targetPortal, sourcePortal, source_username, source_password, verify_cert=False)
I also tried running Copy File Based Items. It does the same thing, says completed, then nothing is copied over.
I am an administrator on both the old and new Portals.
Any ideas as to why this may be the case?
Thank you,
Kevin White
@JamesWhite5are you able to share what changes you made to the code?