Copy Content Between Portals

16228
54
08-23-2019 07:50 AM

Copy Content Between Portals

The attached toolbox contains 3 tools for copying content between Portals. 

 

  • Copy Services
    • copies hosted feature services from the Source Portal to the Target Portal
    • if the hosted feature service exists in the Target Portal, it will be overwritten
      • the tool will then iterate through every user's web maps to see if the original item ID of the feature service exists.  If it does, it will update the web map with the new feature service item ID
  • Copy File Based Items
    • if the file based item already exists, it will be overwritten
    • the following file based items can be copied:

                  File Geodatabase,CSV, Image, KML, Locator Package, Map Document, Shapefile, Microsoft Word, PDF, Microsoft Powerpoint, Microsoft Excel, Layer Package,                   Mobile Map Package, Geoprocessing Package, Service Definition, Scene Package, Tile Package, Vector Tile Package

  • Copy Map and Applications
    • copies web maps, web applications, and dashboards from the Source Portal to the Target Portal
    • Web Maps
      • if the web map exists in the Target Portal, it will be overwritten
        • the tool will then iterate through every user's web applications/dashboards to see if the original item ID of the web map exists.  If it does, it will update the web application/dashboard with the new web map ID
      • when the web map is copied the tool will search for existing layers and update the web map to use these layers
        • if the layer is not found, and it's a hosted feature service, the hosted feature service will be created in the Target Portal
    • Web Applications
      • if the web application exists in the Target Portal, it will be overwritten as well as the underlying web map
        • the tool will then iterate through every user's web applications/dashboards to see if the original item ID of the web map exists.  If it does, it will update the web application/dashboard with the new web map ID
      • if the web application does not exist, it will copy the web application and web map
        • the tool will search for the web map's services in the Target Portal.  If they exist, it will update the web map with these existing services.  If they do not exist, and they are hosted feature services, it will create the hosted feature services
    • Dashboards
      • if the dashboard exists in the Target Portal, it will be overwritten
      • if the dashboard's underlying web map does not exist, it will be copied as well as any hosted feature services that do not exist

 

For the tool to successfully update the web map, web application, and dashboard IDs the client machine executing this tool must be able to reach the Target Portal's server where Portal for ArcGIS (not the Web Adaptor) is installed (i.e. https://portal.esri.com:7443/arcgis).  It uses this URL to make the post requests if Windows Authentication is enabled on Portal's web adaptor.

 

Each tool will share the content with Everyone, and the ArcGIS Enterprise Organization (if it is shared with these in the Source Portal).  It will also share to each Group in the Target Portal if it has the same name as the Source Portal Group. 

 

Note:  This tool is in it's early development and has not been extensively tested.  So far I've tested transferring content where both Portals are version 10.7.1.  Looking to have any issues or recommendations reported in the comments below.

Attachments
Comments

I can confirm that these tools work to copy Vector Tile Packages between Portal versions 10.6 to 10.7.1

Andres Castillo‌ just a note on the above web link, those scripts will not copy any of the services.  See the Conclusion section:

Note, this notebook did not copy over the services that power the service based items. Such items continue to point to the same URL as the ones in source portal did

Jake Skinner

For the Copy Map and Applications tool, when I attempt to specify mulitiple contents, the script tool loads endlessly.

Any suggestions?

I haven’t seen that issue before, sounds like something not right with the script. I suggest reaching out to Esri Support.

E.J. McNaughton

Andres Castillo What version of ArcGIS Pro are you running?

2.5.0

When I try to run outside of pro in vs code, I notice the token is not being stored:

Processing item: u
WARNING: ------------------

Error occurred processing item u
Error: Token Required
(Error Code: 499)
Deleting Connections

some modifications/observations I made to the script:

  •     except Excpetion as e: exception is misspelled, so it won't go into exception block if fqdn is wrong.
  •     switched all mentions of '':7443/arcgis' to '/webadaptor' since my machine did not have access to port.
  •     first mention of updateURL for AGO is missing https:// and a formatted string parameter.
My source portal is 10.7.1 and destination portal is 10.5.1

Andres Castillo‌ if Windows Authentication is not enabled on Portal's web adaptor, you will not have to update the :7443/arcgis to /webadaptor.  I just updated a note about this in the tool's description above.  Also, thank you for catching the misspelling of Exception.  I updated the code and re-uploaded.

I haven't tested going from 10.7.1 to 10.5.1.  Are you able to go from 10.5.1 to 10.7.1?  It could possibly be a version issue.

For the copy services script,

I can successfully copy over hosted feature layers from 10.7.1 to 10.5.1.

                # This seems to be the same value on either side, did you mean to say:
                # oldIDnewIDdict[item.id] = newTargetItem.id
                oldIDnewIDdict[targetItem.id] = newTargetItem.id

                        # NotImplementedError: `user_types` is not implemented at version 5.1.....makes sense, so goes to except block
                        if user.user_types()['name'not in ('Viewer''Editor''Field Worker':(
                        # AttributeError: 'User' object has no attribute 'level'.....don't know why this doesn't have level 2 attribute.
                        if user.level == '2':
curious why you commented out invoking the function 
copyMapFeatureServices(item)

Windows Authentication is enabled on the wa.

Are your comment and the document description above in alignment?

 

The token error I experienced above was due to the fact that I didn't explicitly store my credentials in the GIS() object.

I sometimes get away with not putting my credentials because I've noticed that in other cases, portal auto-reads them using Windows Authentication and my AD credentials.

Testing transfer on Pro 2.5.1 with both AGS Enterprise systems at 10.8.  So far, keep getting 'error occurred processing items' for all items.  I love the concept.  In the future, it might be good to have an 'all' where we can transfer all content from source regardless of user into one folder or the choice to copy content into default folder schema from source.  This will help in staging and transferring AWS containers as well as solution transfers from Dev, Test and Prod.

Jake Skinner This script seems like exactly what we need. I just tried a single web map test with Pro 2.5.1 with the origin portal at 10.6.1 and the destination portal at 10.8.

I also got errors similar to the guy above: "Error occurred processing item" <class 'keyerror'> 402

Any ideas on how to get this running correctly would be greatly appreciated!


verify you are an admin on the portals

verify you are an admin on the portals

Yes Admin and verified.

Evan Marshall‌ do you receive this error for all web maps, or just one in particular?  Are you able to copy feature services successfully?

Luke Savage‌ are you attempting to copy feature services, or web maps/apps?

Any content.

Hey Jake. I had not tried it with any services, as we are just looking to move some apps and maps over and will likely republish some services manually since we will be moving some of them to hosted layers in the portal. The map I tried was a simple web map with only our parcel layer in it.

Evan Marshall‌ how did you add the parcel layer to the web map?  If you add the layer with the URL (i.e. Add > Layer From Web), you will receive an error when trying to copy the web map.  Instead, add the parcel as an Item in Portal (i.e. Content tab > Add Item).  Then, add the parcel layer to the map by going to Add > Search for Layers.

By any chance, can this be used with copying over services from AGO to portal? I put in our AGO URL and it did not accept it.

I tend to use AGO Assist, but copying over feature services is unreliable. Please share if you know of other tools that can accomplish this.

Reza Tehranifar‌ I've had sporadic results going from AGOL to Portal.  I've written the below code to do this and it is much more reliable.  It was originally written to copy all services from all users.  You can specify an individual user to copy services from on line 117:

Removing this line will copy all services from all users.

import arcpy, requests, json, sys, os, time
from arcgis.gis import GIS
from arcgis.gis import User
arcpy.env.overwriteOutput = 1

startTime = time.clock()

# Variables
sourcePortal = 'https://www.arcgis.com'         # AGOL URL
source_username = 'jskinner_CountySandbox'      # AGOL Admin Username
source_password = '********'                    # AGOL Admin Password

targetPortal = 'https://jake2.esri.com/portal'  # Portal URL
target_username = 'jskinner@AVWORLD'            # Portal Admin Username
target_password = '*************'               # Portal Admin Password

targetOwner = 'jskinner@AVWORLD'                # Portal User to copy services to
targetFolder = 'Copy'                           # Portal User's folder to copy services to


# Create GIS objects for source and target portals
source = GIS(sourcePortal, source_username, source_password)
target = GIS(targetPortal, target_username, target_password)

export_folder = arcpy.env.scratchFolder

# Function to log errors
def errorLog(msg, e):
    print(msg)
    print("Error: {0}".format(e))

# Function to Copy Hosted Feature Services
def copyContent(item):
    # Export hosted feature service to FGD, and downlaod to scracth folder
    print("Processing {0}".format(item.title))
    export_name = "{0}".format(item.title)

    try:
        print("Exporting {0}".format(item.title))
        result_item = item.export(export_name, 'File Geodatabase', wait=True)
    except Exception as e:
        errorLog("Error exporting {0} File Geodatabase".format(item.title), e)
        return

    try:
        print("Saving File Geodatabase to: {}".format(export_folder))
        download_result = result_item.download(export_folder)
    except Exception as e:
        errorLog("Error saving {0} File Geodatabase to {1}".format(item.title, export_folder), e)
        result_item.delete()
        return

    try:
        print("Deleted {0} File Geodatabase from source portal".format(item.title))
        result_item.delete()
    except Exception as e:
        errorLog("Error deleting {0} File Geodatabase from source portal".format(item.title), e)
        return

    # Set Item Properties
    item_properties = {
        'title':item.title,
        'type':'File Geodatabase',
        'description':item.description,
        'snippet': item.snippet,
        'tags':item.tags,
        'extent': item.extent,
        'accessInformation': item.accessInformation,
        'licenseInfo': item.licenseInfo
    }

    # Get thumbnail and metadata
    try:
        thumbnail_file = item.download_thumbnail(arcpy.env.scratchFolder)
    except Exception as e:
        errorLog("Error getting thumbnail", e)
        return
    try:
        metadata_file = item.download_metadata(arcpy.env.scratchFolder)
    except Exception as e:
        errorLog("Error getting metadata", e)
        return

    # Add File Geodatabase to portal
    try:
        print("Adding {0} File Geodatabase to target portal".format(item.title))
        if targetFolder == 'ROOT':
            fgd = target.content.add(item_properties=item_properties, owner=targetOwner, data=download_result, thumbnail=thumbnail_file, metadata=metadata_file)
        else:
            fgd = target.content.add(item_properties=item_properties, owner=targetOwner, folder=targetFolder, data=download_result, thumbnail=thumbnail_file, metadata=metadata_file)
    except Exception as e:
        errorLog("Error adding {0}".format(item.title), e)
        return

    # Publish File Geodatabase
    try:
        print("Publishing {0} File Geodatabase".format(item.title))
        published_service = fgd.publish()
    except Exception as e:
        errorLog("Error publishing {0} File Geodatabase".format(item.title), e)
        fgd.delete()
        return

    # Delete File Geodatabase from target portal and scratch folder
    try:
        print("Deleting {0} File Geodatabase in target portal".format(item.title))
        fgd.delete()
    except Exception as e:
        errorLog("Error deleting {0} File Geodatabase in target portal".format(item.title), e)
        return


# Main
source_users = source.users.search('!esri_ & !system_publisher', max_users=10000)
for user in source_users:
    if user.user_types()['name'] not in ('Viewer', 'Editor', 'Field Worker'):
        if user.username == 'jskinner_CountySandbox':
            print("----------------------------------")
            print("User:  {0}".format(user.username))
            user_content = user.items()
            folders = user.folders
            for folder in folders:
                for item in user.items(folder=folder['title']):
                    user_content.append(item)
            for item in user_content:
                if 'Hosted Service' in item.typeKeywords:
                   if 'Map Service' not in item.typeKeywords and 'View Service' not in item.typeKeywords and 'Scene Service' not in item.typeKeywords:
                      copyContent(item)

# Clean up downloaded File Geodatabases
print("Cleaning up Scrath Workspace")
for (path, dirs, files) in os.walk(arcpy.env.scratchFolder):
    for file in files:
        os.remove(os.path.join(path, file))

endTime = time.clock()
elapsedTime = round((endTime - startTime) / 60, 2)
print("Script finished in {0} minutes".format(elapsedTime))

Thanks Jake Skinner‌. I will test this out!

Ideally instead of user, I would prefer to pass item ID's. But there is enough there for me to just change username to item ID.

Jake Skinner

Hi, the code looks very promising for some of my current work.  I've had to hardcode the parameters again as I can't see any script tools in the zipped toolbox, so i'm struggling with some of the parameters and my trial-runs don't do anything... Has anyone got a sample of the parameters that have worked for them please?

Many thanks for creating this.

David Pike‌ what version of ArcGIS Pro are you using?  In the zip file there is a Copy Content.tbx file that will show up as a toolbox in ArcGIS Pro.

I'm using ArcMap, that may be the issue!

I tried to get it to load in my ArcGIS Pro 2.6.2 but I am not seeing it under tool boxes?

 

What about transferring 'Site Application' and 'Site Page' types?

Hello @JakeSkinner ,

I'm looking to use your script to move web maps and apps from one portal (10.6.1) to another(10.8.1) both federated.

Both portals have windows authentication enabled. I have tried the clone portal content script, but it only references the services used in the first portal. We plan to publish all the services to the new underlying hosting server but want to move the web maps and apps. When I loaded this script on the ArcGISPro (2.7). the source portal parameter gives me an error parameter invalid and I have entered the portal address as https:// myportal.com/webadaptor name. IS there something that I need to change in the script to get it to work?

@BPriyaKcan you  send a screen shot of how you have the tool setup and the error message?  Since you have Windows Authentication enabled, you will need to specify an active directory account for the Source/Target Admin Username.

Hello @JakeSkinner ,

I'm using the python script posted in this thread last August above to copy hosted feature services from one portal to another.  The copyContent(item) routine fails on the item.export statement.

Exception: Could not export item: 378ccb291e7b4e8f89d444d0fbc8d691

The weird thing is, when I go to my source portal account, the file geodatabase item is there.  Do you have any suggestions as to why I get an error before I can download it?  Using Pycharm with Python 3.6 that comes with Pro.

@ToddMcNeilinstead of using the script, have you tried executing the GP tool attached with this document?

@JakeSkinner I get an error when I put in my source portal URL.  ERROR 032659 updateParameters.  Looks like a certificate error.... 

This is why I was using the "verify_cert=False" as a parameter in the code.

# Create GIS objects for source and target portals
source = GIS(sourcePortal, source_username, source_password, verify_cert=False)
target = GIS(targetPortal, target_username, target_password, verify_cert=False)

 

@ToddMcNeilI updated the validation to include verify_cert=False.  Try re-downloading the tool and see if you get the same error.

@JakeSkinner I downloaded it and did not see the change.  I added it in to all of the scripts and things started happening...  I copied a hosted feature service, its service definition file and an excel document successfully. 

Adding some notes for the CopyContent.py script that gives a walk-through of the logic of this code:

 

define global variables: 
for source and target portals, 
the content the user wants to process as a semicolon-separated string is what is meant by AGP [string] multiple values
# '"item title - item type";"item title - item type"' aka 
content = '"u - Feature Service (Hosted)";"v - Feature Service (Hosted)"' 
targetOwner string
targetFolder string 'ROOT' or other



Enter main:
    define target portal token
    define fqdn for later use in rest end point post requests to get and update target web map json data

    # Create dictionary for old and new layer Ids:
    oldIDnewIDdict = {}

    get all of the source user's content in all folders by iterating and appending to an empty list, user_content
    split apart the content to get the input content title, as contentData
    if str(item.title) == contentData and 'Service' in str(item.typeKeywords):
        # removes duplicate items to process from duplicate contentData item titles 
        if item not in processItemList:  
            processItemList.append(item)                                                          
            processItem(item)    




Enter processItem function:
    # Get the source portal Groups the item is shared with
    ie.: 
    groupDict = {'Custom Symbols': 'cc5734aba54b4fe08a99a561fb22b072', 'everyone': False, 'org': True}

    if 'Hosted Service':
        start processing the source portal item
        Check if the source item exists in Target Portal in a try except block...if exists, delete it
        Clone Item depending on the folder specified

        # Share Item
        if item.access == 'shared' or item.access == 'public' or item.access == 'org':
            shareItemWithGroup(item, targetOwner, groupDict)
                Since the clone already occured, search the source item in the target portal, and store the first returned item in the targetItem variable.
                iterate over groupDict dictionary using a try/except block that checks:
                the iterable variable (group title) as part of the search query for the target portal's groups, and store the first returned group in the target_group variable.
                use the .share method of the target portal's item to share with the target_group and everyone, or org

        # Get new ID of Item and update all web maps that referenced previous ID
        build the oldIDnewIDdict source portal item as key, and the cloned destination portal item (newTargetItem.id) as value.
        search the target portal's users
        If user type == level 2 or creator, 
            iterate each user.items()
                if item.type == 'Web Map', 
                    run the checkWebMaps() function.
            repeat for every user.folders and for every item in user.items(folder=folder['title'])
            checkWebMaps():
                # If a target Web Map's dependent layer id's are == to the old Item ID key (layerID) from the oldIDnewIDdict dictionary,
                    run the updateWebMaps() function
                        1
                        # get the target web map json (similar to item.get_data()):
                        make a post request (response) to the ArcGIS Rest API (sharing/rest/content/items/{1}/data) to get the target web map json.
                        2
                        # prepare the target web map item id dictionary by overwriting the layer dictionary values with the desired values:
                        for every response["operationalLayers"] 
                            if old Item ID key (layerID) 
                                build an dictionary (dict) with an incrementing key and newTargetItem.id
                            else
                                increment key anyway
                        for every key in the built dict
                            reset the response item id with the new newTargetItem.id, as such: response["operationalLayers"][val]['itemId'] = dict[val]
                        3
                        # update the target web map (similar to item.update(item_properties=desired target web map dictionary)):
                        make a post request (r) to the ArcGIS Rest API (sharing/rest/content/users/{1}/items/{2}/update) to update the target web map using the updated response ({'text': json.dumps(response)}) with the newTargetItem.id
                        if r.status_code == 200
                            web map was updated with newTargetItem.id




    
    The following piece of functionality was commented out in the original code.
    Since the map and feature services are not inherently portal items, some of the workflows applied to the Hosted Service above are not applicable.
    For example, the map/feature service might not have an item id.
    As per the API guide:
    https://developers.arcgis.com/python/guide/cloning-content/
    clone_items() will not clone map services and image services. 
    Since these services can be published to servers other than the hosted server in a configuration, 
    it's impossible for the function to determine where to publish them in the target. 
    As a result, these items will copy over, but will continue to point back to the original source URL."
    
    Elif  'Map Service' in str(item.typeKeywords) and 'MapServer' or 'FeatureServer' in str(item.url)
        Run the copyMapFeatureServices() function, which uploads the maps/feature service as a gis portal item
            Get the properties of the item (as a dictionary) from source and applies it as value to the target item_properties key
            Get thumbnail and metadata
            Add the service to the gis portal

Worked well for me...thank you.

From Portal 10.8 to 10.8.1

LaurentP_1-1618470076946.png

 

 

Spoke too soon. Errors with the Copy Hosted Services. Any trouble shooting options please. I ran it from ArcGIS Pro 2.7 via your scripts.

LaurentP_0-1618471920178.png

 

@LaurentP,

  1. Is this happening for all hosted feature services, or only a few?
  2. Are you able to manuallypublish hosted feature services in the target environment?
  3. Is the target relational Data Store validating successfully in ArcGIS Server Manager?

Is there a specifc location where the toolbox and associated python scripts need to be downloaded to on your computer in order for the toolbox to work as it is empty for me and co-worker?

Hi @MichaelVolz , make sure you are opening the toolbox in ArcGIS Pro.  The tools were built with Python 3 and the ArcGIS API for Python, which are only supported in Pro.

Thanks for the tip Jake as I had went through all the Posts in the thread too quickly in my initial read when first trying to add the toolbox to ArcMap.

I tried copying a hosted feature layer between two 10.7.1 Portals, but the script threw the following error:

Failed to create feature service JSONObject["globalid"] not found  Error code: 500.

Any idea what would cause this error with the script?

@MichaelVolzdo you receive this error for all hosted feature services, or is it just one in particular? 

Jake:

It has occurred for every hosted feature layer that a co-worker and myself have tried.

@MichaelVolz 

The error is vague, so it's difficult to trouble shoot without taking a look at the environment.  I would check to see if the Target environment is working as expected.  A few things to verify:

 

  1. Are you able to manually publish hosted feature services in the target environment?
  2. Is the target relational Data Store validating successfully in ArcGIS Server Manager?

 

1.) We are able to publish hosted feature services in the target environment.

2.) The target relational Data Store validates successfully in AGS Manager.

So the add globalid tool was run on 2 different feature classes and then they were published up as hosted feature layers.  The smaller less complicated feature class with fewer fields was successfully copied, but the more complex feature class with more fields failed.

As far as you know from your testing, does a globalid need to be part of the feature class for the copy tool to work?

@MichaelVolzno, a globalid field is not required. 

That's weird because the tool failed when the feature classes did not have a globalid field.

@JakeSkinner Good morning.  

I am trying to use this tool and had to make a couple of changes so it wouldn't require usernames and passwords as I am on a PKI certificate enabled portal, so I made those optional in the tool rather than required.

 

JamesWhite5_0-1622723957144.png

 

I enter the Source Portal as our older (version 10.7) portal and the Target portal (version 10.8), enter myself as the Source User, leave the Content field blank, then add myself again as Target Owner and use the drop down that is then created to enter Target Folder (I assume it is reading everything fine since it recognizes the folders on my account).  I run the tool and it says completed but nothing is copied over.  I am using Copy Services.  ArcGIS PRO version 2.6.1.

I did add the verify_cert=False to the end of source and target lines:

source = GIS(sourcePortal, sourcePortal, source_username, source_password, verify_cert=False)
target = GIS(targetPortal, sourcePortal, source_username, source_password, verify_cert=False)

I also tried running Copy File Based Items.  It does the same thing, says completed, then nothing is copied over.

I am an administrator on both the old and new Portals.

 

Any ideas as to why this may be the case?

 

Thank you,

Kevin White

@JamesWhite5are you able to share what changes you made to the code?

Version history
Last update:
‎06-09-2021 06:08 AM
Updated by:
Contributors