Copy Content Between Portals

1928
28
08-23-2019 07:50 AM

Copy Content Between Portals

The attached toolbox contains 3 tools for copying content between Portals. 

  • Copy Services
    • copies hosted feature services from the Source Portal to the Target Portal
    • if the hosted feature service exists in the Target Portal, it will be overwritten
      • the tool will then iterate through every user's web maps to see if the original item ID of the feature service exists.  If it does, it will update the web map with the new feature service item ID
  • Copy File Based Items
    • if the file based item already exists, it will be overwritten
    • the following file based items can be copied:

                  File Geodatabase,CSV, Image, KML, Locator Package, Map Document, Shapefile, Microsoft Word, PDF, Microsoft Powerpoint, Microsoft Excel, Layer Package,                   Mobile Map Package, Geoprocessing Package, Service Definition, Scene Package, Tile Package, Vector Tile Package

  • Copy Map and Applications
    • copies web maps, web applications, and dashboards from the Source Portal to the Target Portal
    • Web Maps
      • if the web map exists in the Target Portal, it will be overwritten
        • the tool will then iterate through every user's web applications/dashboards to see if the original item ID of the web map exists.  If it does, it will update the web application/dashboard with the new web map ID
      • when the web map is copied the tool will search for existing layers and update the web map to use these layers
        • if the layer is not found, and it's a hosted feature service, the hosted feature service will be created in the Target Portal
    • Web Applications
      • if the web application exists in the Target Portal, it will be overwritten as well as the underlying web map
        • the tool will then iterate through every user's web applications/dashboards to see if the original item ID of the web map exists.  If it does, it will update the web application/dashboard with the new web map ID
      • if the web application does not exist, it will copy the web application and web map
        • the tool will search for the web map's services in the Target Portal.  If they exist, it will update the web map with these existing services.  If they do not exist, and they are hosted feature services, it will create the hosted feature services
    • Dashboards
      • if the dashboard exists in the Target Portal, it will be overwritten
      • if the dashboard's underlying web map does not exist, it will be copied as well as any hosted feature services that do not exist

For the tool to successfully update the web map, web application, and dashboard IDs the client machine executing this tool must be able to reach the Target Portal's server where Portal for ArcGIS (not the Web Adaptor) is installed (i.e. https://portal.esri.com:7443/arcgis).  It uses this URL to make the post requests if Windows Authentication is enabled on Portal's web adaptor.

Each tool will share the content with Everyone, and the ArcGIS Enterprise Organization (if it is shared with these in the Source Portal).  It will also share to each Group in the Target Portal if it has the same name as the Source Portal Group. 

Note:  This tool is in it's early development and has not been extensively tested.  So far I've tested transferring content where both Portals are version 10.7.1.  Looking to have any issues or recommendations reported in the comments below.

Attachments
Comments

I can confirm that these tools work to copy Vector Tile Packages between Portal versions 10.6 to 10.7.1

Andres Castillo‌ just a note on the above web link, those scripts will not copy any of the services.  See the Conclusion section:

Note, this notebook did not copy over the services that power the service based items. Such items continue to point to the same URL as the ones in source portal did

Jake Skinner

For the Copy Map and Applications tool, when I attempt to specify mulitiple contents, the script tool loads endlessly.

Any suggestions?

I haven’t seen that issue before, sounds like something not right with the script. I suggest reaching out to Esri Support.

E.J. McNaughton

Andres Castillo What version of ArcGIS Pro are you running?

2.5.0

When I try to run outside of pro in vs code, I notice the token is not being stored:

Processing item: u
WARNING: ------------------

Error occurred processing item u
Error: Token Required
(Error Code: 499)
Deleting Connections

some modifications/observations I made to the script:

  •     except Excpetion as e: exception is misspelled, so it won't go into exception block if fqdn is wrong.
  •     switched all mentions of '':7443/arcgis' to '/webadaptor' since my machine did not have access to port.
  •     first mention of updateURL for AGO is missing https:// and a formatted string parameter.
My source portal is 10.7.1 and destination portal is 10.5.1

Andres Castillo‌ if Windows Authentication is not enabled on Portal's web adaptor, you will not have to update the :7443/arcgis to /webadaptor.  I just updated a note about this in the tool's description above.  Also, thank you for catching the misspelling of Exception.  I updated the code and re-uploaded.

I haven't tested going from 10.7.1 to 10.5.1.  Are you able to go from 10.5.1 to 10.7.1?  It could possibly be a version issue.

For the copy services script,

I can successfully copy over hosted feature layers from 10.7.1 to 10.5.1.

                # This seems to be the same value on either side, did you mean to say:
                # oldIDnewIDdict[item.id] = newTargetItem.id
                oldIDnewIDdict[targetItem.id] = newTargetItem.id

                        # NotImplementedError: `user_types` is not implemented at version 5.1.....makes sense, so goes to except block
                        if user.user_types()['name'not in ('Viewer''Editor''Field Worker'😞
                        # AttributeError: 'User' object has no attribute 'level'.....don't know why this doesn't have level 2 attribute.
                        if user.level == '2':
curious why you commented out invoking the function 
copyMapFeatureServices(item)

Windows Authentication is enabled on the wa.

Are your comment and the document description above in alignment?

 

The token error I experienced above was due to the fact that I didn't explicitly store my credentials in the GIS() object.

I sometimes get away with not putting my credentials because I've noticed that in other cases, portal auto-reads them using Windows Authentication and my AD credentials.

Testing transfer on Pro 2.5.1 with both AGS Enterprise systems at 10.8.  So far, keep getting 'error occurred processing items' for all items.  I love the concept.  In the future, it might be good to have an 'all' where we can transfer all content from source regardless of user into one folder or the choice to copy content into default folder schema from source.  This will help in staging and transferring AWS containers as well as solution transfers from Dev, Test and Prod.

Jake Skinner This script seems like exactly what we need. I just tried a single web map test with Pro 2.5.1 with the origin portal at 10.6.1 and the destination portal at 10.8.

I also got errors similar to the guy above: "Error occurred processing item" <class 'keyerror'> 402

Any ideas on how to get this running correctly would be greatly appreciated!


verify you are an admin on the portals

verify you are an admin on the portals

Yes Admin and verified.

Evan Marshall‌ do you receive this error for all web maps, or just one in particular?  Are you able to copy feature services successfully?

Luke Savage‌ are you attempting to copy feature services, or web maps/apps?

Any content.

Hey Jake. I had not tried it with any services, as we are just looking to move some apps and maps over and will likely republish some services manually since we will be moving some of them to hosted layers in the portal. The map I tried was a simple web map with only our parcel layer in it.

Evan Marshall‌ how did you add the parcel layer to the web map?  If you add the layer with the URL (i.e. Add > Layer From Web), you will receive an error when trying to copy the web map.  Instead, add the parcel as an Item in Portal (i.e. Content tab > Add Item).  Then, add the parcel layer to the map by going to Add > Search for Layers.

By any chance, can this be used with copying over services from AGO to portal? I put in our AGO URL and it did not accept it.

I tend to use AGO Assist, but copying over feature services is unreliable. Please share if you know of other tools that can accomplish this.

Reza Tehranifar‌ I've had sporadic results going from AGOL to Portal.  I've written the below code to do this and it is much more reliable.  It was originally written to copy all services from all users.  You can specify an individual user to copy services from on line 117:

Removing this line will copy all services from all users.

import arcpy, requests, json, sys, os, time
from arcgis.gis import GIS
from arcgis.gis import User
arcpy.env.overwriteOutput = 1

startTime = time.clock()

# Variables
sourcePortal = 'https://www.arcgis.com'         # AGOL URL
source_username = 'jskinner_CountySandbox'      # AGOL Admin Username
source_password = '********'                    # AGOL Admin Password

targetPortal = 'https://jake2.esri.com/portal'  # Portal URL
target_username = 'jskinner@AVWORLD'            # Portal Admin Username
target_password = '*************'               # Portal Admin Password

targetOwner = 'jskinner@AVWORLD'                # Portal User to copy services to
targetFolder = 'Copy'                           # Portal User's folder to copy services to


# Create GIS objects for source and target portals
source = GIS(sourcePortal, source_username, source_password)
target = GIS(targetPortal, target_username, target_password)

export_folder = arcpy.env.scratchFolder

# Function to log errors
def errorLog(msg, e):
    print(msg)
    print("Error: {0}".format(e))

# Function to Copy Hosted Feature Services
def copyContent(item):
    # Export hosted feature service to FGD, and downlaod to scracth folder
    print("Processing {0}".format(item.title))
    export_name = "{0}".format(item.title)

    try:
        print("Exporting {0}".format(item.title))
        result_item = item.export(export_name, 'File Geodatabase', wait=True)
    except Exception as e:
        errorLog("Error exporting {0} File Geodatabase".format(item.title), e)
        return

    try:
        print("Saving File Geodatabase to: {}".format(export_folder))
        download_result = result_item.download(export_folder)
    except Exception as e:
        errorLog("Error saving {0} File Geodatabase to {1}".format(item.title, export_folder), e)
        result_item.delete()
        return

    try:
        print("Deleted {0} File Geodatabase from source portal".format(item.title))
        result_item.delete()
    except Exception as e:
        errorLog("Error deleting {0} File Geodatabase from source portal".format(item.title), e)
        return

    # Set Item Properties
    item_properties = {
        'title':item.title,
        'type':'File Geodatabase',
        'description':item.description,
        'snippet': item.snippet,
        'tags':item.tags,
        'extent': item.extent,
        'accessInformation': item.accessInformation,
        'licenseInfo': item.licenseInfo
    }

    # Get thumbnail and metadata
    try:
        thumbnail_file = item.download_thumbnail(arcpy.env.scratchFolder)
    except Exception as e:
        errorLog("Error getting thumbnail", e)
        return
    try:
        metadata_file = item.download_metadata(arcpy.env.scratchFolder)
    except Exception as e:
        errorLog("Error getting metadata", e)
        return

    # Add File Geodatabase to portal
    try:
        print("Adding {0} File Geodatabase to target portal".format(item.title))
        if targetFolder == 'ROOT':
            fgd = target.content.add(item_properties=item_properties, owner=targetOwner, data=download_result, thumbnail=thumbnail_file, metadata=metadata_file)
        else:
            fgd = target.content.add(item_properties=item_properties, owner=targetOwner, folder=targetFolder, data=download_result, thumbnail=thumbnail_file, metadata=metadata_file)
    except Exception as e:
        errorLog("Error adding {0}".format(item.title), e)
        return

    # Publish File Geodatabase
    try:
        print("Publishing {0} File Geodatabase".format(item.title))
        published_service = fgd.publish()
    except Exception as e:
        errorLog("Error publishing {0} File Geodatabase".format(item.title), e)
        fgd.delete()
        return

    # Delete File Geodatabase from target portal and scratch folder
    try:
        print("Deleting {0} File Geodatabase in target portal".format(item.title))
        fgd.delete()
    except Exception as e:
        errorLog("Error deleting {0} File Geodatabase in target portal".format(item.title), e)
        return


# Main
source_users = source.users.search('!esri_ & !system_publisher', max_users=10000)
for user in source_users:
    if user.user_types()['name'] not in ('Viewer', 'Editor', 'Field Worker'):
        if user.username == 'jskinner_CountySandbox':
            print("----------------------------------")
            print("User:  {0}".format(user.username))
            user_content = user.items()
            folders = user.folders
            for folder in folders:
                for item in user.items(folder=folder['title']):
                    user_content.append(item)
            for item in user_content:
                if 'Hosted Service' in item.typeKeywords:
                   if 'Map Service' not in item.typeKeywords and 'View Service' not in item.typeKeywords and 'Scene Service' not in item.typeKeywords:
                      copyContent(item)

# Clean up downloaded File Geodatabases
print("Cleaning up Scrath Workspace")
for (path, dirs, files) in os.walk(arcpy.env.scratchFolder):
    for file in files:
        os.remove(os.path.join(path, file))

endTime = time.clock()
elapsedTime = round((endTime - startTime) / 60, 2)
print("Script finished in {0} minutes".format(elapsedTime))

Thanks Jake Skinner‌. I will test this out!

Ideally instead of user, I would prefer to pass item ID's. But there is enough there for me to just change username to item ID.

Jake Skinner

Hi, the code looks very promising for some of my current work.  I've had to hardcode the parameters again as I can't see any script tools in the zipped toolbox, so i'm struggling with some of the parameters and my trial-runs don't do anything... Has anyone got a sample of the parameters that have worked for them please?

Many thanks for creating this.

David Pike‌ what version of ArcGIS Pro are you using?  In the zip file there is a Copy Content.tbx file that will show up as a toolbox in ArcGIS Pro.

I'm using ArcMap, that may be the issue!

I tried to get it to load in my ArcGIS Pro 2.6.2 but I am not seeing it under tool boxes?

 

What about transferring 'Site Application' and 'Site Page' types?

Version history
Revision #:
1 of 1
Last update:
‎08-23-2019 07:50 AM
Updated by:
 
Contributors