Changing Layer Data Sources from Map to Map with ArcPy

[Oliver_Sandoval_e]

Hi everyone,

Im trying to update the layer data sources in a map with the layer data sources in an another map based off name, both maps in the same project (SDE connection, two different databases). Ive been using the jupyter notebook, and have been successful in comparing the layers list to find matches, however when it comes to using the updateConnectionPropertiesfunction in an IDE nothing happens, original project as well as project copy have no changed data sources. 

I have tried multiple parameter formats, calling the lyr.connectionProperties as well as creating file paths.

import arcpy

# find original layers in old map service map
aprx = arcpy.mp.ArcGISProject("CURRENT")
m = aprx.listMaps("New Service")[0];
original_layers = m.listLayers()
print(str(len(original_layers)))

# find new layers in new map service map
m2 = aprx.listMaps("New Layer Data Sources")[0];
new_layers = m2.listLayers()

# create empty list to view names that match
success_match = list()

# create empty list to view names that don't match
fail_match = list()

# loop through layers in old map service
for lyr in original_layers:

    # if not a group layer then find would be new name
    if not lyr.isGroupLayer:
        old_name = lyr.dataSource
        old_name_split = old_name.split("__")
        new_name1 = old_name_split[1]
        print(new_name1)

        # create variable for old connection properties
        old_desc = arcpy.Describe(lyr)
        old_data_path = old_desc.path
        old_data_source = "r" + "'" + str(old_data_path) + "/" + lyr.name + "'"

        # create for loop to loop through new names
        for lyr2 in new_layers:
            # isolate new layer name to match with original name
            full_new_name = lyr2.name
            full_new_name_split = full_new_name.split("_")
            new_name2 = full_new_name_split[1]

            # create variable for new connnection
            new_desc = arcpy.Describe(lyr2)
            new_data_path = new_desc.path
            new_data_source = "r" + "'" + str(new_data_path) + "/" + lyr2.name + "'"

            # if old layer new name is equal to new layer name, then chance data source
            if new_name1 == new_name2:
                success_match.append(new_name2)
               
                lyr.updateConnectionProperties(lyr.connectionProperties, lyr2.connectionProperties)
        # if old layer does not match new name layers
        if new_name1 not in success_match:
            fail_match.append(new_name1)

print(len(success_match))
print(len(fail_match))

Any advice would be greatly appreciated. Thank you!

[Mahdi_Ch]

Before getting to the real problem, some points are not clear to me.

For each layer in the first map you are going over all the layers in the second map. That doesn't seem to be efficient. You could extract list of names and do the comparison based on that. 

You are also creating some path and never use them. Also you already have these info and no need to re-create them: 

# you have these line:
# create variable for new connnection properties
new_desc = arcpy.Describe(lyr2)
new_data_path = new_desc.path # this basically gives you the path to the geodatabase/folder
new_data_source = "r" + "'" + str(new_data_path) + "/" + lyr2.name + "'"


# From what I undrestand this new_data_source that you created manually above, is the same as:
new_data_source = lyr2.dataSource

 Also, to concatenate strings in Python you can use f-string. That would be more readable and more efficient. This is not a good example (we will get to that) but if you wanted to do new_data_source using f-string it will be like this: 

# concat strings all together (returns str)
new_data_source = f'{new_data_path}/{lyr2.name}'
# you can mix str and int (returns str)
number = 1
new_text = f'{number}- My Layer name is: {lyr2.name}'
# it prints:  '1- My Layer name is: actual_name'

That would take care of  converting integers to str() and everything automatically. 

Adding parts of the path manually is not a good practice. It would be better to use os or  pathlib modules:

# os. path joins directories (folders) and file names into an os compatible path string
new_data_source = os.path.join(main_directory, new_data_path, lyr2.name)

Regarding the lyr.updateConnectionProperties() I am not quite sure about the reason, Can you print the dictionaries you are passing to see what it inside and if they look as they should? 

print(lyr.connectionProperties)

print(lyr2.connectionProperties)

May be that shouldn't contain the names of geodatabases since they are part of the same workspace?

[Oliver_Sandoval_e]

Hi Mahdi,

Thank you for your input, I will work on improvements and best practices! As you can tell i'm not that experienced in python lol.

The filepaths I included were to show the other ways I tried to update the connection properties as I stated in my post.

Both lists of layers are enterprise data, sde connections, from different databases and users/permissions. I will keep looking for reasons why the function isn't working.

[Mahdi_Ch]

Hi Oliver,

No worries! We are all learning and there is always room for improvement 🙂 

Thanks for the clarification, I got it now. 

Did you take a look at the layer.connectionProperties dicts for those two layers? It should be a dictionary with destination and source info (and more) but based on the type of the connection and whether you have joins or relates on the layers, the dictionary might be different. 

I think something inside those dictionaries is triggering this. updateConnectionProperties() works if it can validate the new connection data source. So it worth carefully comparing the content of those dictionaries.

Also, if dealing with long hierarchical dictionaries, instead of using print on those dicts (that I mentioned above), you can use pprint module which “pretty-prints”  the Python dictionary for you: 

import pprint 

# read layers and etc as you did in the original code

# then pprint the connection properties dicts
print('Old Layer:')
pprint.pprint(lyr.connectionProperties)

# '\n\n' shifts whatever comes after it, two lines down
print('\n\nNewLayer:') 
pprint.pprint(lyr2.connectionProperties)

Hope it helps. 

[Oliver_Sandoval_e]

Hey Mahdi!

Thanks for the help! I just did the pprint on the dictionaries and was able to notice that they don't have the same key value pairs!

The old data has user and password keys while the new doesn't.

Also the authentication value is DBMS for the old while the authentication for the new is OSA.

Would these difference be causing the issue with the function? Do I have to make the keys equal?

[Oliver_Sandoval_e · Accepted solution]

Found out that you cannot just call the lyr.connectionProperties of the target datasource feature class and use that for the new_connection_info parameter of updateConnectionProperties since if there is a user and password, the password is delivered as a bunch of ********. I didnt realize that was the actual value being returned. I recently had the need to use this function again, so I created a new better script that includes the actual user and pass for the new_connection_info dictionary for each layer in the Pro map.

# Script to change datasources of ArcGISPro Map
import arcpy
import pprint
arcpy.env.overwriteOutput = True


aprx = arcpy.mp.ArcGISProject("Current")
m = aprx.listMaps("Map")[0];
lyrs = m.listLayers()
print(str(len(lyrs)))

# loop through layers in map
for lyr in lyrs:
    
    # If layer is not group layer, change datasource
    if not lyr.isGroupLayer:
        
        # Create variable from current connectionProperties, essentially the full fc name
        connect_dict = lyr.connectionProperties
        # pprint.pprint(connect_dict)
        fc_name = connect_dict['dataset']
        print("Old GIS dataset: " + fc_name)

        # Create variable for new fc name
        new_fc_name = "DBname.DBOwner." + fc_name.split(".", 2)[2]
        print("New GIS dataset: " + new_fc_name)
        
        # Dictionary for new_connection_info
        replace_dict = {'connection_info':{#i left this blank bc is senstive info},
                            'dataset': new_fc_name,
                            'workspace_factory': 'SDE'}
        # Use arcpy m.updateConnectionProperties function
        m.updateConnectionProperties(lyr.connectionProperties,replace_dict)