Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins · ‎04-04-2024

Greetings,

I have successfully updated connection properties in all scenarios EXCEPT enterprise to enterprise (dev to production). The following code is my example. I have 2.9 (yes i know outdated) Python 1.9.1 ( again. outdated).

Is this a limitation of the updateConnectionProperties?

Is this related the attached screenshot, where the Update Data Sources is full of "Unsupported Items"

Feel like I am missing something obvious... ? help...

import arcpy,pprint
# ## SGTomlins  3/27/24
# ## Examples of workspaces that can be used.  Either a .gdb or a connection string.
# ## arcpy.env.workspace = r"C:\SGT\ESRI\ArcGIS_Pro\LoadLayerFiles\LoadLayerFiles.gdb"
# ## arcpy.env.workspace = r"C:\SGT\ESRI\_ConnectStrings\SDE@underdog-az@gis_prod_util_test.sde"
#

# ## find_dict = r'C:\SGT\SDE@underdog-az@gis_prod_util_test.sde.gdb'
# ## replace_dict = r'C:\Data\!Database_Connections\GISReader@underdog-az@gis_prod_util.sde'

find_dict = {'connection_info': {'user': 'sde',
                                 'password': 'guessmenow!',
                                 'version': 'sde.DEFAULT',
                                 'server': 'underdog-az',
                                 'instance': 'sde:sqlserver:underdog-az',
                                 'dbclient': 'sqlserver',
                                 'database' : 'gis_prod_util',
                                'db_connection_properties' : 'underdog-az'
                                 }}

replace_dict ={'connection_info': {'user': 'sde',
                                 'password': 'guessmelater',
                                 'version': 'sde.DEFAULT',
                                 'server': 'tinkerbell-az',
                                 'instance': 'sde:sqlserver:tinkerbell-az\TINKER2019',
                                 'dbclient': 'sqlserver',
                                 'database' : 'gis_prod_util',
                                 'db_connection_properties' : 'tinkerbell-az'
                                 }}



# ## sets some variables and the output path for the created .lyrx files
aprfilenew = r'c:\SGT\TINKER2019-ResourceNoUtil31.aprx'

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'c:\SGT\TINKER2019-Resource_NoUtil.aprx'
p = arcpy.mp.ArcGISProject(aprfilein)

# ## Create lists for "Maps" and "Layers" in the .aprx
cnt=1
aprx = arcpy.mp.ArcGISProject(aprfilein)
for m in aprx.listMaps():
    for lyr in m.listLayers():
        print (lyr)
        lyr.updateConnectionProperties(find_dict, replace_dict)

# ## Return the values of variable to the screen
print("working on it")

# ## Save a copy of the .aprx used in the code above.
p.saveACopy(aprfilenew)

print (".")
print ("..")

SGTomlins

Issue: Updating data sources for feature services failed when using a Python script in ArcGIS Pro.

Resolution: Logged a defect (BUG-000167099) since the issue could be reproduced internally.

View solution in original post

JeffMoulds

Steve, try the the following:

aprx = arcpy.mp.ArcGISProject('current')
m = aprx.listMaps()[0]
# put your layer name here
lyr = m.listLayers('Water1')[0]
lyrCIM = lyr.getDefinition("V2")
# put your URL here
# note that it has to start with 'URL='
lyrCIM.featureTable.dataConnection.workspaceConnectionString = 'URL=https://services2.arcgis.com/k4wsDILUIGeQ5HvW/arcgis/rest/services/Water2/FeatureServer'
lyr.setDefinition(lyrCIM)

This workaround uses the CIM. You can learn more about the CIM here: https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/python-cim-access.htm

And there are more sample of using the cim to update data sources here: https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/updatingandfixingdatasources.htm#ESRI_SECTION...

View solution in original post

SGTomlins

Here is my "working" .py to accomplish my goal of re-sourcing URL's at while being constrained by Pro 2.x

import arcpy,pprint,re
from arcpy.cim.__init__ import GetJSONForCIMObject

from datetime import date, datetime
import time
from timeit import default_timer as timer

#  ## Set a bunch of variables
StartTime = datetime.now()
start = timer()
today = date.today()
d1 = today.strftime("%Y%m%d")
now = datetime.now()
seconds = now.strftime("%S")
milliseconds = (seconds * 10)
# Current Time
Time = time.strftime("%I:%M:%S %p", time.localtime())
# Current Day
Day = time.strftime("%m-%d-%Y", time.localtime())

# Create the logging file
txtFile = open("c:\SGT\Logs\ChangeSrcDB_Connection_CIM.txt", "a")
print('Count Feature classes: ' + str(Day) + " " + str(Time))
txtFile.write('#####################################################################################' + "\n")
txtFile.write('Change connection properties started at: ' + str(Day) + " " + str(Time) + "\n")
txtFile.write('#####################################################################################')
txtFile.write("\n")
txtFile.close()

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'C:\SGT\ESRI\Python\ChangeSource\OriginalSourcePROD.aprx'

# ## Set the name of the aprx to be saved with new data sources
aprfilenew = r'C:\SGT\ESRI\Python\ChangeSource\NewSourceDEV.aprx'

p = arcpy.mp.ArcGISProject(aprfilein)
m = p.listMaps()[0]
l = m.listLayers()[0]

# ## Set the new URL connection string
urlprfx = 'URL=https://tinkerbell-az.ci.janesville.wi.us/devserver/rest/services/'
replaceurl = 'URL=https://tinkerbell-az.ci.janesville.wi.us/devserver/rest/services/Utilities/Utilities/FeatureServer'

# Just having fun with example of taking string apart for use later.. not needed in this example
def extract_base_url(url):
    # Define the regex pattern to match the base URL
    pattern = r"URL=(https?://[^/]+/[^/]+/[^/]+/[^/]+/[^/]+/[^/]+/)"
    match = re.search(pattern, url)
    if match:
        return match.group(1)
    else:
        return None


# Get the Layer name, get some information to satisfy need for operation to change source url
for lyr in m.listLayers():
    #pprint.pprint(lyr.connectionProperties)
    # with open (txtFile, 'a') as f:
    #     pprint.pprint(lyr.connectionProperties, stream=f)


    txtFile = open("c:\SGT\Logs\ChangeSrcDB_Connection_CIM.txt", "a")
    txtFile.write('The is the layer name: ' + str(lyr))
    txtFile.write("\n")
    txtFile.write(' ------------------------------------------------------------------' + '\n')
    txtFile.close()

    print ('This is the layer name :' + str(lyr))

    # Get the layer definition information CIM
    lyrCIM = lyr.getDefinition("V2")
    c = lyr.getDefinition('V2')
    dc = c.featureTable.dataConnection
    j = GetJSONForCIMObject(dc, 'V2')
    print(j)

    connstr = lyrCIM.featureTable.dataConnection.workspaceConnectionString
    print ("This is the connection string from the CIM= " + connstr)

    # Extract the base URL
    base_url = extract_base_url(connstr)
    # This loop needs some error trapping or string reconstruction depending on the from/to you are using
    if base_url:
        print("Base URL:", base_url)
        print("Changing the CIM connection properties")
        # Change the URL data source
        lyrCIM.featureTable.dataConnection.workspaceConnectionString = replaceurl
        lyr.setDefinition(lyrCIM)
        #lyrCIM = lyr.getDefinition("V2")
        c = lyr.getDefinition('V2')
        dc = c.featureTable.dataConnection
        j = GetJSONForCIMObject(dc, 'V2')
        print("J again" + j)

    else:
        print("Base URL not found.")

    print('...... next .....')

# Save a copy of the new apr with it's newly sourced services.
p.saveACopy(aprfilenew)

View solution in original post

SGTomlins · ‎04-08-2024

Just grasping at straws.. this did not work either....

find_dict = {'connection_info': {'authentication_mode': 'DBMS',
'database': 'gis_prod_util',
'db_connection_properties': 'underdog-az',
'dbclient': 'sqlserver',
'instance': 'sde:sqlserver:underdog-az',
'password': 'blahblahblah!',
'server': 'underdog-az',
'user': 'sde',
'version': 'sde.DEFAULT'},
'dataset': lyr ,
'workspace_factory': 'FeatureService'}

# } #,
# 'workspace_factory': 'SDE'}}

replace_dict = {'connection_info': {'authentication_mode': 'DBMS',
'database': 'gis_prod_util',
'db_connection_properties': 'tinkerbell-az',
'dbclient': 'sqlserver',
'instance': 'sde:sqlserver:tinkerbell-az\TINKER2019',
'password': 'noropeheredude!',
'server': 'tinkerbell-az',
'user': 'sde',
'version': 'sde.DEFAULT'},
'dataset': lyr,
'workspace_factory': 'FeatureService'}

SGTomlins

Issue: Updating data sources for feature services failed when using a Python script in ArcGIS Pro.

Resolution: Logged a defect (BUG-000167099) since the issue could be reproduced internally.

JeffMoulds

I'm not aware of any bugs when updating the data sources of enterprise geodatabase layers between DEV and PROD. Have you tried this syntax:

Project/Map/Layer/Table.updateConnectionProperties('DEV.sde', 'PROD.sde')

That syntax is easier than the connectionProperties dictionary. However, the dictionary workflow should work. If you need to stick with that dict workflow, can you give me more info about your layers? Make a new project with a map that just has 1.) a DEV layer, and, 2.) the corresponding PROD layer. Then run this script, and let me know what the output of the print statements are. Also, are the user names/passwords the same between DEV and PROD?

from arcpy.cim.__init__ import GetJSONForCIMObject

aprx = arcpy.mp.ArcGISProject('current')
m = aprx.listMaps()[0]
lyr = m.listLayers('MY DEV LAYER NAME')[0]
c = lyr.getDefinition('V2')
dc = c.featureTable.dataConnection
j = GetJSONForCIMObject(dc, 'V2')
print(j)

lyr = m.listLayers('MY PROD LAYER NAME')[0]
c = lyr.getDefinition('V2')
dc = c.featureTable.dataConnection
j = GetJSONForCIMObject(dc, 'V2')
print(j)

Can you also send me screen shots of Layer > Properties > Source for both the DEV and PROD layers? Thanks.

SGTomlins

@JeffMoulds Attached are the screen shots and python output you asked for.

I have tried this. I am SAML connected to the Portal that loads each the PROD and DEV. So I believe I am connected same/same.

# ## SGTomlins  6/10/24

aprfilenew = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds2\JMoulds2_99.aprx'

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds2\JMoulds2.aprx'
p = arcpy.mp.ArcGISProject(aprfilein)

# ## Execute the command to change the versions of the layers in the .aprx
p.updateConnectionProperties(r'C:\SGT\ESRI\Python\ChangeSource\dc@TINKER2019@TRAD@gis_prod_util.sde',
                                 r'C:\SGT\ESRI\Python\ChangeSource\dc@underdog-az@gis_prod_util.sde')

p.saveACopy(aprfilenew)

SGTomlins

Probably missing something obvious. Particulars about the connection properties or user mis-understanding... 😞 The JMoulds4.doc shows the connection properties for the Feature Service and same the Database feature. I can make the following work on the Db connection but I cannot get a URL to old/new.

import arcpy,pprint
# ## SGTomlins  6/10/24

aprfilenew = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds4_out.aprx'

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds4.aprx'
p = arcpy.mp.ArcGISProject(aprfilein)

# ## Execute the command to change the versions of the layers in the .aprx
p.updateConnectionProperties(r'C:\SGT\ESRI\Python\ChangeSource\SAML@underdog-az@gis_prod_util.sde',
                                 r'C:\SGT\ESRI\Python\ChangeSource\SAML@tinker2019@gis_prod_util.sde')

p.saveACopy(aprfilenew)

What is there to change or use for the URL old/new update?

Thanks in advance.

-Steve

SGTomlins

@JeffMoulds , I just saw this today. Went looking after I got the email about the bug not being "reproducible". I will take your direction and proceed. Thanks for the follow-up.

JeffMoulds

Steve,

I'm glad you were able to get the enterprise geodatabase workflow to work. E.g.

Project/Map/Layer/Table.updateConnectionProperties('DEV.sde', 'PROD.sde')

Regarding using the same code pattern for updating Feature Service URLs, e.g.

Layer.updateConnectionProperties(DevURL, ProdURL)

Are you able to upgrade your version of ArcGIS Pro to version 3.2 or higher? That's the version we added support for updating URLs. If you are unable to upgrade, I can show you a CIM workaround.

SGTomlins

@JeffMoulds The CIM would be helpful. We are bound by some legacy software version in-compatibility and an upgrade is pending and especially needed. I am setting up 3.2 / 10.9 on the dev environment. Hence, transferring connection properties between equipment is very handy.I take the working PROD and stand it up in DEV and updateconnections... when it is done, I send it back. wash rinse repeat.

All the best,

-Steve

JeffMoulds

Steve, try the the following:

aprx = arcpy.mp.ArcGISProject('current')
m = aprx.listMaps()[0]
# put your layer name here
lyr = m.listLayers('Water1')[0]
lyrCIM = lyr.getDefinition("V2")
# put your URL here
# note that it has to start with 'URL='
lyrCIM.featureTable.dataConnection.workspaceConnectionString = 'URL=https://services2.arcgis.com/k4wsDILUIGeQ5HvW/arcgis/rest/services/Water2/FeatureServer'
lyr.setDefinition(lyrCIM)

This workaround uses the CIM. You can learn more about the CIM here: https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/python-cim-access.htm

And there are more sample of using the cim to update data sources here: https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/updatingandfixingdatasources.htm#ESRI_SECTION...