topic Re: Update data sources. enterprise server. dictionary. updateConnectionProperties in Python Questions

Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Thu, 04 Apr 2024 13:27:30 GMT

Greetings,

I have successfully updated connection properties in all scenarios EXCEPT enterprise to enterprise (dev to production). The following code is my example. I have 2.9 (yes i know outdated) Python 1.9.1 ( again. outdated).

Is this a limitation of the updateConnectionProperties?

Is this related the attached screenshot, where the Update Data Sources is full of "Unsupported Items"

Feel like I am missing something obvious... ? help...

import arcpy,pprint
# ## SGTomlins  3/27/24
# ## Examples of workspaces that can be used.  Either a .gdb or a connection string.
# ## arcpy.env.workspace = r"C:\SGT\ESRI\ArcGIS_Pro\LoadLayerFiles\LoadLayerFiles.gdb"
# ## arcpy.env.workspace = r"C:\SGT\ESRI\_ConnectStrings\SDE@underdog-az@gis_prod_util_test.sde"
#

# ## find_dict = r'C:\SGT\SDE@underdog-az@gis_prod_util_test.sde.gdb'
# ## replace_dict = r'C:\Data\!Database_Connections\GISReader@underdog-az@gis_prod_util.sde'

find_dict = {'connection_info': {'user': 'sde',
                                 'password': 'guessmenow!',
                                 'version': 'sde.DEFAULT',
                                 'server': 'underdog-az',
                                 'instance': 'sde:sqlserver:underdog-az',
                                 'dbclient': 'sqlserver',
                                 'database' : 'gis_prod_util',
                                'db_connection_properties' : 'underdog-az'
                                 }}

replace_dict ={'connection_info': {'user': 'sde',
                                 'password': 'guessmelater',
                                 'version': 'sde.DEFAULT',
                                 'server': 'tinkerbell-az',
                                 'instance': 'sde:sqlserver:tinkerbell-az\TINKER2019',
                                 'dbclient': 'sqlserver',
                                 'database' : 'gis_prod_util',
                                 'db_connection_properties' : 'tinkerbell-az'
                                 }}



# ## sets some variables and the output path for the created .lyrx files
aprfilenew = r'c:\SGT\TINKER2019-ResourceNoUtil31.aprx'

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'c:\SGT\TINKER2019-Resource_NoUtil.aprx'
p = arcpy.mp.ArcGISProject(aprfilein)

# ## Create lists for "Maps" and "Layers" in the .aprx
cnt=1
aprx = arcpy.mp.ArcGISProject(aprfilein)
for m in aprx.listMaps():
    for lyr in m.listLayers():
        print (lyr)
        lyr.updateConnectionProperties(find_dict, replace_dict)

# ## Return the values of variable to the screen
print("working on it")

# ## Save a copy of the .aprx used in the code above.
p.saveACopy(aprfilenew)

print (".")
print ("..")

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Mon, 08 Apr 2024 13:13:49 GMT

Just grasping at straws.. this did not work either....

find_dict = {'connection_info': {'authentication_mode': 'DBMS',
'database': 'gis_prod_util',
'db_connection_properties': 'underdog-az',
'dbclient': 'sqlserver',
'instance': 'sde:sqlserver:underdog-az',
'password': 'blahblahblah!',
'server': 'underdog-az',
'user': 'sde',
'version': 'sde.DEFAULT'},
'dataset': lyr ,
'workspace_factory': 'FeatureService'}

# } #,
# 'workspace_factory': 'SDE'}}

replace_dict = {'connection_info': {'authentication_mode': 'DBMS',
'database': 'gis_prod_util',
'db_connection_properties': 'tinkerbell-az',
'dbclient': 'sqlserver',
'instance': 'sde:sqlserver:tinkerbell-az\TINKER2019',
'password': 'noropeheredude!',
'server': 'tinkerbell-az',
'user': 'sde',
'version': 'sde.DEFAULT'},
'dataset': lyr,
'workspace_factory': 'FeatureService'}

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Tue, 21 May 2024 12:31:06 GMT

Issue: Updating data sources for feature services failed when using a Python script in ArcGIS Pro.

Resolution: Logged a defect (BUG-000167099) since the issue could be reproduced internally.

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

JeffMoulds — Tue, 28 May 2024 21:49:40 GMT

I'm not aware of any bugs when updating the data sources of enterprise geodatabase layers between DEV and PROD. Have you tried this syntax:

Project/Map/Layer/Table.updateConnectionProperties('DEV.sde', 'PROD.sde')

That syntax is easier than the connectionProperties dictionary. However, the dictionary workflow should work. If you need to stick with that dict workflow, can you give me more info about your layers? Make a new project with a map that just has 1.) a DEV layer, and, 2.) the corresponding PROD layer. Then run this script, and let me know what the output of the print statements are. Also, are the user names/passwords the same between DEV and PROD?

from arcpy.cim.__init__ import GetJSONForCIMObject aprx = arcpy.mp.ArcGISProject('current') m = aprx.listMaps()[0] lyr = m.listLayers('MY DEV LAYER NAME')[0] c = lyr.getDefinition('V2') dc = c.featureTable.dataConnection j = GetJSONForCIMObject(dc, 'V2') print(j) lyr = m.listLayers('MY PROD LAYER NAME')[0] c = lyr.getDefinition('V2') dc = c.featureTable.dataConnection j = GetJSONForCIMObject(dc, 'V2') print(j)

Can you also send me screen shots of Layer > Properties > Source for both the DEV and PROD layers? Thanks.

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Mon, 10 Jun 2024 12:20:16 GMT

@JeffMoulds , I just saw this today. Went looking after I got the email about the bug not being "reproducible". I will take your direction and proceed. Thanks for the follow-up.

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Mon, 10 Jun 2024 15:31:22 GMT

@JeffMoulds Attached are the screen shots and python output you asked for.

I have tried this. I am SAML connected to the Portal that loads each the PROD and DEV. So I believe I am connected same/same.

# ## SGTomlins  6/10/24

aprfilenew = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds2\JMoulds2_99.aprx'

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds2\JMoulds2.aprx'
p = arcpy.mp.ArcGISProject(aprfilein)

# ## Execute the command to change the versions of the layers in the .aprx
p.updateConnectionProperties(r'C:\SGT\ESRI\Python\ChangeSource\dc@TINKER2019@TRAD@gis_prod_util.sde',
                                 r'C:\SGT\ESRI\Python\ChangeSource\dc@underdog-az@gis_prod_util.sde')

p.saveACopy(aprfilenew)

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Mon, 10 Jun 2024 18:57:38 GMT

Probably missing something obvious. Particulars about the connection properties or user mis-understanding... 😞 The JMoulds4.doc shows the connection properties for the Feature Service and same the Database feature. I can make the following work on the Db connection but I cannot get a URL to old/new.

import arcpy,pprint
# ## SGTomlins  6/10/24

aprfilenew = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds4_out.aprx'

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'C:\SGT\ESRI\Python\ChangeSource\JMoulds4.aprx'
p = arcpy.mp.ArcGISProject(aprfilein)

# ## Execute the command to change the versions of the layers in the .aprx
p.updateConnectionProperties(r'C:\SGT\ESRI\Python\ChangeSource\SAML@underdog-az@gis_prod_util.sde',
                                 r'C:\SGT\ESRI\Python\ChangeSource\SAML@tinker2019@gis_prod_util.sde')

p.saveACopy(aprfilenew)

What is there to change or use for the URL old/new update?

Thanks in advance.

-Steve

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

JeffMoulds — Mon, 10 Jun 2024 23:36:23 GMT

Steve,

I'm glad you were able to get the enterprise geodatabase workflow to work. E.g.

Project/Map/Layer/Table.updateConnectionProperties('DEV.sde', 'PROD.sde')

Regarding using the same code pattern for updating Feature Service URLs, e.g.

Layer.updateConnectionProperties(DevURL, ProdURL)

Are you able to upgrade your version of ArcGIS Pro to version 3.2 or higher? That's the version we added support for updating URLs. If you are unable to upgrade, I can show you a CIM workaround.

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Tue, 11 Jun 2024 12:41:03 GMT

@JeffMoulds The CIM would be helpful. We are bound by some legacy software version in-compatibility and an upgrade is pending and especially needed. I am setting up 3.2 / 10.9 on the dev environment. Hence, transferring connection properties between equipment is very handy.I take the working PROD and stand it up in DEV and updateconnections... when it is done, I send it back. wash rinse repeat.

All the best,

-Steve

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

JeffMoulds — Wed, 12 Jun 2024 14:22:53 GMT

Steve, try the the following:

aprx = arcpy.mp.ArcGISProject('current') m = aprx.listMaps()[0] # put your layer name here lyr = m.listLayers('Water1')[0] lyrCIM = lyr.getDefinition("V2") # put your URL here # note that it has to start with 'URL=' lyrCIM.featureTable.dataConnection.workspaceConnectionString = 'URL=https://services2.arcgis.com/k4wsDILUIGeQ5HvW/arcgis/rest/services/Water2/FeatureServer' lyr.setDefinition(lyrCIM)

This workaround uses the CIM. You can learn more about the CIM here: https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/python-cim-access.htm

And there are more sample of using the cim to update data sources here: https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/updatingandfixingdatasources.htm#ESRI_SECTION1_DD16BE1F4AC54F078A04C84BD22C8D9B

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Wed, 12 Jun 2024 15:38:55 GMT

Worked like a champ.... !! Much Obliged.... !!

-Steve

Re: Update data sources. enterprise server. dictionary. updateConnectionProperties

SGTomlins — Wed, 12 Jun 2024 17:56:30 GMT

Here is my "working" .py to accomplish my goal of re-sourcing URL's at while being constrained by Pro 2.x

import arcpy,pprint,re
from arcpy.cim.__init__ import GetJSONForCIMObject

from datetime import date, datetime
import time
from timeit import default_timer as timer

#  ## Set a bunch of variables
StartTime = datetime.now()
start = timer()
today = date.today()
d1 = today.strftime("%Y%m%d")
now = datetime.now()
seconds = now.strftime("%S")
milliseconds = (seconds * 10)
# Current Time
Time = time.strftime("%I:%M:%S %p", time.localtime())
# Current Day
Day = time.strftime("%m-%d-%Y", time.localtime())

# Create the logging file
txtFile = open("c:\SGT\Logs\ChangeSrcDB_Connection_CIM.txt", "a")
print('Count Feature classes: ' + str(Day) + " " + str(Time))
txtFile.write('#####################################################################################' + "\n")
txtFile.write('Change connection properties started at: ' + str(Day) + " " + str(Time) + "\n")
txtFile.write('#####################################################################################')
txtFile.write("\n")
txtFile.close()

# ## Set your ArcGIS Project to excute the code on.
aprfilein = r'C:\SGT\ESRI\Python\ChangeSource\OriginalSourcePROD.aprx'

# ## Set the name of the aprx to be saved with new data sources
aprfilenew = r'C:\SGT\ESRI\Python\ChangeSource\NewSourceDEV.aprx'

p = arcpy.mp.ArcGISProject(aprfilein)
m = p.listMaps()[0]
l = m.listLayers()[0]

# ## Set the new URL connection string
urlprfx = 'URL=https://tinkerbell-az.ci.janesville.wi.us/devserver/rest/services/'
replaceurl = 'URL=https://tinkerbell-az.ci.janesville.wi.us/devserver/rest/services/Utilities/Utilities/FeatureServer'

# Just having fun with example of taking string apart for use later.. not needed in this example
def extract_base_url(url):
    # Define the regex pattern to match the base URL
    pattern = r"URL=(https?://[^/]+/[^/]+/[^/]+/[^/]+/[^/]+/[^/]+/)"
    match = re.search(pattern, url)
    if match:
        return match.group(1)
    else:
        return None


# Get the Layer name, get some information to satisfy need for operation to change source url
for lyr in m.listLayers():
    #pprint.pprint(lyr.connectionProperties)
    # with open (txtFile, 'a') as f:
    #     pprint.pprint(lyr.connectionProperties, stream=f)


    txtFile = open("c:\SGT\Logs\ChangeSrcDB_Connection_CIM.txt", "a")
    txtFile.write('The is the layer name: ' + str(lyr))
    txtFile.write("\n")
    txtFile.write(' ------------------------------------------------------------------' + '\n')
    txtFile.close()

    print ('This is the layer name :' + str(lyr))

    # Get the layer definition information CIM
    lyrCIM = lyr.getDefinition("V2")
    c = lyr.getDefinition('V2')
    dc = c.featureTable.dataConnection
    j = GetJSONForCIMObject(dc, 'V2')
    print(j)

    connstr = lyrCIM.featureTable.dataConnection.workspaceConnectionString
    print ("This is the connection string from the CIM= " + connstr)

    # Extract the base URL
    base_url = extract_base_url(connstr)
    # This loop needs some error trapping or string reconstruction depending on the from/to you are using
    if base_url:
        print("Base URL:", base_url)
        print("Changing the CIM connection properties")
        # Change the URL data source
        lyrCIM.featureTable.dataConnection.workspaceConnectionString = replaceurl
        lyr.setDefinition(lyrCIM)
        #lyrCIM = lyr.getDefinition("V2")
        c = lyr.getDefinition('V2')
        dc = c.featureTable.dataConnection
        j = GetJSONForCIMObject(dc, 'V2')
        print("J again" + j)

    else:
        print("Base URL not found.")

    print('...... next .....')

# Save a copy of the new apr with it's newly sourced services.
p.saveACopy(aprfilenew)