Two problems with using layer.replaceDataSource.

[RichardMontague]

We have a little over 300 layers in 275 or so .lyr files.  These files have been created over the course of years and have many different workspace paths.  We are replacing our GIS server so I need to change the connection file for all of the layers.

The first and most important is the datasource is not changed.  Maybe I am too close to it but I have tried many different ways to write this but I can't seem to get the layer file to change datasources.

The second and not so critical problem is memory usage.  The directory has a little over 300 layers, some of which are in group layers.  As the program progresses, the memory usage keeps building until PyScripter crashes.  I get the same results if I run the script in the python window in ArcCatalog.  If I run a subset of the files the memory usages builds and isn't released until I close out PyScripter or ArcCatalog.  In a command window it uses up all of the available memory after around 60 layers.

Below is the code I am using.  Thanks for any help I can get.


import arcpy
import os
from arcpy import env

env.overwriteOutput = True

TheLayerDir = 'J:\\ArcMap Layers\\dbx\\'
for dirname, dirnames, filenames in os.walk(TheLayerDir):
    [INDENT]for filename in filenames:
        [INDENT]newFilename =  os.path.join(dirname, filename)
        if newFilename[-4:] == '.lyr':
            [INDENT]lyrFile = arcpy.mapping.Layer(newFilename)
            for lyr in arcpy.mapping.ListLayers(lyrFile):
               [INDENT] if lyr.supports("SERVICEPROPERTIES"):
                    [INDENT]if not lyr.isGroupLayer:
                        [INDENT]servProp = lyr.serviceProperties
                        NewDataSource = 'J:\\SDEConnections\\dbx\\GISuser-' + servProp.get('Database', 'N/A') + '.sde'
                        try:
      [INDENT] lyr.replaceDataSource(NewDataSource,"SDE_WORKSPACE")[/INDENT]
                        except:
                           [INDENT] print "This is where it breaks: " + lyr.name
                            print "The data source is: " + NewDataSource
                            print arcpy.GetMessages(2)[/INDENT][/INDENT][/INDENT][/INDENT][/INDENT][/INDENT]
            lyrFile.save[/INDENT]
print "That's all folks"

[MichaelVolz]

Richard:

Are you able to update the datasource for any lyr files?

What kind of machine are you running this script on?  I am resourcing both lyr files and SDE connections inside 1000s of mxds.  I was also encountering memory usage which kept increasing and eventually halted my script.  I am now running the script on a Windows 2008 Server server and I am able to get through batches with 2000 files.

Can you create a print statement for your NewDataSource to see if it is getting the correct value to be added to the lyr.replaceDataSource statement?

[RichardMontague]

Richard:

Are you able to update the datasource for any lyr files?

What kind of machine are you running this script on?  I am resourcing both lyr files and SDE connections inside 1000s of mxds.  I was also encountering memory usage which kept increasing and eventually halted my script.  I am now running the script on a Windows 2008 Server server and I am able to get through batches with 2000 files.

Can you create a print statement for your NewDataSource to see if it is getting the correct value to be added to the lyr.replaceDataSource statement?

Mxd files are my next headache, this one just deals with .lyr files.  I am running this on a new Dell desktop.  I can break this down to multiple sub-directories and run them individually so that is only an annoyance. 

I have created a print statement for NewDataSource and it the correct path to my connection file.  I can not get the datasource to change, even when I run it on just one .lyr file.

[MichaelVolz]

What does this call servProp.get('Database', 'N/A') return?  Can you psot this info to the thread?

[MichaelVolz]

Richard:

Is the statement

NewDataSource = 'J:\\SDEConnections\\dbx\\GISuser-' + servProp.get('Database', 'N/A') + '.sde'

supposed to represent a database connection that you have already created for your new GIS server?

My issue is different from yours where I am connecting to an Oracle database.  The database was originally at Oracle 10g, but we upgraded the Oracle database to 11g but did not upgrade the desktop clients so we were connecting to an 11g Oracle database with 10g Oracle client connections.

Everything works, but we want to move to an ESRI supported environment where the Oracle client is at the same version as the database.  Originally I was going to create the SDE connections beforehand (there would be about 150 different connections to development, test and production SDE databases), but I found it was easier to create the new SDE 11g connections on the fly.

[RichardMontague]

What does this call servProp.get('Database', 'N/A') return?  Can you psot this info to the thread?

We have our SDE stored in 4 or 5 different instances of SQL server.  The call gives me the instance that the layer is stored in.  I set up all of the connection files the same way and just plug in the servProp call to make sure it goes to the right instance.The expanded code that I found is this:

for lyr in arcpy.mapping.ListLayers(lyrFile):
               [INDENT] if lyr.supports("SERVICEPROPERTIES"):
                    [INDENT]if not lyr.isGroupLayer:
                        [INDENT]servProp = lyr.serviceProperties
                        NewDataSource = 'J:\\SDEConnections\\dbx\\GISuser-' + servProp.get('Database', 'N/A') + '.sde'
                        print "Service Type: " + servProp.get('ServiceType', 'N/A')
                        print "    Database:       " + servProp.get('Database', 'N/A')
                        print "    Server:         " + servProp.get('Server', 'N/A')
                        print "    Service:        " + servProp.get('Service', 'N/A')
                        print "    Version:        " + servProp.get('Version', 'N/A')
                        print "    User name:      " + servProp.get('UserName', 'N/A')
                        print "    Authentication: " + servProp.get('AuthenticationMode', 'N/A')
                        print ""[/INDENT][/INDENT][/INDENT]

[MichaelVolz]

Try using lyrFile.saveACopy instead of lyrFile.save


This is the syntax for my python code that works.  It has been a long time since I wrote this code and I forget why I am using saveACopy instead of save (There might have been a bug).

What version of Arc are you using as I am at v10.0?

[RichardMontague]

First of my apologies for a rookie mistake.  It seems that I was checking the data source in ArcCatalog but using the original layers instead of the copies I was testing on.  Once I realized my error and checked the copies that I was actually trying to change I found the code had worked.  Further testing seem to show that you do need the save() or saveACopy() function. 

Still have the issue with the memory. 

Thanks Michael for your help.

[MichaelVolz]

Richard:

You are running out of memory after only getting through a portion of the 275 lyr files?

Are you running on a desktop computer or a server that has more resources?

I would suggest running the script on a server, if you access to one, as that is where I am running my script to get as large a batches as possible.

The job of converting layers in mxds is a much bigger job than converting lyr files (approximately 150 times bigger for me as I need to convert over 50,000 connections), so I would think you would want to be able to process much larger batches than you can currently process.  I have opened incident(s) with ESRI in regards to the memory issue, but their answer after internal investigation was always to break the job down into smaller batches.

[RichardMontague]

Michael

It wasn't too bad breaking up the lyr files into smaller batches.  If I ever have to do this again I will definitely go the server route.

Since the mxd files could be anywhere on the network or local machines I decided to let the users fix their own as needed.  I am creating a python add-in button that they can push to fix any broken sde connection in their document.

Thanks again for your help.