topic Re: Problem updating tableview broken data sources. in Python Questions

Problem updating tableview broken data sources.

MathewCoyle — Wed, 21 Mar 2012 18:13:34 GMT

I have several mxds with broken data sources to SDE after a database move. I am running a script to update the broken data sources and keep getting an error on broken links to table views. All other layers update correctly.

Here is the error I get

Runtime error <type 'exceptions.ValueError'>: StandaloneTableObject: Unexpected error

Here's the relevant code where I am getting the error.

brokenlist = arcpy.mapping.ListBrokenDataSources(mxd)
for lyr in brokenlist:
    if ".sde" in lyr.dataSource:
        try:
            lyr.replaceDataSource(new_datasource, "SDE_WORKSPACE", lyr.name)

Has anyone encountered this or know a work around?

Edit: I've tried it with and without the lyr.name specified.

Re: Problem updating tableview broken data sources.

MichaelVolz — Thu, 22 Mar 2012 01:27:00 GMT

Mathew:

I do not have an answer for your question, but I am interested in the performance of your script.

I have written a replacedatasource script for SDE layers that loops through a directory searching for mxds and then resourcing the SDE layers. Unfortunately, the script eventually uses all the computer's RAM and it just hangs so I have to kill it in Task Manager.

Would you know if the memory on your python script keeps increasing? Your script might not have frozen because you did not process enough mxd files in your batch.

Any insight on your script and system would be greatly appreciated. Thanks.

Re: Problem updating tableview broken data sources.

MathewCoyle — Thu, 22 Mar 2012 12:29:29 GMT

I have actually run into that issue, but it was more to do with encountering a "corrupt" MXD (I put corrupt in quotes since MXD Doctor found no corrupted items, but once I ran a fix with MXD Doctor it worked fine, so I would call that corrupted). I am not sure what was wrong with it since it opened with ArcMap fine, just with broken links. Running through MXD Doctor showed a bunch of strange modules that were removed when I ran a fix and it processed fine after that. I've mostly seen this behaviour from some v10 .mxd files that were converted from prior version .mxt files, but that may be anecdotal. Not sure if this would be related to your situation or not. When I batched through I fixed about 500 mxds that ran fine once the "corrupted" ones were fixed or removed.

Re: Problem updating tableview broken data sources.

MichaelVolz — Thu, 22 Mar 2012 13:29:50 GMT

Mathew:

Can you share your code of batching through 500 mxds? The reason I ask is because maybe this is the cause of the memory issue I am having with replacing SDE connections. This phenonmenon happens everytime where I get through a certain amount of mxds before python freezes up.

This same problem did not occur if I batch converted sources to file geodatabases or personal geodatabases.

Re: Problem updating tableview broken data sources.

MathewCoyle — Thu, 22 Mar 2012 13:48:42 GMT

Here's the script I used. Are you freezing on the same mxd every time? How many are you converting?

Re: Problem updating tableview broken data sources.

MichaelVolz — Thu, 22 Mar 2012 13:59:54 GMT

Mathew:

I freeze on different mxds depending upon the subdirectory that I choose to convert the SDE datasources for, so it is not the same file every time.

Eventually I will need to convert up to 3 thousand mxds with each mxd having from 1 SDE connection to 40 or 50 SDE connections. This process will need to be done when my organization migrates from Oracle Client 10g to Oracle Client 11x, as this information is embedded in the direct connection string for every SDE connection. This is why this information is important to me.

Re: Problem updating tableview broken data sources.

MichaelVolz — Mon, 26 Mar 2012 16:10:45 GMT

Mathew:

I have a few more questions about how you run your script, since I have an almost identical script that hangs up after a short time.

Does your script strictly run against v10 mxds? Or can it run against previous versions of ArcGIS mxds?

Does your script strictly run against SDE v10? Or can it run against previous versions of ArcSDE connections?

Your feedback to this question is greatly appreciated. Thanks.

Re: Problem updating tableview broken data sources.

MathewCoyle — Mon, 26 Mar 2012 17:13:53 GMT

It did run on a mix of 9.3.1 and 10 mxds. Converting them all to 10 in the process. It was only run on a v10 SDE instance though, as that was what we migrated to in our upgrade. All the broken links were to a 9.3.1 SDE. I don't have any 9.3.1 SDE instances left to test this on unfortunately, that may be your problem, though I don't know why that would be an issue.

Re: Problem updating tableview broken data sources.

MichaelVolz — Mon, 26 Mar 2012 17:40:53 GMT

Mathew:

I have a incident logged with ESRI about this issue and the ESRI tech duplicated the same phenomenon on his system so he created a NIM for this bug. This is why I am surprised you are able to run your script on so many mxds.

Would you be able (and willing) to run your script on a batch of mxds and see if the pythonwin.exe or python.exe process in Task Manager keeps increasing in mem usage as that is what happens on my system?

I'm thinking that you might have a much more powerful system where you just never reached the mem usage threshold to lock up your python process.

Your help and feedback is greatly appreciated.

Re: Problem updating tableview broken data sources.

MathewCoyle — Mon, 26 Mar 2012 17:46:40 GMT

Sure I'll try re running it again. I ran it over night and didn't have much logging on as it was really just a test, but everything worked so I never bothered with checking if there were any issues other than if all the mxds could be opened and had the correct new SDE isntance. I have made my python process large address aware so that may have helped.

Re: Problem updating tableview broken data sources.

MichaelVolz — Mon, 26 Mar 2012 17:57:08 GMT

Mathew:

Sorry for my ignorance in this particular area, but how does one make a "python process large address aware"?

Thank you.

Re: Problem updating tableview broken data sources.

MathewCoyle — Mon, 26 Mar 2012 18:04:42 GMT

Here's the post that got me going. A few other memory management tips in the thread too.
http://forums.arcgis.com/threads/35655-Pool-from-multiprocessing-issues?p=121358&viewfull=1#post121358

Re: Problem updating tableview broken data sources.

MichaelVolz — Mon, 26 Mar 2012 18:49:18 GMT

Mathew:

Thanks for the following information:

Here's the post that got me going. A few other memory management tips in the thread too.
http://forums.arcgis.com/threads/356...l=1#post121358

If I understand the instructions correctly, all I need to do is run the exe on the computer where the python script is running. Then I select the .exe file that I want to be "large address aware" and check it to add it. Then when I run the python script on the computer again, I will have access to more RAM? That's it.

Also, if you run your script on another batch of files, can you turn off "large address aware" on the python.exe file to see if you have the memory leak issue that I am experiencing?

Your assistance and feedback is greatly appreciated. Thanks.

Re: Problem updating tableview broken data sources.

MathewCoyle — Mon, 26 Mar 2012 18:57:31 GMT

After more than 50 mxds there hasn't been any noticeable memory increase. Fluctuates from 260MB to 500MB but seems to be clearing alright every iteration. It does seem to clear more memory when it finishes a folder chain if that helps. If there is a memory leak it is in the KBs per mxd, at least in my environment.

Re: Problem updating tableview broken data sources.

MichaelVolz — Mon, 26 Mar 2012 19:05:27 GMT

When you ran this test, were you running python as "large address aware"?

Maybe you could try running the same test with python with "large address aware" turned off? Unless I read the instructions incorrectly, I thought all this involved was a checkbox when directed to this .exe file.

Re: Problem updating tableview broken data sources.

MathewCoyle — Mon, 26 Mar 2012 19:15:36 GMT

Yes it was with large address aware. I have a few other things running right now so I won't be able to revert it back to it's original format until tomorrow.

I did notice at over 100 mxds it isn't clearing down to the level it was before. Seems to shoot up when it hits a bunch of errors with the tables that I originally posted about but then drops back down, just not all the way. Just a guess, Arc may be trying to log errors with the GP tool in the background and not managing it very well.

Re: Problem updating tableview broken data sources.

MathewCoyle — Mon, 26 Mar 2012 19:40:59 GMT

Failure!

At around 200 mxds it crashed. System process was using close to 3GB, pythonw process using less than 400MB. My memory must have failed me, I must not have run it on all the MXDs I thought I had the first round. Must have broken it up into sub directories and ran it a couple of times instead of running off everything under the root dir. Of course I was also using the computer at the same time as it was processing and I left it to run over night before, maybe that had something to do with it.

Re: Problem updating tableview broken data sources.

lelaharrington — Tue, 28 Apr 2015 13:50:57 GMT

Did you ever get closure on this. i have the same issue i am just listing broken data sources in a file location using an os.walk script with listbrokendatasources

i have tried to source the table with list table views but hat is not working. this is the second script i have ever made and i am not to savvy on how to approach this.

Re: Problem updating tableview broken data sources.

MichaelVolz — Tue, 28 Apr 2015 14:28:56 GMT

Lela:

I don't believe python was ever able to handle that situation, so I went through all the mxd files and flagged the broken tableviews into a log file and then fixed them manually (This was a very minor portion of the SDE connections - < 0.1%, thus a small job to perform manually).

Re: Problem updating tableview broken data sources.

lelaharrington — Tue, 28 Apr 2015 14:41:19 GMT

see where as mine is a huge deal. i am looking to either use an except system that just ignores them. that would be ok to. the broken sources actually stops my os.walk script when it hits a broken table. is there a way to say "if you run into a broken table ignore it and keep running"

Thank you

Lela Harrington