Use Subprocess to Alleviate Memory Usage Issues

[MichaelVolz]

To All Python Users:

I have a script that is updating mxds and then saving them.  After saving the mxd I am deleting the mxd object, but memory usage keeps increasing until the python script just hangs.

I have read that I can use a subprocess to fix this problem, but I am unsure of the syntax to use.  I will be drilling down through a directory and all of its subdirectories for mxd files (This will be my main python script).  Once I have an mxd file as the focus, I want to use a subprocess to start another python script that will open the mxd, replace data sources, save the mxd and then delete the mxd object from memory.  Then I will return to the main script to get another file which should free up memory as the subprocess has terminated.

Does anyone know the syntax I would use to call the secondary python script with an argument being the mxd file that has the focus?  Any help or hints are greatly appreciated.  Thank you.

[ChrisFrost]

 
import subprocess

procs = []

cmd = "python foo.py arg1 arg2"
proc = subprocess.Popen(cmd,shell=True)
procs.append(proc)

you will probably want to keep track of the number of running processes and limit how many are run at once.

 
def numRunningProcs(procs):
    numProcs = 0
    for proc in procs:
        proc.poll()
        if proc.returncode == None:
            numProcs = numProcs + 1
        
    return numProcs

[MichaelVolz]

Chris:

I tried running the first portion of your code, but the subprocess is not executing.

Here is my syntax to call the subprocess script:

cmd = "C:\\Python26\\ArcGIS10.0\\python.exe \\\\tgapparc01\\c$\\GIS Testing\\ReplaceSource.py full_path"


In the cmd line, full_path is the file that has the focus.  File is stored in the full_path variable.

In the ReplaceSource.py script, I am trying to use the arcpy.GetParameterAsText(0) to bring in this argument.  I'm not sure if this is the problem?

Could this issue possibly be how I am calling out the python executable and the python script that gets run in the subprocess?

Your assistance is greatly appreciated.  Thanks.

[ChrisFrost]

Chris:

I tried running the first portion of your code, but the subprocess is not executing.

Here is my syntax to call the subprocess script:

cmd = "C:\\Python26\\ArcGIS10.0\\python.exe \\\\tgapparc01\\c$\\GIS Testing\\ReplaceSource.py full_path"


In the cmd line, full_path is the file that has the focus.  File is stored in the full_path variable.

In the ReplaceSource.py script, I am trying to use the arcpy.GetParameterAsText(0) to bring in this argument.  I'm not sure if this is the problem?

Could this issue possibly be how I am calling out the python executable and the python script that gets run in the subprocess?

Your assistance is greatly appreciated.  Thanks.

I believe the problem is with your command string.  The space in the path to the python script causes it to be treated as two arguments. So python.exe only gets
"\\\\tgapparc01\\c$\\GIS" as the script argument.  If you cannot change the directory name, then you might try putting the full path to the python script in single quotes.

[MichaelVolz]

Chris:

Can you please be a little bit more specific in what your code is looking for with the cmd = "python foo.py arg1 arg2" line?  I still cannot get my code to evaluate the subprocess python script.

Does python refer to python.exe that has been installed on the computer that you are running this script from (e.g. c:\Python26\ArcGIS10.0\python.exe)?

Does foo.py refer tp the python script that you are running in the subprocess?  If so, wouldn't you need to specify the full path to this file?

Are arg1 and arg2 variables from the primary script that you are sending to the subprocess script?  If so, what is the syntax to get the subprocess script to accept these arguments?  I ask because I am using arcpy.GetParameterAsText(0) to evaluate the single argument that I am trying to pass.

Your assistance is greatly appreciated.  Thanks.

[ChrisFrost]

not sure if you saw my last response...

[MichaelVolz]

Chris:

I did see your last response and I moved the python script, that is called, to a folder with no spaces.  No matter how I format the cmd line, the subprocess never gets executed.

[ChrisFrost]

Chris:

Can you please be a little bit more specific in what your code is looking for with the cmd = "python foo.py arg1 arg2" line?  I still cannot get my code to evaluate the subprocess python script.

Does python refer to python.exe that has been installed on the computer that you are running this script from (e.g. c:\Python26\ArcGIS10.0\python.exe)?

Does foo.py refer tp the python script that you are running in the subprocess?  If so, wouldn't you need to specify the full path to this file?

Are arg1 and arg2 variables from the primary script that you are sending to the subprocess script?  If so, what is the syntax to get the subprocess script to accept these arguments?  I ask because I am using arcpy.GetParameterAsText(0) to evaluate the single argument that I am trying to pass.

Your assistance is greatly appreciated.  Thanks.

1. python refers to python.exe. if it is not in the system path, then use the full path to python.exe
2. foo.py is the python script to run in subprocess.  use the full path.  there cannot be spaces in the path.
3. arg1,arg2 are arguements to the subprocess script. use arcpy.GetParameterAsText(0) as usual.

[ChrisFrost]

Chris:

I did see your last response and I moved the python script, that is called, to a folder with no spaces.  No matter how I format the cmd line, the subprocess never gets executed.

can you post the code in which you are making the subprocess call.

[MichaelVolz]

Chris:

Here's the code I am trying to run to call a subprocess to actually modify properties of mxd files:

import os, sys, string, arcpy
import subprocess

mxd_match = ".mxd"
fileErrList = []
procs = []

Directory_Search = r"\\tgapparc00\e$\restore5\Planning_Run\Broken_Links\Test"

for root, dirs, files in os.walk(Directory_Search):

    fListLength = len(files)

    if (fListLength != 0):
        n = 0
        for f in files:

            if f.endswith(mxd_match):

                full_path = root + "\\" + str(f)
                cmd = "'C:\Python26\ArcGIS10.0\python.exe \\\\tgapparc01\\c$\\ReplaceSource.py' full_path"

                    proc = subprocess.Popen(cmd,shell=True)
                    
                    procs.append(proc)