nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Use Subprocess to Alleviate Memory Usage Issues

mvolz47

To All Python Users:

I have a script that is updating mxds and then saving them. After saving the mxd I am deleting the mxd object, but memory usage keeps increasing until the python script just hangs.

I have read that I can use a subprocess to fix this problem, but I am unsure of the syntax to use. I will be drilling down through a directory and all of its subdirectories for mxd files (This will be my main python script). Once I have an mxd file as the focus, I want to use a subprocess to start another python script that will open the mxd, replace data sources, save the mxd and then delete the mxd object from memory. Then I will return to the main script to get another file which should free up memory as the subprocess has terminated.

Does anyone know the syntax I would use to call the secondary python script with an argument being the mxd file that has the focus? Any help or hints are greatly appreciated. Thank you.

Trouver plus de publications taguées avec

Accepted answers

All comments

mvolz47

Chris:

I tried your approach to set lyrs equal to a variable and then delete the lyrs variable after using it in each mxd, but it did not help the memory issue. I also tried to delete the servProp variable and that did not help the memory issue either.

ChrisFrost

the only thing suspect i can see is the arcpy.mapping.ListLayers(mxd) value that might not get released. maybe try setting it to a variable and then deleting the variable when done.

if f.endswith(mxd_match):
                
                mxd = arcpy.mapping.MapDocument(full_path)
                lyrs = arcpy.mapping.ListLayers(mxd)
                for lyr in lyrs:
                    try:
                .....
                del lyrs

mvolz47

Thanks Chris. The script with the subprocess did exactly what you said it would and flooded the processor with parallel processes.

Here is my code with just one python script where the memory usage keeps increasing until the python scripts freezes up. The script is meant to loop through a selected directory, get the mxds and resource the SDE layers from the original SDE connection to a different SDE connection. You can open up Task Manager while running this script and see how memory usage for the pythonw.exe process keeps increasing. Please note that you will need to run this script on a directory with a good number of mxd files that have many SDE layers in order to see the performance hit.

import os, sys, string, arcpy

mxd_match = ".mxd"

Directory_Search = r"\\server00\e$\restore5\Experiment\Test_10"
new_connPrefix = r"C:\Documents and Settings\Application Data\ESRI\Desktop10.0\ArcCatalog"

def Conn_Info(usr):
    if usr == "user01":
        new_connection = new_connPrefix + "\\" + usr + "_dir_conn@production.sde"
        return new_connection
    elif usr == "user02":
        new_connection = new_connPrefix + "\\" + usr + "_dir_conn@development.sde"
        return new_connection
    elif usr == "user03":
        new_connection = new_connPrefix + "\\" + usr + "_dir_conn@production.sde"
        return new_connection
    del new_connection
        
for root, dirs, files in os.walk(Directory_Search):
# for root, dirs, files in os.walk(arcpy.GetParameterAsText(0)):

    fListLength = len(files)

    if (fListLength != 0):
        n = 0
        for f in files:

            full_path = root + "\\" + str(f)

            if f.endswith(mxd_match):
                
                mxd = arcpy.mapping.MapDocument(full_path)

                for lyr in arcpy.mapping.ListLayers(mxd):
                    try:
                        if lyr.supports("DATASOURCE"):
                            
                            if lyr.supports("SERVICEPROPERTIES"):
                                servProp = lyr.serviceProperties
                                user = str(servProp.get('UserName', 'N/A'))
                                
                                new_conn = Conn_Info(user)
                                
                                lyr.replaceDataSource(new_conn, "SDE_WORKSPACE", lyr.datasetName)


                    except:
                        print arcpy.GetMessages(2)
                    del lyr
                    
                try:        
                    # mxd.saveACopy(root + "\\" + f[:-4] + "_New.mxd", "9.3")
                    mxd.save()
                    
                except:
                    print arcpy.GetMessages(2)

                del mxd
                
        del full_path

Thanks for all your help!!

ChrisFrost

Chris:

Why do I need to worry about the number of subprocesses? I wrote my main python script to loop through a directory, its subdirectories and files looking for mxds. Then I call the subprocess python script for the file that has focus and I manipulate that mxd. Once that is done and the mxd gets saved, I thought the code goes back to the main python script and releases the subprocess python script from memory. Then I loop to the next mxd and call the subprocess python script again.

The reason I went this route in the first place was due to memory usage always climbing while looping through a directory until the python script just hung. Could it be that I just needed to release additional items from temporary memory when I was just using one script to obtain the results?

Thanks.

You need to worry about the number of processes because the script that fires off Popen does not wait for Popen to finsh. Therefore you will likely end up with many processes running concurrently (more than the number of cores/processors), saturating the cpu and causing the machine to thrash.

It's possible that your original script is written in a manner that is causing the high memory usage because temp data is not released for garbage collection. Post the script if you like and i'll take a look.

mvolz47

Chris:

Why do I need to worry about the number of subprocesses? I wrote my main python script to loop through a directory, its subdirectories and files looking for mxds. Then I call the subprocess python script for the file that has focus and I manipulate that mxd. Once that is done and the mxd gets saved, I thought the code goes back to the main python script and releases the subprocess python script from memory. Then I loop to the next mxd and call the subprocess python script again.

The reason I went this route in the first place was due to memory usage always climbing while looping through a directory until the python script just hung. Could it be that I just needed to release additional items from temporary memory when I was just using one script to obtain the results?

Thanks.

ChrisFrost

as an alternative to subprocess.Popen, you might consider using the multiprocessing.Pool class. With Popen you will need to manage the number of processes that are running at once. With Pool you can define the number of worker processes in the Pool (#cores/processors -1) and it will keep that number of worker processes running.

mvolz47

Thanks Chris

That was the syntax that was needed to call the subprocess python script in Windows.

I could not find any other examples of that syntax when searching the web.

ChrisFrost

It seems that the syntax I was using only works on linux, not windows.

for windows,

cmd = [r"C:\Python26\ArcGIS10.0\python.exe", r"\\tgapparc01\c$\ReplaceSource.py", full_path]
proc = subprocess.Popen(cmd)

mvolz47

Chris:

Here's the code I am trying to run to call a subprocess to actually modify properties of mxd files:

import os, sys, string, arcpy
import subprocess

mxd_match = ".mxd"
fileErrList = []
procs = []

Directory_Search = r"\\tgapparc00\e$\restore5\Planning_Run\Broken_Links\Test"

for root, dirs, files in os.walk(Directory_Search):

    fListLength = len(files)

    if (fListLength != 0):
        n = 0
        for f in files:

            if f.endswith(mxd_match):

                full_path = root + "\\" + str(f)
                cmd = "'C:\Python26\ArcGIS10.0\python.exe \\\\tgapparc01\\c$\\ReplaceSource.py' full_path"

                    proc = subprocess.Popen(cmd,shell=True)
                    
                    procs.append(proc)

ChrisFrost

Chris:

I did see your last response and I moved the python script, that is called, to a folder with no spaces. No matter how I format the cmd line, the subprocess never gets executed.

can you post the code in which you are making the subprocess call.

ChrisFrost

Chris:

Can you please be a little bit more specific in what your code is looking for with the cmd = "python foo.py arg1 arg2" line? I still cannot get my code to evaluate the subprocess python script.

Does python refer to python.exe that has been installed on the computer that you are running this script from (e.g. c:\Python26\ArcGIS10.0\python.exe)?

Does foo.py refer tp the python script that you are running in the subprocess? If so, wouldn't you need to specify the full path to this file?

Are arg1 and arg2 variables from the primary script that you are sending to the subprocess script? If so, what is the syntax to get the subprocess script to accept these arguments? I ask because I am using arcpy.GetParameterAsText(0) to evaluate the single argument that I am trying to pass.

Your assistance is greatly appreciated. Thanks.

1. python refers to python.exe. if it is not in the system path, then use the full path to python.exe
2. foo.py is the python script to run in subprocess. use the full path. there cannot be spaces in the path.
3. arg1,arg2 are arguements to the subprocess script. use arcpy.GetParameterAsText(0) as usual.

mvolz47

Chris:

I did see your last response and I moved the python script, that is called, to a folder with no spaces. No matter how I format the cmd line, the subprocess never gets executed.

ChrisFrost

not sure if you saw my last response...

mvolz47

Chris:

Can you please be a little bit more specific in what your code is looking for with the cmd = "python foo.py arg1 arg2" line? I still cannot get my code to evaluate the subprocess python script.

Does python refer to python.exe that has been installed on the computer that you are running this script from (e.g. c:\Python26\ArcGIS10.0\python.exe)?

Does foo.py refer tp the python script that you are running in the subprocess? If so, wouldn't you need to specify the full path to this file?

Are arg1 and arg2 variables from the primary script that you are sending to the subprocess script? If so, what is the syntax to get the subprocess script to accept these arguments? I ask because I am using arcpy.GetParameterAsText(0) to evaluate the single argument that I am trying to pass.

Your assistance is greatly appreciated. Thanks.

ChrisFrost

Chris:

I tried running the first portion of your code, but the subprocess is not executing.

Here is my syntax to call the subprocess script:

cmd = "C:\\Python26\\ArcGIS10.0\\python.exe \\\\tgapparc01\\c$\\GIS Testing\\ReplaceSource.py full_path"

In the cmd line, full_path is the file that has the focus. File is stored in the full_path variable.

In the ReplaceSource.py script, I am trying to use the arcpy.GetParameterAsText(0) to bring in this argument. I'm not sure if this is the problem?

Could this issue possibly be how I am calling out the python executable and the python script that gets run in the subprocess?

Your assistance is greatly appreciated. Thanks.

I believe the problem is with your command string. The space in the path to the python script causes it to be treated as two arguments. So python.exe only gets
"\\\\tgapparc01\\c$\\GIS" as the script argument. If you cannot change the directory name, then you might try putting the full path to the python script in single quotes.

mvolz47

Chris:

I tried running the first portion of your code, but the subprocess is not executing.

Here is my syntax to call the subprocess script:

cmd = "C:\\Python26\\ArcGIS10.0\\python.exe \\\\tgapparc01\\c$\\GIS Testing\\ReplaceSource.py full_path"

In the cmd line, full_path is the file that has the focus. File is stored in the full_path variable.

In the ReplaceSource.py script, I am trying to use the arcpy.GetParameterAsText(0) to bring in this argument. I'm not sure if this is the problem?

Could this issue possibly be how I am calling out the python executable and the python script that gets run in the subprocess?

Your assistance is greatly appreciated. Thanks.

ChrisFrost

 
import subprocess

procs = []

cmd = "python foo.py arg1 arg2"
proc = subprocess.Popen(cmd,shell=True)
procs.append(proc)

you will probably want to keep track of the number of running processes and limit how many are run at once.

 
def numRunningProcs(procs):
    numProcs = 0
    for proc in procs:
        proc.poll()
        if proc.returncode == None:
            numProcs = numProcs + 1
        
    return numProcs

Rejoindre la conversation

Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.

Liens rapides

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.