Using multiple CPUs in Python script

Question

Hi, I have developed a Python script that iterates through a list of feature classes, in order to apply the same series of geoprocessing tools upon each feature class. In the script, I implemented it like this:

fclist = ("fc1","fc2","fc3", ...")

for i in fclist:

arcpy.management.AddField(i, ...)

arcpy.management.CalculateField(i, ...)

...

It works fine, but I am wondering whether it is possible to speed up processing time by telling Pythin to use not just one but multiple CPUs (for example, use 50% of the available CPUs), i.e. how can I tell Python to perform the "for" loop in parallel?

As the result of the "for" loop for one feature class is not input to or dependend on the results of the other feature classes, I think parallel processing should be possible.

All feature classes are located in the same feature datasets, so I am furthermore wondering whether I am running into "system lock" problems if I am going to parallise this processing?

I appreciate of someone could share his experiences with multi-core / multip-CPU processing and could provide some code snippets.

CarstenSchuermann · Answer

Thank you for the additional remarks, Jeff.

CarstenSchuermann · Answer

Hi Don, thanks for the pseudo code and the hint about potential naming conflicts with temporary feature classes. I will try it out. So far I have no intention to develop a python toolbox, a script should be perfect for my purposes.

Anonymous User · Answer

Mine is similar to DonMorrisons, but also assigns the pythonw.exe to use and returns some info from the tasks:

def function_To_Process(fc, etc args):
    try:
        # Do stuff to the fc
        return {'result': True}

except Exception as ex:
        return { 'result': False, 'error': ex}

if __name__ == '__main__':
    ctx = multiprocessing.get_context('spawn')
    ctx.set_executable(r'path to your pythonw.exe')

# create list of fc's/args etc that you want processed
    jobs = [(fc1,args), (fc2, args), (fc3, args), etc...)]

cpuNum = ctx.cpu_count()
    # throttle cpu to 80% if wanted
    # cpuNum = int(math.ceil(cpuNum * (80 / 100)))

with ctx.Pool(processes=cpuNum) as pool:
        res = pool.starmap(function_To_Process, jobs)

if len(res) > 0:
        for result in res:
            print(f'Failed: {result.get("error", "None this time")}')

r