topic Re: Using multiple CPUs in Python script in Python Questions

Using multiple CPUs in Python script

CarstenSchuermann — Sun, 31 Oct 2021 07:52:44 GMT

Hi, I have developed a Python script that iterates through a list of feature classes, in order to apply the same series of geoprocessing tools upon each feature class. In the script, I implemented it like this:

fclist = ("fc1","fc2","fc3", ...")

for i in fclist:

arcpy.management.AddField(i, ...)

arcpy.management.CalculateField(i, ...)

...

It works fine, but I am wondering whether it is possible to speed up processing time by telling Pythin to use not just one but multiple CPUs (for example, use 50% of the available CPUs), i.e. how can I tell Python to perform the "for" loop in parallel?

As the result of the "for" loop for one feature class is not input to or dependend on the results of the other feature classes, I think parallel processing should be possible.

All feature classes are located in the same feature datasets, so I am furthermore wondering whether I am running into "system lock" problems if I am going to parallise this processing?

I appreciate of someone could share his experiences with multi-core / multip-CPU processing and could provide some code snippets.

Re: Using multiple CPUs in Python script

DanPatterson — Sun, 31 Oct 2021 10:55:20 GMT

Did you see the documentation on cpu and gpu support?

Processor Type (Environment setting)—ArcGIS Pro | Documentation

Parallel Processing Factor (Environment setting)—ArcGIS Pro | Documentation

env—ArcGIS Pro | Documentation

Re: Using multiple CPUs in Python script

DonMorrison1 — Sun, 31 Oct 2021 12:37:11 GMT

I've done quite a lot of this and from what you describe it should work with no problem using the python multiprocessing package. The pseudocode below shows the basic flow. As long as each process only touches its own feature class there should be no locking problem. My computer has 8 threads and I usually max it out at 8, there is some odd pleasure in seeing the CPU meters all running at 100%. Throughput increases typically about 5 times what you get with a single processor. Even at 100% I can still use my system to do lightweight interactive work (eg. browsing web sites).

import multiprocessing p = multiprocessing.Pool(<number of processors>) p.map(<target function>, <list of feature class names>) p.close() def <target_function> (feature_class_name): ... your code to process one feature class return

One final note - I HAVE had some problems to get this to work running in a python toolbox, but no problem in a standalone script.

And another final note - you need to be careful with naming conflicts if you are creating temporary feature classes as part of the processing. I usually solve this by appending something like the process ID to the name of the temporary object

Re: Using multiple CPUs in Python script

Anonymous User — Sun, 31 Oct 2021 13:33:38 GMT

Mine is similar to DonMorrisons, but also assigns the pythonw.exe to use and returns some info from the tasks:

def function_To_Process(fc, etc args): try: # Do stuff to the fc return {'result': True} except Exception as ex: return { 'result': False, 'error': ex} if __name__ == '__main__': ctx = multiprocessing.get_context('spawn') ctx.set_executable(r'path to your pythonw.exe') # create list of fc's/args etc that you want processed jobs = [(fc1,args), (fc2, args), (fc3, args), etc...)] cpuNum = ctx.cpu_count() # throttle cpu to 80% if wanted # cpuNum = int(math.ceil(cpuNum * (80 / 100))) with ctx.Pool(processes=cpuNum) as pool: res = pool.starmap(function_To_Process, jobs) if len(res) > 0: for result in res: print(f'Failed: {result.get("error", "None this time")}')

Re: Using multiple CPUs in Python script

CarstenSchuermann — Wed, 03 Nov 2021 06:14:46 GMT

Hi Don, thanks for the pseudo code and the hint about potential naming conflicts with temporary feature classes. I will try it out. So far I have no intention to develop a python toolbox, a script should be perfect for my purposes.

Re: Using multiple CPUs in Python script

CarstenSchuermann — Wed, 03 Nov 2021 06:16:41 GMT

Thank you for the additional remarks, Jeff.