does the in_memory workspace be mutually influenced in every subprocess in multiprocess?

pyfans — Fri, 05 Mar 2021 03:07:45 GMT

I have write a py script with multiprocess, like this:

import arcpy

from multiprocessing import Pool

def worker(fc):

arcpy.featureclasstofeatureclass(fc,r"in_memory","tempfc")

## do something for tempfc

arcpy.delete_management("in_memory")

if __name__=="__main__":

p=Pool(3)

fcs=..... ## get some featureclasses

p.map(worker,fcs)

p.close()

p.join()

I want to know if the created "tempfc" and deleted in_memory workspace will be mutually influenced in different subprocess?

Re: does the in_memory workspace be mutually influenced in every subprocess in multiprocess?

JoshuaBixby — Fri, 05 Mar 2021 17:58:25 GMT

Processes within a pool don't share memory unless objects are passed as specific multiprocessing data types, e.g., multiprocessing.Array . Since you are not doing that in your code (I am not even sure any ArcPy objects can be passed that way), the in-memory workspace in each worker process is independent of the others.

An example:

from multiprocessing import Pool import arcpy import time i = 0 def f(x): global i i += 1 arcpy.CreateTable_management("in_memory", "xx{}".format(i)) time.sleep(1) arcpy.env.workspace = "in_memory" return arcpy.ListTables() if __name__ == '__main__': with Pool(4) as p: print(p.map(f, range(6)))

Yields:

[['xx1'], ['xx1'], ['xx1'], ['xx1'], ['xx1', 'xx2'], ['xx1', 'xx2']] >>>

As you can see, each of the four worker processes is able to create the same-named table in its in-memory workspace, and each arcpy.ListTables() only sees the tables in its own in-memory workspace.

topic Re: does the in_memory workspace be mutually influenced in every subprocess in multiprocess? in Python Questions

does the in_memory workspace be mutually influenced in every subprocess in multiprocess?

Re: does the in_memory workspace be mutually influenced in every subprocess in multiprocess?