How to use arcpy environment settings with python multiprocessing in arcgis pro?

[JörgEbert]

Hi everyone,

I want to use arcpy environment settings with python mutliprocessing, but using attribut 'workspace' raises the exception: "AttributeError: 'GPEnvironment' object has no attribute 'workspace'"

My code snippet from parent script:
mp_exe = os.path.join(sys.exec_prefix, 'pythonw.exe')
mp.set_executable(mp_exe)
with mp.Pool(4) as pool:
    pool.starmap(
        mp_update,
        [("some_sde_connection_file", tbl_name) for tbl_name in tbl_names],
    )

My code snippet from multiprocessing script:

import arcpy

def mp_update(cnn_file, tbl_name):

    arcpy.env.workspace = cnn_file

    pass

I tested bove arcpy.env.workspace and arcpy.EnvManager(Workspace=...).

The exception "AttributeError: 'GPEnvironment' object has no attribute 'workspace'" is always raised wenn das Skript als Skript-Tool in ArcGIS Pro (3.4.5) ausgeführt wird.

But the script executes successfully without an exception in my development environment (PyScripter).

Does anyone have an idea why arcpy environment settings are unavailable during multiprocessing?

[HaydenWelch]

arcpy in general does not play well with multiprocessing since the underlying C code is not written to be threadsafe (sometimes). There's a lot of blocking behavior in the function calls and the library needs to be loaded into each process (something that will often also cause license check errors).

I have had much better luck with the concurrent.futures ThreadPoolExecutor though in the past:

import concurrent.futures as fut

from arcpy.da import SearchCursor

gdb = r'<gdb>\{}'
fcs = ['FC1', 'FC2', 'FC3', 'FC4'] * 5


def read(fc: str) -> list[dict[str, object]]:
    rows = list[tuple[object, ...]]()
    fields = list[str]()
    with SearchCursor(gdb.format(fc), '*') as cur:
        fields = cur.fields
        rows = list(cur)
    return [dict(zip(fields, row, strict=True)) for row in rows]


def mp_run(workers: int = 3):
    with fut.ThreadPoolExecutor(workers, initializer=lambda: globals()) as pool:
        return [
            f.result()
            for f in fut.as_completed(pool.submit(read, fc) for fc in fcs)
        ]


def run():
    return [read(fc) for fc in fcs]

This does not work with everything though. There is also almost no speedup for most operations since the root arcpy process maintains all the locks and won't release them until an operation completes.

[JoshuaBixby]

Using pythonw.exe instead of python.exe eliminates standard communication streams between the parent and child, which commonly applications rely on to pass state and environment information to the spawned children.  I would start by just switching to python.exe.  Since you are running this as a script tool, a console window won't be opened anyways.