So I have some library code that handles loading in file databases and indexing the contained data. There are then grouped out into dictionaries so I iteratively use da.Walk to extract each datatype from the dataset.
This can of course be pretty slow, especially when loading in a database that's on a network fileserver. No problem though, this is something that can be solved pretty simply by creating some threads using concurrent.futures.ThreadPoolExecutor:
from concurrent.futures import ThreadPoolExecutor, as_completed
from arcpy.da import Walk
from pathlib import Path
def walk(ds: str, dtype: str | None = None):
"""walk a dataset filtering on the supplied datatype"""
paths: list[Path] = []
for root, _, items in Walk(ds, datatype=dtype):
for itm in items:
paths.append(Path(root)/itm)
return paths
def extract_types(ds: str, dtypes: list[str]) -> dict[str, list[Path]]:
"""Extract paths from a dataset grouped by type"""
data: dict[str, list[Path]] = {}
with ThreadPoolExecutor(max_workers=len(dtypes)) as executor:
futures = {executor.submit(walk, ds, dtype): dtype for dtype in dtypes}
for future in as_completed(futures):
data[futures[future]] = future.result()
return data
Seems simple enough, spool up one thread per Walk call and await the results so they can be done concurrently, lets run it:
>>> extract_types("My_GDB", ['FeatureClass', 'Table'])
{'FeatureClass': [], 'Table': []}
Hmmm. There's no output, but there is definitely both Tables and Feature Classes in that gdb... Let's try a syncronous extract method:
def extract_types_sync(ds: str, dtypes: list[str]) -> dict[str, list[Path]]:
"""Extract paths from a dataset grouped by type"""
return {
dtype: walk(ds, dtype)
for dtype in dtypes
}
And run that:
>>> extract_types_sync("My_GDB", ['FeatureClass', 'Table'])
{'FeatureClass': [Path("My_GDB/FC1"), Path("My_GDB/FC2")],
'Table': [Path("My_GDB/Table1"), Path("My_GDB/Table2")]}
Okay, so there IS data in the database, and Walk is able to find it. Lets try the concurrent version again:
>>> extract_types("My_GDB", ['FeatureClass', 'Table'])
{'FeatureClass': [Path("My_GDB/FC1"), Path("My_GDB/FC2")],
'Table': [Path("My_GDB/Table1"), Path("My_GDB/Table2")]}
So now the concurrent version is able to find the data, but only **after** running a Walk syncronously? This little bug persists through interpreter sessions it seems. So let's see if warming up the Walk function can fix it:
def extract_types(ds: str, dtypes: list[str]) -> dict[str, list[Path]]:
"""Extract paths from a dataset grouped by type"""
for _ in Walk(ds): break
data: dict[str, list[Path]] = {}
with ThreadPoolExecutor(max_workers=len(dtypes)) as executor:
futures = {executor.submit(walk, ds, dtype): dtype for dtype in dtypes}
for future in as_completed(futures):
data[futures[future]] = future.result()
return data
And run one more time:
>>> extract_types("My_GDB", ['FeatureClass', 'Table'])
{'FeatureClass': [Path("My_GDB/FC1"), Path("My_GDB/FC2")],
'Table': [Path("My_GDB/Table1"), Path("My_GDB/Table2")]}
Now it works! This is really odd though. I'm guessing that da.Walk relies on some global state that isn't initialized in a sub thread and must be initialized in the main thread. This is definitely odd behavior though, and I figured that I'd share it here in case anyone else happens to run into it. I am also curious how this pattern will work when 3.14 is adopted and we have access to the InterpreterPoolExecutor. Will the arcpy global state need to be shared for functions as simple as da.Walk?
