Multiprocessing with ArcGIS Pro arcpy

ckoenig77 — Wed, 01 May 2024 16:21:37 GMT

I'm experimenting with attempting to use multiprocessing to perform some geoprocessing with a standalone Python script. I have ArcGIS Pro 3.2.2 installed. I'm calling this script in a command window with "C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\propy".

The script will create usually around half of the feature classes without issue, but others fail randomly due to the following errors:

ERROR 000732: Feature Class Location: Dataset C:\Mapping\FileGDB/CIMS_CLU_Test.gdb/FeatDS does not exist or is not supported
ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support
ERROR 160193: This release of the GeoDatabase is either invalid or out of date.
ERROR 160706: Cannot acquire a lock.

I've experimented with decreasing / increasing the processes in the Pool declaration to no avail. Is it realistic to expect this kind of multiprocessing to work with file GDBs or is there a trick I'm missing in the code that will help? Thanks in advance for any suggestions you can provide.

import arcpy from multiprocessing import Pool path = r"C:\Mapping\FileGDB" filename = "Test.gdb" # Set local variables out_path = path + "/" + filename + "/FeatDS" geometry_type = "POLYGON" release = "04_2024" has_m = "DISABLED" has_z = "DISABLED" def createFC(state): try: arcpy.CreateFeatureclass_management(out_path, state + "_" + release, geometry_type, None, has_m, has_z) print(f"{ state } created") except Exception as e: print(f"{ state } error: { e }") def work(): states = [ 'AL', 'AR', 'AZ', 'CA', 'CO', 'DC', 'FL', 'GA', 'IA', 'ID', 'IL', 'KS', 'KY', 'LA', 'MD', 'MI', 'MN', 'MO', 'MS', 'MT', 'NC', 'ND', 'NE', 'NM', 'NV', 'NY', 'OH', 'OK', 'PA', 'SC', 'SD', 'TN', 'TX', 'UT', 'VA', 'WA', 'WI', 'WY', 'AK', 'CT', 'DE', 'HI', 'ME', 'MA', 'NH', 'NJ', 'RI', 'VT', 'WV' ] with Pool(processes=4) as pool: pool.map(createFC, states) pool.join() pool.close() if __name__ == '__main__': work() print('end')

Re: Multiprocessing with ArcGIS Pro arcpy

JoshuaBixby — Wed, 01 May 2024 17:12:38 GMT

With multiple processes accessing the same file geodatabase, especially to make edits, you will run into locking issues. Might be worth reading through: FAQ: How Are the Various Lock Mechanisms Implemented in ArcGIS Enterprise and the Geodatab (esri.com)

Re: Multiprocessing with ArcGIS Pro arcpy

Luke_Pinner — Thu, 02 May 2024 02:13:14 GMT

The usual way is to ensure all sub-processes have their own workspace/gdbs. Your sub-processes can write their FCs to those without any locking issues from other sub-processes. Then the main process can collect all the FCs and copy them to the final output workspace.

Re: Multiprocessing with ArcGIS Pro arcpy

ckoenig77 — Thu, 02 May 2024 16:53:43 GMT

This makes sense thank you for your suggestion!

topic Re: Multiprocessing with ArcGIS Pro arcpy in Python Questions

Multiprocessing with ArcGIS Pro arcpy

Re: Multiprocessing with ArcGIS Pro arcpy

Re: Multiprocessing with ArcGIS Pro arcpy

Re: Multiprocessing with ArcGIS Pro arcpy