Python - arcpy - multiprocessing

GregHorne — Fri, 09 May 2025 15:39:17 GMT

I have a number of Python/arcpy applications that use multiprocessing. Not all arcpy commands like to be multiprocessed but some do work. For example, I can create multiple processes that all "append" into the same SQLServer table from different sources simultaneously. With ArcGIS Pro v3.3.2 I am finding Python /arcpy applications that multiprocess that used work no longer do. The applications worked with v3.2.3.

Is anybody else running into such issues?

Test code below to demonstrate issue. I have installed ArcGIS Pro v3.3.2. The results can vary.

- Run as is. The results should produce "<-- did something ..." statement 5 times. This should work.
- Uncomment line #2 (from arcpy import Exists). Run app. I start to see issues here in that I no longer get 5 statements. Other errors can appear.
- Uncomment line #3 (time.sleep(15)). Run app. Maybe this line helps sometimes\?
- Uncomment line #9 (print(Exists("c:\dev\test-upgrade")). Run app. Errors.
- Uncomment line #24 (# time.sleep(10)). Run app. In the past, I have used delays to get the new processes to spawn properly.

---------- ---------- ---------- ----------
import time
# from arcpy import Exists
# time.sleep(15)
from multiprocessing import Process
import random

def do_something(title):
print(" ---> do something", title)
# print(Exists("c:\work\a-file-name"))
time.sleep(random.randint(5, 15))
print(" <--- did something", title)

def main():
to_process = [ "Albany", "Boston", "Chicago", "Detroit", "Emporia" ]
procs = []

for item in to_process:

print(" ===> process spawn: ", item)
proc = Process(target = do_something, args = (item, ))
proc.start()
procs.append(proc)
# time.sleep(10)
for proc in procs:
proc.join()

if __name__ == '__main__':
main()
---------- ---------- ---------- ----------

Re: Python - arcpy - multiprocessing

GregHorne — Mon, 12 May 2025 17:47:13 GMT

Solved. The thing that was causing all of my wacky multiprocessing issues is ---> Cisco Secure Endpoint (formerly AMP for Endpoints)

Re: Python - arcpy - multiprocessing

JoshuaBixby — Wed, 14 May 2025 14:40:29 GMT

Thanks for sharing what you found. It used to just be antivirus software that could cause issues, but the breadth and depth of security software has exploded over the past decade, especially in corporate environments. Often times the big-brother software impacts performance, but occasionally it can break functionality as well.

topic Python - arcpy - multiprocessing in Python Questions

Python - arcpy - multiprocessing

Re: Python - arcpy - multiprocessing

Re: Python - arcpy - multiprocessing