I am running a complex multiprocessing workflow that involves segmenting an AOI to run through a complex workflow that requires tools from the spatial analyst extension. Occasionally I get an error where one of the processes hangs stopping the script, and I can trace it despite how much I try to log each process. What is more, the script fails at different points for each run that it fails on. In watching the command prompt window during execution I saw that it said "FATAL ERROR [INFADI] MISSING DIRECTORY."
For each process I create a unique folder that I set the scratch workspace environment variable to. I set the "TEMP" & "TMP" directories to the scratch workspace folder for the process using os.environ. At the end of each process I clean up the scratch workspace and use arcpy.Delete_management("in_memory"). Finally, before I execute each spatial analyst tool I check to see if the dataset I am going to execute on exists using arcpy.Exists, and raise an exception if it doesn't. So far I haven't seen a single exception thrown indicating dataset doesn't exist. I have been multiprocessing with arcpy and spatial analyst for years and have never seen this before.
The machine I am getting the errors on is a AWS EC2 m4.16xlarge running Windows Server 2012 r2, w/ ArcGIS Desktop 10.5 (w/ background processing loaded), utilizing a floating license (which I have checked out for 30 days).
