How to use arcpy.env with multithreading?

[JörgEbert]

I have a tool to automate map export to TIFF for thousands of tiles.
The active view (ArcGIS Pro 3.0.3) is exported to three (or more) TIFFs for evry tile in the main thread and in a second thread the TIFFs of a tile are merged and clipped with arcpy.management.clip().
My tool works fine.

But is it possible to clip TIFFs in a secondray thread to TIFFs with custom compression type "JPEG", "PackBits" or "NONE"?

The arcpy.management.clip() tool uses arcpy.env.compression as the compression type of the output raster. (Default LZW for TIFF).
Setting arcpy.env.compression in the secondary thread throws an exception "AttributeError: ERROR 87934" - I think arcpy.env from the main thread cannot be used in a secondary thread.
The arcpy.management.clip() tool in Secondary does not accept arcpy.env from the main thread. It seems that the tool works with the standard arcpy.env/without any settings.

Does anyone have any ideas (other than working with subprocesses)?

[DanPatterson]

from AttributeError: ERROR 87934 replication and multit... - Esri Community

Solved: threading vs multiprocessing for concurrent gp too... - Esri Community

[JörgEbert]

Sorry for my late reaction/response.

I have some trouble with the python environment in ArcGIS Pro.
How to "Run stand-alone scripts" is documeted here: Run stand-alone scripts—ArcGIS Pro | Documentation

• I check the arcpy.env in the Python window of ArcGIS Pro with len(arcpy.env.keys())
  arcpy.env has 68 attributes
• I open the ArcGIS Python environment via Windows and check arcpy.env
  c:\Progra~1\ArcGIS\Pro\bin\Python\scripts\propy.bat
  arcpy.env has 68 attributes - okay
• I open a subprocess from the Python window of ArcGIS Pro and check arcpy.env in the new subprocess
  import suprocess
  sp = subprocess.Popen([r'c:\Progra~1\ArcGIS\Pro\bin\Python\scripts\propy.bat'])
  arcpy.env has 68 attributes - okay
• I open a subprocess via a toolbox script of ArcGIS Pro and check arcpy.env in the new subprocess.
  arcpy.env has only 9 attributes - why?
  >>> arcpy.env.keys()
  ['scratchGDB', 'scratchFolder', 'packageWorkspace', 'scriptWorkspace', 'addOutputsToMap', 'buildStatsAndRATForTempRaster', 'autoCancelling', 'isCancelled', 'overwriteOutput']
  
  get and set the other attributes causes an exception
  >>> arcpy.env.compression
  Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  AttributeError: 'GPEnvironment' object has no attribute 'compression'
  >>> arcpy.env.compression = 'PackBits'
  Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  AttributeError: 'GPEnvironment' object has no attribute 'compression'

I need arcpy.env attributes  'compression', 'pyramid' and 'rasterStatisticsfor'  to merge and clip (with arcpy.management.clip()) thousands of tiles in a second/parallel process.

Does somebody has any idea?

[ShaunWalbridge]

Can you share information about how you're calling the subprocess in your last case? It sounds like this might be something best handled through support to get enough details to understand the nature of the problem, it may be a bug, but also may be some more complicated configuration is required.

[JörgEbert]

I created a simple tool (without parameters).

Running the tool will launch the Python console, just like above from the ArcGIS Pro Python window.
In the Python console I check arcpy.env

Thank you for your efforts

And yes, I also sent this issue to Esri Support yesterday.

[ShaunWalbridge · Accepted solution]

OK, I took a look at this. It looks like the subprocess call is copying the full environment variable table that Pro is using, and that is causing ArcPy to be run as if it were in the application which subsequently fails to load some components of the software (including the ones which include your env properties).

Running this should let you work around this limitation:

import os
import subprocess

cmd_env = os.environ.copy()
if 'GP_PROXY_OPS' in cmd_env.keys():
    cmd_env.pop('GP_PROXY_OPS')
subprocess.run([r"C:\Program Files\ArcGIS\Pro\bin\Python\Scripts\propy.bat"], env=cmd_env)

Once that is set,  the rest of ArcPy should load and you should see the expected full env dictionary.

Cheers, Shaun

[JörgEbert]

Great, it works perfectly
Thanks for your quick analysis and response