Where do deep learning processes write temporary files?

[HollyTorpey_LSA]

Hi all,

Based on this Esri document (Redirect where ArcGIS Pro writes temporary or scratch files during processing), it seems like Pro's geoprocessing tools write temporary files to the path specified by the ARCTMPDIR environment variable. Failing that, it uses the paths in the %TMP% and %TEMP% environment variables. In our environment, all three of these environment variables point to locations on the C: drive of the Azure VMs we work on.

Can anyone confirm that this holds true for deep learning tools, particularly the Train Deep Learning Model tool? Are any other large temporary files written anywhere during the training process (like in the tool's output folder)? This does not appear to be the case, aside from the relatively small checkpoint files that appear to be written there for the most recently completed epoch (mine total about 181 MB).

We somehow managed to overwhelm our company's network cloud storage system recently and we're trying to troubleshoot the issue. I had been training a model for 24 hours when this happened and caused me to lose all my progress (the model failed with a "no space left on device" error, presumably when it tried to update the checkpoint file), so naturally I'd like to know if this caused the issue or if we can eliminate model training as a potential cause.

Thanks in advance!

[RichardDaniels]

You are correct, by default DeepLearning tools will default to your current project location and locations set by your Environment variables. It is important to consider disk drive space and location to read or write files to speed up the training process. For example, I recommend always setting the output file location to a different DRIVE than the Source location for your learning chips/training samples. Secondly, remember training can take hours if not days, so anything you can do to speed up the process should be considered.

[RichardDaniels]

Take a look at setting the ARCTMPDIR Environment Variable (or resetting the TEMP variable, which may already exist on most Windows machines). If not set temp files are saved to your default Windows User Profile Location. Support for this var was ported from ArcMap and first implemented in ArcGIS Pro at   version 3.1. For details see How To: Redirect Where ArcGIS pro Writes Temporary or Scratch Files during Processing.