Hi all,
Based on this Esri document (Redirect where ArcGIS Pro writes temporary or scratch files during processing), it seems like Pro's geoprocessing tools write temporary files to the path specified by the ARCTMPDIR environment variable. Failing that, it uses the paths in the %TMP% and %TEMP% environment variables. In our environment, all three of these environment variables point to locations on the C: drive of the Azure VMs we work on.
Can anyone confirm that this holds true for deep learning tools, particularly the Train Deep Learning Model tool? Are any other large temporary files written anywhere during the training process (like in the tool's output folder)? This does not appear to be the case, aside from the relatively small checkpoint files that appear to be written there for the most recently completed epoch (mine total about 181 MB).
We somehow managed to overwhelm our company's network cloud storage system recently and we're trying to troubleshoot the issue. I had been training a model for 24 hours when this happened and caused me to lose all my progress (the model failed with a "no space left on device" error, presumably when it tried to update the checkpoint file), so naturally I'd like to know if this caused the issue or if we can eliminate model training as a potential cause.
Thanks in advance!
- Holly
