Optimal Path As Line - issues with memory leak and Python environment

[ChristaRohrbach]

I am having issues with the OptimalPathAsLine tool in ArcPy as well as in ArcGIS Pro. My ArcPy workflow, which I run in a cloned environment in PyCharm Community Edition, takes point features and a cost surface as input and runs DistanceAccumulation and OptimalPathAsLine iteratively for the annual timesteps 2008 - 2023.

Issue 1: [SOLVED by bugfix in ArcGIS Pro v3.2.2]

It all worked well, until I decided to transform the scale of the cost surface input from a range of ~33.8 - ~930.2 to 1 - 100 using the Rescale by Function tool (linear function) (EDIT: Raster Calculator also tested with same result). After I had changed the cost surface input to my script, OptimalPathAsLine stopped working for two specific timesteps (two specific combinations of input rasters and destination features): It threw no error, but the calculation would never end and memory consumption kept increasing until I stopped the script.

The behaviour is reproducible by manually running the tool with the same inputs in ArcGIS Pro (EDIT: see screenshots). I did not find any difference between the old and new accumulated cost rasters (both showing 32 bit float as pixel type).

Issue 2: [SOLVED by avoiding int64 field type in the input feature class]

Then I updated ArcGIS Pro from 3.1 (or any 3.x, not sure) to 3.2 in an attempt to fix issue 1. Since then, OptimalPathAsLine in my script does not work anymore. It always throws the following error:

arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 010778: Big Integer field type is currently not supported.
Failed to execute (OptimalPathAsLine).

I have no idea why I get this error message and to which data and field it refers to.

EDIT: I found out that issue 2 has nothing to do with the rasters, but is caused by the destination feature class: The destination field is a Big Integer. My script imports this feature class from a GeoPackage into a FileGDB using CopyFeatures_management. All integers in the GeoPackage are saved as int64. Since the update to ArcGIS Pro 3.2, these fields are, not surprisingly, imported as Big Integer. I am not sure why the script worked earlier - if CopyFeatures converted int64 to something supported by ArcGIS, or if the GeoPackage changed. Anyway, issue 2 is solved by saving int32 to the GeoPackage.

[EugenioArima]

@ElizabethGraham my research assistants reported that the 3.2.2 patch did not fully correct the errors described in this blog. The function continues to run out of memory or produce incomplete paths in certain cases. 

[ElizabethGraham]

Hi Eugenio,

Please email me directly, hopefully, you can share a reproducible case and at least a subset of your data.  

Thanks, 

Liz Graham (egraham@esri.com)

[JamesTenbrink]

Hi Christa:

Regarding your first issue: I interpret your situation to mean that you are rescaling the accumulated distance raster generated by DistanceAccumulation, then using that rescaled version as input to Optimal Path As Line, along with the back direction raster generated from the same Distance Accumulation execution? Theoretically, this might be legal but this has never been tested and I doubt the SA team would want to support this. The back direction and accumulated distance raster are assumed to be generated by the exact same numerical method, so I suspect that the 'aspect' information stored in the back direction raster is now inconsistent with the 'elevation'  information stored in the rescaled accumulation raster and this is causing an infinite loop. Can you rescale everything after running optimal path as line?

I recommend submitting a tech support ticket for the second problem.

Regards,

Jim TenBrink (retired)

[JamesTenbrink]

hi Christa. I think I misread your post. Are you transforming the input friction raster or the output accumulation raster? Doing the former should not cause a problem, but note that any friction raster (transformed or not) should not have negative or zero values in it.

regards,

-jt

[ChristaRohrbach]

Hi James,

Correct, I rescaled the cost/friction raster, not the outputs of Distance Accumulation. I rescaled to a range of 1-100. Then I re-ran a standalone script to run Distance Accumulation and Optimal Path as Line for all timesteps of my data, and Optimal Path as Line ran into the "endless tracing and memory leak" issue with certain timesteps (input combinations).

Regards,

Christa

[ChristaRohrbach]

I just found the cause for the second issue with the Big Integer error message:

The destination field of the destination feature class is a Big Integer. My script imports this feature class from a GeoPackage into a FileGDB using CopyFeatures_management. All integers in the GeoPackage are saved as int64. Since the update to ArcGIS Pro 3.2, these fields are, not surprisingly, imported as Big Integer. I am not sure why the script worked earlier - if CopyFeatures converted int64 to something supported by ArcGIS, or if the GeoPackage changed.

Anyway, issue 2 is solved by saving int32 to the GeoPackage. The script now runs through - until it hits the original, still unresolved issue with the endless tracing and memory leak as in 3.1

Christa

[DanPatterson · Accepted solution]

Some updates on the tool

BUG-000164136 for ArcGIS Pro (esri.com)

The Optimal Path As Line tool can get stuck in an infinite loop

fixed in 3.2.2

[ChristaRohrbach]

Hi Dan, thanks for letting me know! After updating to 3.2.2, I just re-ran my script with the same data which caused the tool to run into a memory leak, and can confirm the bugfix works for me.

@EugenioArimaI suggest you open a new thread, as the memory leak issue appears to be fixed, at least the specific bug I was facing.