topic Re: Optimal Path As Line - issues with memory leak and Python environment in ArcGIS Spatial Analyst Questions

Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Sun, 18 Feb 2024 20:13:57 GMT

I am having issues with the OptimalPathAsLine tool in ArcPy as well as in ArcGIS Pro. My ArcPy workflow, which I run in a cloned environment in PyCharm Community Edition, takes point features and a cost surface as input and runs DistanceAccumulation and OptimalPathAsLine iteratively for the annual timesteps 2008 - 2023.

Issue 1: [SOLVED by bugfix in ArcGIS Pro v3.2.2]

It all worked well, until I decided to transform the scale of the cost surface input from a range of ~33.8 - ~930.2 to 1 - 100 using the Rescale by Function tool (linear function) (EDIT: Raster Calculator also tested with same result). After I had changed the cost surface input to my script, OptimalPathAsLine stopped working for two specific timesteps (two specific combinations of input rasters and destination features): It threw no error, but the calculation would never end and memory consumption kept increasing until I stopped the script.

The behaviour is reproducible by manually running the tool with the same inputs in ArcGIS Pro (EDIT: see screenshots). I did not find any difference between the old and new accumulated cost rasters (both showing 32 bit float as pixel type).

Issue 2: [SOLVED by avoiding int64 field type in the input feature class]

Then I updated ArcGIS Pro from 3.1 (or any 3.x, not sure) to 3.2 in an attempt to fix issue 1. Since then, OptimalPathAsLine in my script does not work anymore. It always throws the following error:

arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 010778: Big Integer field type is currently not supported.
Failed to execute (OptimalPathAsLine).

I have no idea why I get this error message and to which data and field it refers to.

EDIT: I found out that issue 2 has nothing to do with the rasters, but is caused by the destination feature class: The destination field is a Big Integer. My script imports this feature class from a GeoPackage into a FileGDB using CopyFeatures_management. All integers in the GeoPackage are saved as int64. Since the update to ArcGIS Pro 3.2, these fields are, not surprisingly, imported as Big Integer. I am not sure why the script worked earlier - if CopyFeatures converted int64 to something supported by ArcGIS, or if the GeoPackage changed. Anyway, issue 2 is solved by saving int32 to the GeoPackage.

Re: Optimal Path As Line - issues with memory leak and Python environment

DavidPike — Sun, 03 Dec 2023 12:22:26 GMT

Could it just be a field type change from when you rescaled by function? Have you checked the field types and precision/scale of your inputs?

Re: Optimal Path As Line - issues with memory leak and Python environment

DanPatterson — Sun, 03 Dec 2023 13:50:45 GMT

Rescale by Function (Spatial Analyst)—ArcGIS Pro | Documentation

has a number of warnings regarding raster statistics and potential environment settings that should/could be set. I would examine those regarding the reclass by function, which may also lead to the invalid field type

Re: Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Sun, 03 Dec 2023 20:32:40 GMT

I compared the pixel type in the properties, all inputs (cost surfaces and accumulated cost surfaces) have 32bit float, and I ran Get Raster Properties to check "cell value type", which also returned "floating point" for all inputs. Is there anything else I can check?

Re: Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Sun, 03 Dec 2023 20:59:48 GMT

Thank you for the link. I read the raster statistics can be calculated ad-hoc by pressing the button in the tool (which is what I did), but I do not understand potential issues caused by this tool. To exclude it as a potential root cause, I tested re-scaling the values using Raster Calculator and recreating an affected accumulated cost raster and back direction raster from it (manually in ArcGIS Pro).

But still, specific combinations of inputs lead to the endless cell tracing and memory leak. For example, acc.cost surface 2013 + back dir. raster 2013 + destination points 2014. If I swap the back dir. raster with the one from 2016 (but e.g. 2011 will not work neither) or swap the destination points with the ones from 2015, the tool runs successfully.

Re: Optimal Path As Line - issues with memory leak and Python environment

DanPatterson — Sun, 03 Dec 2023 21:46:44 GMT

It is these comments in the link

You may not have valid statistics for the following conditions:

The environment processing extent, cell size, or snap raster is set prior to launching the tool dialog box.
The input raster does not have statistics.

It would be prudent to ensure that you have these set in the Environments tab for the tool.

I would expect that the extent, cell size would be the same for the inputs and that the snap raster would be common about them to avoid the unnecessary work of aligning cells.

Various topics related to these concerns can be found on the left of the help pane beginning with

Cell Alignment (Environment setting)—ArcGIS Pro | Documentation

Re: Optimal Path As Line - issues with memory leak and Python environment

ElizabethGraham — Sun, 03 Dec 2023 23:46:34 GMT

Hello,

Thank you for posting this issue. I am a Product Engineer on the Spatial Analyst team and would like to help investigate the problem. Please email me directly, hopefully, you can share a reproducible case and at least a subset of your data.

Thanks,

Liz Graham (egraham@esri.com)

Re: Optimal Path As Line - issues with memory leak and Python environment

JamesTenbrink — Mon, 04 Dec 2023 00:00:09 GMT

Hi Christa:

Regarding your first issue: I interpret your situation to mean that you are rescaling the accumulated distance raster generated by DistanceAccumulation, then using that rescaled version as input to Optimal Path As Line, along with the back direction raster generated from the same Distance Accumulation execution? Theoretically, this might be legal but this has never been tested and I doubt the SA team would want to support this. The back direction and accumulated distance raster are assumed to be generated by the exact same numerical method, so I suspect that the 'aspect' information stored in the back direction raster is now inconsistent with the 'elevation' information stored in the rescaled accumulation raster and this is causing an infinite loop. Can you rescale everything after running optimal path as line?

I recommend submitting a tech support ticket for the second problem.

Regards,

Jim TenBrink (retired)

Re: Optimal Path As Line - issues with memory leak and Python environment

JamesTenbrink — Tue, 05 Dec 2023 16:56:51 GMT

hi Christa. I think I misread your post. Are you transforming the input friction raster or the output accumulation raster? Doing the former should not cause a problem, but note that any friction raster (transformed or not) should not have negative or zero values in it.

regards,

-jt

Re: Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Fri, 08 Dec 2023 17:17:47 GMT

Hi Liz,

I just sent you an e-mail with some data, thanks!

Christa

Re: Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Fri, 08 Dec 2023 20:16:55 GMT

Hi James,

Correct, I rescaled the cost/friction raster, not the outputs of Distance Accumulation. I rescaled to a range of 1-100. Then I re-ran a standalone script to run Distance Accumulation and Optimal Path as Line for all timesteps of my data, and Optimal Path as Line ran into the "endless tracing and memory leak" issue with certain timesteps (input combinations).

Regards,

Christa

Re: Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Sat, 09 Dec 2023 19:08:58 GMT

I just found the cause for the second issue with the Big Integer error message:

The destination field of the destination feature class is a Big Integer. My script imports this feature class from a GeoPackage into a FileGDB using CopyFeatures_management. All integers in the GeoPackage are saved as int64. Since the update to ArcGIS Pro 3.2, these fields are, not surprisingly, imported as Big Integer. I am not sure why the script worked earlier - if CopyFeatures converted int64 to something supported by ArcGIS, or if the GeoPackage changed.

Anyway, issue 2 is solved by saving int32 to the GeoPackage. The script now runs through - until it hits the original, still unresolved issue with the endless tracing and memory leak as in 3.1

Christa

Re: Optimal Path As Line - issues with memory leak and Python environment

EugenioArima — Fri, 26 Jan 2024 14:04:17 GMT

I am also experiencing computational inefficiencies with both the new "Distance Accumulation" function and "Optimal path". One distance accumulation calculation on a 3300x3000 raster took 12 1/2 hours to complete using one single-point origin. The optimal path function is soaking 29 GB of my 32 RAM memory and is also taking forever to compute. Something is not right with these new functions. A request: could you convey to your team to NOT DEPRECATE the former cost accumulation function based on the regular Dijkstra algorithm? We should have the option to choose which accumulation algorithm to use.

Re: Optimal Path As Line - issues with memory leak and Python environment

EugenioArima — Fri, 26 Jan 2024 14:33:28 GMT

Update: the optimal path failed to compute. Returned an error "Unable to allocate memory" It seems that the function is trying to trace paths from all cells to the origin instead of focusing only on the destination point back to the origin.

Re: Optimal Path As Line - issues with memory leak and Python environment

DanPatterson — Thu, 08 Feb 2024 20:27:31 GMT

Some updates on the tool

BUG-000164136 for ArcGIS Pro (esri.com)

The Optimal Path As Line tool can get stuck in an infinite loop

fixed in 3.2.2

Re: Optimal Path As Line - issues with memory leak and Python environment

EugenioArima — Mon, 12 Feb 2024 20:41:20 GMT

@ElizabethGraham my research assistants reported that the 3.2.2 patch did not fully correct the errors described in this blog. The function continues to run out of memory or produce incomplete paths in certain cases.

Re: Optimal Path As Line - issues with memory leak and Python environment

ChristaRohrbach — Sun, 18 Feb 2024 20:08:29 GMT

Hi Dan, thanks for letting me know! After updating to 3.2.2, I just re-ran my script with the same data which caused the tool to run into a memory leak, and can confirm the bugfix works for me.

@EugenioArimaI suggest you open a new thread, as the memory leak issue appears to be fixed, at least the specific bug I was facing.

Re: Optimal Path As Line - issues with memory leak and Python environment

ElizabethGraham — Tue, 02 Apr 2024 21:41:41 GMT

Hi Eugenio,

Please email me directly, hopefully, you can share a reproducible case and at least a subset of your data.

Thanks,

Liz Graham (egraham@esri.com)

Re: Optimal Path As Line - issues with memory leak and Python environment

JoannaMerson2 — Tue, 18 Feb 2025 19:00:59 GMT

I believe I am experiencing this error in 3.4.2.
I have python script that runs the OptimalPathAsLine tool for many OD pairs. It works in some cases, and is stuck for hours and I have to kill the program for others.
@ElizabethGrahamCan I get some support on this?

Re: Optimal Path As Line - issues with memory leak and Python environment

EugenioArima — Wed, 19 Feb 2025 15:34:00 GMT

I have not used this method in a while. I would suggest verifying that you do not have any points over "NODATA values" of your cost (friction) raster.