topic Re: Path Distance erratic and crashes in a loop in Python Questions

Path Distance erratic and crashes in a loop

MLLO — Sat, 11 Dec 2021 07:55:25 GMT

Hi,

I am using arcgis 10.5.1 64 bit.

I have a set of 10 point features that I am trying to connect using PathDistance. Each point has an id, say 0...9. Effectively I want to generate the path from 0 to 1, 1 to 2, and so on.

I am using both my own tables to define vertical and horizontal factors. I have created these paths manually using the exact same factors. So I know those work.

The two main files I am using are: 1) a sample DEM (5m resolution, tif) called test_dem.tif 2) a point shapefile test_loc.shp which contains 11 points with several columns but the main one used here is 'Id' column. These files and others used for the script can be found below.

The files used for the vertical and horizontal factors are the following,

hsine.txt	*vertical3.txt*
0 1 10 1.02 20 1.06 30 1.13 40 1.23 50 1.36 60 1.5 70 1.66 80 1.83 90 2 100 2.17 110 2.34 120 2.5 130 2.64 140 2.77 150 2.87 160 2.94 170 2.98 180 3	-90 20000 -80 5000 -70 1500 -60 300 -50 100 -40 40 -30 12 -20 3 -10 1 0 3 10 7 20 14 30 24 40 40 50 75 60 180 70 750 80 1500 90 5000

I have the following code:

import arcpy
import arcpy.sa as sa
import arcpy.management as mng
from math import pi

# Checking out SA extension
class SAerror(Exception):
    pass
try:
    if arcpy.CheckExtension('Spatial') == 'Available':
       arcpy.CheckOutExtension('Spatial')
    else:
       raise SAerror
except SAerror:
    print('Spatial Analyst license is unavailable!')
except arcpy.ExecuteError:
    print(arcpy.GetMessages(2))

# ArcGIS directories
arcpy.env.scratchWorkspace = r'C:\GIS\projects\network\potential\SCRATCH'
arcpy.env.workspace = r'C:\GIS\projects\network\potential'

# input rasters
rDEM = arcpy.Raster(r'C:\GIS\projects\network\potential\tst_dem.tif')
shpFeat = r'C:\GIS\projects\network\potential\testloc.shp'

# PARAMETERS
deg2rad = pi/180.0
rad2deg = 180.0/pi
wFactor = 0.2
I = 4.0
wPthPot = 0.75
Gmax = 5.0

# path distance parameters
rCost = "" # use unit
rSurface = rDEM
rVertical = rDEM
hCostfn = r'C:\GIS\projects\network\potential\hsine.txt'
HF = sa.HfTable(hCostfn)
vCostfn = r'C:\GIS\projects\network\potential\vertical3.txt'
VF = sa.VfTable(vCostfn)
maxDist = ""
nbrfn = r'C:\GIS\projects\network\potential\expd.txt'

#Set geoprocessing environmental preferences
arcpy.env.extent = rDEM.extent
arcpy.env.cellSize = rDEM.meanCellHeight
arcpy.env.overwriteOutput= True

# INITIAL CONDITIONS
rTPot = rDEM / rDEM.maximum
rDirRel = rTPot
rG0 = sa.CreateConstantRaster(0.0, 'FLOAT', arcpy.env.cellSize, arcpy.env.extent)
rGmax = sa.CreateConstantRaster(Gmax, 'FLOAT', arcpy.env.cellSize, arcpy.env.extent)
rGt_1 = sa.CreateConstantRaster(0.0, 'FLOAT', arcpy.env.cellSize, arcpy.env.extent)

# sample sequence
pthseq = list(range(11))

# create a feature layer
mng.MakeFeatureLayer(shpFeat, 'lyr')

# MAIN
for o,d in zip(pthseq[:-1], pthseq[1:]):

    # lambda direction
    rDir = sa.Aspect(rDirRel)

    # CALCULATE PATH
    
    mng.SelectLayerByAttribute('lyr', "NEW_SELECTION", ' "Id" = '+str(o)) # origin

    # accumulated cost surface
    rAcs= sa.PathDistance('lyr', "", rSurface, rDir, HF, rVertical, VF, "", 'rBL.tif')

    mng.SelectLayerByAttribute('lyr', "NEW_SELECTION", ' "Id" = '+str(d)) # destination

    # path
    rPath = sa.CostPath('lyr', rAcs, 'rBL.tif', "EACH_CELL")
    rPath = sa.Con(sa.IsNull(rPath), 0, 1)
    pathFn = 'path'+str(o)+'_'+str(d)+'.tif'
    rPath.save(pathFn)

    rW = (rG0 - rGt_1) * wFactor
    rMark = rPath * (1.0 - (rGt_1 / rGmax)) * I
    rGt = rGt_1 + rW + rMark
    rGt.save('rGt.tif')

    # DECAY
    rPthPot = sa.FocalStatistics(rGt, sa.NbrWeight(nbrfn),'SUM')
    rPthPot = rPthPot.maximum - rPthPot
    rPthPot = rPthPot / rPthPot.maximum

    # Update directional relation
    rDirRel = rPthPot * wPthPot + rTPot * (1.0 - wPthPot)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

    # Update rGt_1  *** missing earlier ***  
    rGt_1 = rGt‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Overall, I get very weird behavior. Some times, it will fail when trying to generate the backlink. In other occasions, it will calculate the first path and then fail. It will calculate three paths and fail. Only once it has ran through the entire sequence. After crashing I make sure to delete all possible files (usually the backlink file is created but it is corrupted).

I have tried many things:

1. Creating a fresh feature layer and erasing it at the end of each iteration.

2. Generating with each iteration different feature layers, one for the origin and another for the destination, and deleting them after each iteration.

3. I have tried saving to the disk the layer feature as well as all the other rasters (I set arcpy to overwrite).

4. I have tried saving to the disk the backlink

5. I have used relative and absolute filenames.

6. I am using only .tif files

7. I have use try...except to catch information about the error with no success

I have spent much time reading through the different forums, etc. I know there were problems with this method in previous iterations of the software. The lack of any feedback relating to possible errors is a real problem! Often it will be python that crashes all together. It is very finicky, there are certain Horizontal Factor settings (even those found in the help pages) that will not generate a complete cost accumulation raster. This is not my case though.

My question is whether there is something that I am missing, something that needs to be reset or deleted? or perhaps someone can suggest how to find more information after the procedure fails???

thank you,

Marc

Re: Path Distance erratic and crashes in a loop

DanPatterson_Retired — Wed, 25 Oct 2017 19:50:55 GMT

Your script doesn't set any environment factors such as extent, cell size, snap raster etc. Try to explicitly set those prior to working with rasters to remove the potential for erratic behaviour, particularly if you are experiencing it between runs of the same code.

Re: Path Distance erratic and crashes in a loop

MLLO — Wed, 25 Oct 2017 20:06:53 GMT

Thanks,

Indeed, all of that was done previously. All environmental variables and working directories were set up before creating anything or making any selection.

Re: Path Distance erratic and crashes in a loop

DanPatterson_Retired — Wed, 25 Oct 2017 20:15:45 GMT

So to clarify, there are parts of your script not present? Global settings in arcmap need nor persist even within a session.

Re: Path Distance erratic and crashes in a loop

MLLO — Wed, 25 Oct 2017 20:28:51 GMT

Yes... correct I did not include the rest of the code. Up to this point it had to do with several settings. Everything runs fine up to there (I have check this several times) it is only when I get into the loop that I encounter the behavior I described. Some times it runs fine other it does not.

Could you elaborate on what you mean that 'Global settings in arcmap need nor persist...'. Presumably if I define environmental and set overwrite to True within a script, those will be preserved,no?

Re: Path Distance erratic and crashes in a loop

DanPatterson_Retired — Wed, 25 Oct 2017 21:25:11 GMT

No... things like

But check each tools' script and Environments section (bottom of each page) to see which environments are supportes

Re: Path Distance erratic and crashes in a loop

MLLO — Wed, 25 Oct 2017 22:24:03 GMT

Did that... is there any known memory bug with this routine? I just cannot understand why is so unpredictable!

Re: Path Distance erratic and crashes in a loop

DanPatterson_Retired — Wed, 25 Oct 2017 22:27:22 GMT

Esri support... PathDistance

and PathDistance in general in Esri Support

Re: Path Distance erratic and crashes in a loop

MLLO — Wed, 25 Oct 2017 22:41:05 GMT

Thanks... there does not seem to be anything there that seems to point to any causes or solutions. Thanks nonetheless.

Re: Path Distance erratic and crashes in a loop

XanderBakker — Wed, 25 Oct 2017 23:00:27 GMT

I think first it would be best to share the entire script, since the error may be caused somewhere else. Also when sharing code use instructions like those described here: Posting code with Syntax Highlighting on GeoNet

I just copied you snippet of code and several syntax errors occurred:

there is an additional quote on this line:
- rAcs= arcpy.PathDistance_spatial('lyr', rCost, rSurface , rDir, HF, rVertical, VF, '"", 'BL.tif')
a space between the arcpy. and CostPath:
- rPath = arcpy. CostPath_spatial('lyr', rAcs, 'BL.tif', "EACH_CELL")

I'm sure those are not the errors that are causing your problems (since the code would not execute with those errors) but it is important to understand that it will make it much more difficult to help in case you don't share the actual code.

In case possible, share a part of your data too so we can see if the error or any strange behavior can be reproduced.

Re: Path Distance erratic and crashes in a loop

MLLO — Wed, 25 Oct 2017 23:33:12 GMT

Thanks Xander... I will repost with all the script and the data. keep tuned

Re: Path Distance erratic and crashes in a loop

MLLO — Wed, 25 Oct 2017 23:39:16 GMT

Just reposted the script.

Re: Path Distance erratic and crashes in a loop

DanPatterson_Retired — Wed, 25 Oct 2017 23:45:31 GMT

Xander... I will leave it to you now that the script has appeared

Re: Path Distance erratic and crashes in a loop

XanderBakker — Thu, 26 Oct 2017 02:56:45 GMT

Hi Marcos Llobera ,

Thanks for sharing the code, that helps, but I am wondering... is this really the code you used? I'm asking since lines 11, 12 and 13 are missing an indentation and line 80 still has a space between "sa." and "CostPath".

It will really be necessary to have access to the data you are using, in the first place to help understand the code and in the second place to see if strange behavior might be related to the data.It is weird though that running the same code on the same data has different results (crashes).

Dan Patterson , don't go, you will miss out on all the fun.

Re: Path Distance erratic and crashes in a loop

NeilAyres — Thu, 26 Oct 2017 06:33:02 GMT

And, please a full description of the input Dem. Is it in projected space?

Re: Path Distance erratic and crashes in a loop

MLLO — Thu, 26 Oct 2017 16:55:25 GMT

When pasting the code all indentation was lost. So I manually had to indent using spaces. I am happy to share the data (it is just for testing) but I am not familiar with the procedure to post it. If you let me know how to do that I will.

Re: Path Distance erratic and crashes in a loop

MLLO — Thu, 26 Oct 2017 17:01:25 GMT

xander_bakker‌

Found how to post the data. So, you should have everything to run. Let me know if there is anything else.

Re: Path Distance erratic and crashes in a loop

XanderBakker — Thu, 26 Oct 2017 17:03:31 GMT

Hi M LLO ,

You should open the thread: Path Distance erratic and crashes in a loop and when you create a comment you will see in the upper right corner the link "use advanced editor":

In the advanced editor in the lower left corner you will find a link to attach data to the thread:

Re: Path Distance erratic and crashes in a loop

DanPatterson_Retired — Thu, 26 Oct 2017 17:34:51 GMT

Code formatting.... the basics++ python obviously, to get syntax and highlighting right

Re: Path Distance erratic and crashes in a loop

XanderBakker — Sat, 11 Dec 2021 07:55:28 GMT

So I tried to run the code and a number of errors occurred. The first one was with the Focal Statistics using the weight kernel from the file EXPD.TXT. Although it seems perfectly OK, it throws an error:

To be sure it wasn't any problem with the path I changed the location of the file, but with no luck:

The overwrite setting is not working very well, since when running the code again it tells me this:

I had to close my IDE to be able to delete the previous files. So I decided to throw in a different Focal Statistics operation, which is obviously not the same, but something the allows the code to continue:

rPthPot = sa.FocalStatistics(rGt, sa.NbrCircle(20, 'CELL'), 'MEAN', 'DATA')‍

The additional DATA parameter might be good to include in your code to reduce NoData values in the output. After this the code continued, but returned another error:

While inspecting "rPthPot.maximum" I noticed that the entire raster contains NoData values. So I went back and tried to trace where that raster is based on. When I look at this parte of the code:

    rW = (rG0 - rGt_1) * wFactor  # (0 - 0) * 0.2 = 0
    rMark = rPath * (1.0 - (rGt_1 / rGmax)) * I  # 1 * (1-(0/5)) * 4 = 4 or 0
    rGt = rGt_1 + rW + rMark  # 0 + 0 + 4 or 0 = 4 o 0
    rGt.save('rGt.tif')‍‍‍‍

You can see that rW is always 0 and rGt_1 is 0, so rGt is actually rMark and that raster contains value 4 for the route and value 0 for all other pixels. When this raster is used in the calculation it yields an error since it expects a raster or layer and None does not comply to that, since rPthPot.maximum is None.

I switch off that part in case the rPthPot.maximum is None and let the code run again and this time it finished the first iteration but came up with this error:

The raster contains all NoData cells and can't be used for the Path Distance. So maybe you can functionally describe what you are trying to do, since there are a number of things in the code that don't make sense to me.