"python.exe has stopped working" on raster.save

1993
5
Jump to solution
04-03-2013 09:11 AM
AustinMilt
New Contributor III
I am attempting to perform raster calculations with in-memory rasters in a script using arcpy (ArcMap v10.0). I get this error from multiple different inputs and it occurs from multiple different areas in my code depending on slight modifications.

The basic workflow is this:

  • load in a raster using arcpy.sa.Raster

  • perform a raster calculation

  • save the raster using raster.save (this is where the error occurs)


For example, I have

# make a raster from the bounding region result = arcpy.PolygonToRaster_conversion(boundingRegion, vField, outputRegion, '', '', cellSize) outputRaster = arcpy.sa.Raster(outputRegion) outputRaster.save('C:/temp/temprast.tif') # no error occurs          # convert any NoData to "outside" the output region outputRaster = arcpy.sa.Con(arcpy.sa.IsNull(outputRaster), rastOutVal, rastInVal) outputRaster.save('C:/temp/temprast.tif') # python crashes with a pop-up window and windows error "python.exe has stopped working"


Sometimes saving the raster has no ill effect. And saving the raster to a different filename (e.g. C:/temp/temprast2.tif) for consecutive calculations also averts the crash. However, the crash that I am concerned about happens when I attempt to save the raster only once and do so at the end of several raster calculations.

I have a couple guesses as to what may be causing the crash.

1) arcpy attempts to write the file before the raster calculation is complete. I dont think this is the case. I have put in a sleep command for 10 seconds, which should be plenty of time. Arcpy would also have to be creating a sub-process that can run behind the main process, which I dont think is happening.

2) arcpy attempts to save a raster with no information or incompatible information that has been produced during calculations. I dont think this is the case, because I can get an output file saved if I save all the intermediate rasters to files before the final one. So unless writing each intermediate raster to a file somehow makes the information compatible with future calculations, this cant be it.

3) In attempting to save the in-memory raster to the hard drive, arcpy is attempting to access/overwrite the same RAM that is currently occupied by the in-memory raster. I dont have a way to test this.

Any suggestions would be a huge help. One boon of arcpy in v10.x is the ability to perform raster calculations in-memory. The only solution I can come up with (save all intermediate rasters as separate files) takes away this advantage.

I've attached a sub-section of the problem script. I cant attach the whole thing because it is part of a package I hope to release in the near future.

UPDATE: I have just discovered another curious way to avoid the problem I was having.
1) If I put in a breakpoint (import pdb; pdb.set_trace) just before the spot where the final raster is to be saved to disk, and
2) then call a different temporary raster that is stored in memory (i.e. >> raster), and finally
3) call the output raster (i.e. >> outputRaster)

then the problem does not happen. The mechanism that seems to be at work here is that when the temporary rasters are accessed in Python (by me), they are written to the hard disk (confirmed by watching files in the temporary directory). This subsequently means all arcpy has to do when the outputRaster.save command is called is change the name/copy the temporary raster to its permanent location.

However, this does not solve my problem! If I try something like ">> print raster" without the break statement, I get the same crash. For some reason I need to be interactively calling the rasters for them to be written to the hard disk.

It also appears the the output raster is wrong with this method.

Hopefully it gives some additional helpful information.

UPDATE 2: Doing the same thing in UPDATE 1, but putting the breakpoint at the beginning of the raster calculations seems to solve the problem of the output being false. I guess there's an issue with raster calculations in memory.
Tags (2)
0 Kudos
1 Solution

Accepted Solutions
AustinMilt
New Contributor III
UPDATE I have gotten my script to work, though I dont really know how. Attached is the updated section of code with fixes. There were several places where things broke for unknown reasons. In each place I tried one of 3 things:

  • save the intermediate raster to the hard drive, then load it back into memory using arcpy.sa.Raster

  • break compound raster calculations into several lines, e.g. Con(IsNull(raster)) becomes r1=IsNull(raster); r2 = Con(r1)

  • create separate temporary file names for rasters that are saved and overwritten multiple times in a loop (i.e. each intermediate raster has its own name)


Different things worked in different places, without any recognizable pattern. As such, I still dont know what was causing Python to crash. My best guess is that arcpy was attempting to overwrite a raster that was in memory with an updated version of itself, though this still would explain why it crashed on raster.save.

Anyone who can shed light on this, it would be greatly appreciated.

View solution in original post

0 Kudos
5 Replies
AustinMilt
New Contributor III
UPDATE I have gotten my script to work, though I dont really know how. Attached is the updated section of code with fixes. There were several places where things broke for unknown reasons. In each place I tried one of 3 things:

  • save the intermediate raster to the hard drive, then load it back into memory using arcpy.sa.Raster

  • break compound raster calculations into several lines, e.g. Con(IsNull(raster)) becomes r1=IsNull(raster); r2 = Con(r1)

  • create separate temporary file names for rasters that are saved and overwritten multiple times in a loop (i.e. each intermediate raster has its own name)


Different things worked in different places, without any recognizable pattern. As such, I still dont know what was causing Python to crash. My best guess is that arcpy was attempting to overwrite a raster that was in memory with an updated version of itself, though this still would explain why it crashed on raster.save.

Anyone who can shed light on this, it would be greatly appreciated.
0 Kudos
ForrestStevens1
New Contributor
UPDATE I have gotten my script to work, though I dont really know how. Attached is the updated section of code with fixes. There were several places where things broke for unknown reasons. In each place I tried one of 3 things:

  • save the intermediate raster to the hard drive, then load it back into memory using arcpy.sa.Raster

  • break compound raster calculations into several lines, e.g. Con(IsNull(raster)) becomes r1=IsNull(raster); r2 = Con(r1)

  • create separate temporary file names for rasters that are saved and overwritten multiple times in a loop (i.e. each intermediate raster has its own name)



Different things worked in different places, without any recognizable pattern. As such, I still dont know what was causing Python to crash. My best guess is that arcpy was attempting to overwrite a raster that was in memory with an updated version of itself, though this still would explain why it crashed on raster.save.

Anyone who can shed light on this, it would be greatly appreciated.


I'm afraid I can't shed any light on it, but can confirm that I'm experiencing the exact same type of inconsistent behavior.  It depends on the machine I'm running it on, the time of day,  angle of the sun... who knows... But it's infuriating as to when it chooses to crash and when it doesn't.  (ArcGIS 10.0 SP5)
0 Kudos
ChrisSnyder
Regular Contributor III
Not sure what's going on, but where do you have arcpy.env.workspace set to? (this is where the temp scratch rasters get written)

in_memory?
0 Kudos
AustinMilt
New Contributor III
Not sure what's going on, but where do you have arcpy.env.workspace set to? (this is where the temp scratch rasters get written)

in_memory?


I have not set the workspace in this script, but I would not generally set the workspace to in_memory. Should I be setting it to that?
0 Kudos
ForrestStevens1
New Contributor
I have not set the workspace in this script, but I would not generally set the workspace to in_memory. Should I be setting it to that?


I typically don't set my workspace explicitly, but the Con() statement is clearly writing temporary output to the folder in which the raster I'm processing resides as an ESRI grid file.  It disappears if processing succeeds, but is left around if Python crashes.  Most often when this occurs I can workaround the issue by running a SetNull() that does nothing, for example in a raster where there are no values of 65534 this will work:

outCon1 = Raster(raster_path)
outCon2 = SetNull(outCon1==65534, outCon1)
outCon3 = Con(outCon2, 0, outCon2, "VALUE >= 65530 AND VALUE <= 65534")

But this will not:

outCon1 = Raster(raster_path)
outCon2 = Con(outCon1, 0, outCon1, "VALUE >= 65530 AND VALUE <= 65534")
0 Kudos