ERROR 999998: Unexpected Error raster algebra

4818
42
02-08-2018 11:30 AM
PauloFlores
New Contributor II

Hi there,

 I am new at Pythin and during the last couple months I have working on writing some Python scripts to automatize some of my drone imagery analysis, such as the code bellow. I have written few scripts and used them on different datasets and they were working just fine. A couple days ago I realize that would be better to save my final raster (yellow) on the same folder that my original RGB raster (blue), so I can use that same path in other scripts for further analysis without having to copy rasters around. Then, I started to get a message ERROR 999998: Unexpected Error, and now does not matter which code I try to run, I get the same message. 

I have been looking around for a solution for my issue but I have been unsuccessful on that regard. By reading some material online, it seems that problem is related to memory issues, but I am not quite sure about that. Here are some details about my machine and software.

- desktop computer (Windows 10, version 1511, OS Build 10586.663, Processor Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 3.41 GHz, RAM 32GB, System type 64-bit operating system, x64-based processor) . I have ArcGis 10.5  and I am using PythonWin (PythonWin 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32.) to write my scripts.

- I ran the code bellow on my Surface 3 laptop (Windows 10 Pro, version 10.0.16299 build 16299, system type x64-based PC, processor Intel Core i5-4300 CPU @ 1.90GHz, 2501 Mhz, 2 cores, 4 logical processors; RAM 8GB) this morning and it worked just fine.

-Last night I ran the same code from within ArcGIS and it worked just fine.

-I can not understand how everything was working just fine until a couple days ago, and suddenly the computer does not have enough memory to handle the same task. My RGB file is (around 1.8 GB in size, but I used files larger than that in the past with no problems.

 

Any help to fix this issue would be greatly appreciated.

Code:

import arcpy, string, os, errno

from arcpy import env

from arcpy.sa import*

from datetime import datetime

arcpy.CheckOutExtension("spatial")

arcpy.env.overwriteOutput = True

 

start_time = datetime.now()

 

imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'

 

path = os.path.dirname(imagery)

base1 = os.path.basename(imagery)

base = os.path.splitext(base1)[0]

folder = "Results"

Results = os.path.join(path,folder)

if not os.path.exists(Results):

    os.makedirs(Results)

print "Results folder successfully created"

 

exgr_filename = base + "_ExGr.tif"

arcpy.env.workspace = imagery

bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]

print bands

 

arcpy.env.workspace = Results

Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])

Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])

Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])

exgr = (2 * Ngreen) - Nred - Nblue

exgr.save(exgr_filename)

print "Excess Green successfully calculated"

 

#Build pyramids and calculate statistics

arcpy.BuildPyramids_management(exgr_filename)

arcpy.BuildPyramids_management(exgr_filename)

print "Pyramids and Statistics successful"

 

end_time = datetime.now()

print('Duration: {}'.format(end_time - start_time))

 

Interactive Window messages:

E:\UAV2017\PythonField8ExGr

Field8_90ft_08012017

Results folder successfully created

Traceback (most recent call last):

  File "C:\Python27\ArcGIS10.5\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 325, in RunScript

    exec codeObject in __main__.__dict__

  File "C:\Users\Paulo.Flores\Desktop\Python Codes\New versions\ExGr_calc_02072018.py", line 45, in <module>

    exgr.save(exgr_filename)

RuntimeError: ERROR 999998: Unexpected Error.

0 Kudos
42 Replies
XanderBakker
Esri Esteemed Contributor

When you post your code, you should follow these instructions (Posting code with Syntax Highlighting on GeoNet ) to make it more readable, especially when there is an error message that refers to a specific line:

import arcpy, string, os, errno
from arcpy import env
from arcpy.sa import*
from datetime import datetime
arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True

start_time = datetime.now()

imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'

path = os.path.dirname(imagery)
base1 = os.path.basename(imagery)
base = os.path.splitext(base1)[0]
folder = "Results"
Results = os.path.join(path,folder)
if not os.path.exists(Results):
    os.makedirs(Results)
print "Results folder successfully created"

exgr_filename = base + "_ExGr.tif"
arcpy.env.workspace = imagery
bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]
print bands

arcpy.env.workspace = Results
Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])
Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])
Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])
exgr = (2 * Ngreen) - Nred - Nblue
exgr.save(exgr_filename)
print "Excess Green successfully calculated"

#Build pyramids and calculate statistics
arcpy.BuildPyramids_management(exgr_filename)
arcpy.BuildPyramids_management(exgr_filename)
print "Pyramids and Statistics successful"

end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

As you can see the code you shared only has 40 lines, but the error message refers to line 45 which in your code is line 31. So there is about 15 lines that are missing. Can you share the entire code?

What I normally do is not depend too much on the setting the workspace to define the output location. I would rather define the exgr_filename as:

exgr_filename = os.path.join(Results, base + "_ExGr.tif")

So the error occurs when you try to save the result. I'm sure you ruled out all the obvious things like permissions and enough storage space available in the output workspace. Have you tried to write the result as a raster in a file geodatabase? Also avoid creating an output raster name that is very long.

I notice you build the pyramids twice at the end of the script. Although this is after the error occurs, is there a specific reason why you do this?

If possible can you share the imagery or a part of it to see if I an reproduce the error?

You mention that it might be a memory issue. By using ArcMap you will only have access to a very limited part of your 32 RAM. Have you tried using ArcGIS Pro? One would expect the code to run, if the code does work inside the Python Window of ArcMap and no error occurs. Is 64 bit processing enabled? Background Geoprocessing (64-bit)—Help | ArcGIS Desktop 

0 Kudos
PauloFlores
New Contributor II

Xander,

Thanks for your reply. Below you can find the entire code that I am working on, including your suggestion regarding the where to save the "exgr_filename" variable. Regarding the building the pyramids twice, that was my mistake. The second one is supposed to be "CalculateStatistics_management" (added to the code below). As you can see at the bottom, it still giving me the same error.

I tried to save the file with shorter name and it did not work either. I definitely have enough available storage space in the output workspace. I have not tried to save the result as a raster in a file geodatabase. I usually do not use geodatabases. The thing that I can not understand is how everything was working just fine before and suddenly none of my similar codes work. There were no changes made to my computer regarding software or hardware on that meantime.

I have not used ArcGIS Pro. I was actually reading about it last night. I asked to our IT folks on campus to see if we have access to ArcGIS Desktop 10.6. Do you that could help? I will send them an email regarding access to ArcGIS Pro. I believe that 64 bit processing is not enable. I was looking around about that, and it seems that it needs to be downloaded and installed on my computer. I have done all the ArcGIS software installation on my desktop and I did not installed anything else other than the executable file that we got from our IT folks on campus.

Here is a link for the imagery that I am using on this particular case Link | NDSU Secure File Transfer . The link will expire by Saturday (02/17/2018).

Thanks again for your help.

import arcpy, string, os, errno

from arcpy import env
from arcpy.sa import*
from datetime import datetime
arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True

start_time = datetime.now()

imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'
#Get path information to create Results folder, where NDVI and NDRE will be saved
path = os.path.dirname(imagery)
print path
base1 = os.path.basename(imagery)
base = os.path.splitext(base1)[0]
print base
folder = "Results"
Results = os.path.join(path,folder)
if not os.path.exists(Results):
    os.makedirs(Results)
print "Results folder successfully created"

#Saving name for Excess Green mosaic - using basename from the original imagery
exgr_filename = os.path.join(Results, base + "_ExGr.tif")

#Set workspace to multiband raster, then list rasters to get band names
arcpy.env.workspace = imagery
bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]
print bands

#Set workspace to the location were the results will be saved
arcpy.env.workspace = Results
#Excess Green calculation - (2*Ngreen-Nred-Nblue)
Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])
Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])
Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])
exgr = ((2 * Ngreen) - Nred - Nblue)
exgr.save(exgr_filename)
print "Excess Green successfully calculated"

#Build pyramids and calculate statistics for each raster in env.worspace folder
arcpy.BuildPyramids_management(exgr_filename)
arcpy.CalculateStatistics_management(exgr_filename)
print "Pyramids and Statistics successful"

end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))

>>> E:\UAV2017\PythonField8ExGr
Field8_90ft_08012017
Results folder successfully created
[E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_1, E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_2, E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_3, E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_4]
Traceback (most recent call last):
  File "C:\Python27\ArcGIS10.5\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 325, in RunScript
    exec codeObject in __main__.__dict__
  File "C:\Users\Paulo.Flores\Desktop\Python Codes\New versions\ExGr_calc_02122018help.py", line 40, in <module>
    exgr.save(exgr_filename)
RuntimeError: ERROR 999998: Unexpected Error.
>>> ‍‍‍‍‍‍‍‍‍‍‍
0 Kudos
XanderBakker
Esri Esteemed Contributor

Hi Paulo Flores ,

I don't think you posted the entire code. There are no import statements at the start of the script.

I'm not sure if upgrading to 10.6 will make a big difference. Most innovation for Desktop happens at Pro and not in ArcMap. It definitely is strange that some functionality suddenly stopped working when there is not change in hardware, software and data. Could be good to contact Esri Support and see what might be the problem.

I am downloading the imagery and try to run your code on my system with 10.6 and will report back with the results.

0 Kudos
PauloFlores
New Contributor II

I apologize for that.  I edited the previous code and inserted the missing portion of it.

Thanks again.

DanPatterson_Retired
MVP Esteemed Contributor

Can you throw in some print messages that print the results of the various steps, especially, I would be curious to see the grid/raster name going in or out.  

Use *.tiff's as the preferred raster format as well now.  In the original post, line 45 is the terminus, so either the process produced nothing prior to it and/or the output name is not correct... that is the problem with generic error messages and nothing is going to provide more information than print statements before and after steps in the problem area... not even try-except blocks

0 Kudos
XanderBakker
Esri Esteemed Contributor

Hi Dan Patterson  and Paulo Flores ,

I receive the same error when I run the code in 10.6 and create the tif with the same name you used.

Here some results from the print statements:

C:\GeoNet\UAV2017 (path)
Field8_90ft_08012017 (base)
Results folder successfully created


C:\GeoNet\UAV2017\Results\Field8_90ft_08012017_ExGr.tif (exgr_filename, the output tif name)

bands:

After the calculation I included some statements on the output raster (var "exgr") to see if it is valid. And it seems to be valid:

490992,48986 5262286,49651 491152,99946 5262438,19331 NaN NaN NaN NaN (extent)
-0.774999976158 (minimum value)
2.0 (maximum value).

Will do some more testing... First: write to fgdb raster...

0 Kudos
PauloFlores
New Contributor II
import arcpy, string, os, errno

from arcpy import env
from arcpy.sa import*
from datetime import datetime
arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True

start_time = datetime.now()

imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'
#Get path information to create Results folder, where NDVI and NDRE will be saved
path = os.path.dirname(imagery)
print path
base1 = os.path.basename(imagery)
base = os.path.splitext(base1)[0]
print base
folder = "Results"
Results = os.path.join(path,folder)
if not os.path.exists(Results):
    os.makedirs(Results)
print "Results folder successfully created"

#Saving name for Excess Green mosaic - using basename from the original imagery
exgr_filename = os.path.join(Results, base + "_ExGr.tiff")
ngreen_filename = os.path.join(Results, base + "_Ngreen.tiff")
nred_filename = os.path.join(Results, base + "_Nred.tiff")
nblue_filename = os.path.join(Results, base + "_Nblue.tiff")

#Set workspace to multiband raster, then list rasters to get band names
arcpy.env.workspace = imagery
bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]
print bands

#Set workspace to the location were the results will be saved
arcpy.env.workspace = Results
#Excess Green calculation - (2*Ngreen-Nred-Nblue)
Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])
Ngreen.save(ngreen_filename)
print "Ngreen succesfully calculated"
Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])
Nred.save(nred_filename)
print "Nred succesfully calculated"
Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])
Nblue.save(nblue_filename)
print "Nblue succesfully calculated"
exgr = ((2 * Ngreen) - Nred - Nblue)
exgr.save(exgr_filename)
print "Excess Green successfully calculated"

#Build pyramids and calculate statistics for each raster in env.worspace folder
arcpy.BuildPyramids_management(exgr_filename)
arcpy.CalculateStatistics_management(exgr_filename)
print "Pyramids and Statistics successful"

end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))

Dan,

I made the changes that you suggested and ran the code. When I just add the print statements under Ngreen, Nred, and Nblue, the code runs just like before, prints the "print" messages and gives the same errors as before.

When I ran the code above with your suggestions of saving the files as TIFF and tried to save those 3 rasters (Ngreen, Nred, and Nblue) to my disk, I got the following message:

E:\UAV2017\PythonField8ExGr
Field8_90ft_08012017
Results folder successfully created
[E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_1, E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_2, E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_3, E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif\Band_4]
Traceback (most recent call last):
  File "C:\Python27\ArcGIS10.5\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 325, in RunScript
    exec codeObject in __main__.__dict__
  File "C:\Users\Paulo.Flores\Desktop\Python Codes\New versions\ExGr_calc_02122018help.py", line 40, in <module>
    Ngreen.save(ngreen_filename)
RuntimeError: ERROR 010093: Output raster format UNKNOWN is unsupported.
>>> 

I tried to save just the exgr.save(exgr_filename) and get the "print" messages for the 3 first rasters and the same error message for the exgr one.

0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

They aren't raster objects yet, I suspect

http://desktop.arcgis.com/en/arcmap/latest/extensions/spatial-analyst/map-algebra/working-with-raste...

you have to 'Raster' the result then 'yourraster'.save(path_and_filename.tiff) if I remember 

Zip a raster (or a portion) so I can try something with it in numpy... the math is pretty simple and the process is less cumbersome.

0 Kudos
PauloFlores
New Contributor II

Dan,

You can download the imagery from here Link | NDSU Secure File Transfer . I just ran the code that i sent (only difference is that saves the raster as TIF instead of TIFF) on my Surface Pro tablet (Windows 10, ArcGIS 10.4) and it worked just fine. Any thoughts about that?

Thanks,

0 Kudos