Select to view content in your preferred language

ERROR 999998: Unexpected Error raster algebra

8870
42
02-08-2018 11:30 AM
PauloFlores
Emerging Contributor

Hi there,

 I am new at Pythin and during the last couple months I have working on writing some Python scripts to automatize some of my drone imagery analysis, such as the code bellow. I have written few scripts and used them on different datasets and they were working just fine. A couple days ago I realize that would be better to save my final raster (yellow) on the same folder that my original RGB raster (blue), so I can use that same path in other scripts for further analysis without having to copy rasters around. Then, I started to get a message ERROR 999998: Unexpected Error, and now does not matter which code I try to run, I get the same message. 

I have been looking around for a solution for my issue but I have been unsuccessful on that regard. By reading some material online, it seems that problem is related to memory issues, but I am not quite sure about that. Here are some details about my machine and software.

- desktop computer (Windows 10, version 1511, OS Build 10586.663, Processor Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 3.41 GHz, RAM 32GB, System type 64-bit operating system, x64-based processor) . I have ArcGis 10.5  and I am using PythonWin (PythonWin 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32.) to write my scripts.

- I ran the code bellow on my Surface 3 laptop (Windows 10 Pro, version 10.0.16299 build 16299, system type x64-based PC, processor Intel Core i5-4300 CPU @ 1.90GHz, 2501 Mhz, 2 cores, 4 logical processors; RAM 8GB) this morning and it worked just fine.

-Last night I ran the same code from within ArcGIS and it worked just fine.

-I can not understand how everything was working just fine until a couple days ago, and suddenly the computer does not have enough memory to handle the same task. My RGB file is (around 1.8 GB in size, but I used files larger than that in the past with no problems.

 

Any help to fix this issue would be greatly appreciated.

Code:

import arcpy, string, os, errno

from arcpy import env

from arcpy.sa import*

from datetime import datetime

arcpy.CheckOutExtension("spatial")

arcpy.env.overwriteOutput = True

 

start_time = datetime.now()

 

imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'

 

path = os.path.dirname(imagery)

base1 = os.path.basename(imagery)

base = os.path.splitext(base1)[0]

folder = "Results"

Results = os.path.join(path,folder)

if not os.path.exists(Results):

    os.makedirs(Results)

print "Results folder successfully created"

 

exgr_filename = base + "_ExGr.tif"

arcpy.env.workspace = imagery

bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]

print bands

 

arcpy.env.workspace = Results

Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])

Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])

Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])

exgr = (2 * Ngreen) - Nred - Nblue

exgr.save(exgr_filename)

print "Excess Green successfully calculated"

 

#Build pyramids and calculate statistics

arcpy.BuildPyramids_management(exgr_filename)

arcpy.BuildPyramids_management(exgr_filename)

print "Pyramids and Statistics successful"

 

end_time = datetime.now()

print('Duration: {}'.format(end_time - start_time))

 

Interactive Window messages:

E:\UAV2017\PythonField8ExGr

Field8_90ft_08012017

Results folder successfully created

Traceback (most recent call last):

  File "C:\Python27\ArcGIS10.5\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 325, in RunScript

    exec codeObject in __main__.__dict__

  File "C:\Users\Paulo.Flores\Desktop\Python Codes\New versions\ExGr_calc_02072018.py", line 45, in <module>

    exgr.save(exgr_filename)

RuntimeError: ERROR 999998: Unexpected Error.

0 Kudos
42 Replies
XanderBakker
Esri Esteemed Contributor

To give you an idea of what could be done:

import arcpy
import os
from datetime import datetime
from arcpy.sa import *

arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True

start_time = datetime.now()

# imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'
imagery = r'C:\GeoNet\UAV2017\Field8_90ft_08012017.tif'
res_folder_name = 'testx02' # 'Results'
fgdb_name = 'ExcessGreen.gdb'
exgr_postfix = '_ExGr'

#Get path information to create Results folder, where NDVI and NDRE will be saved
img_path, img_name_ext = os.path.split(imagery)
img_name, img_ext = os.path.splitext(img_name_ext)

print("img_path     : {}".format(img_path))
print("img_name_ext : {}".format(img_name_ext))
print("img_name     : {}".format(img_name))
print("img_ext      : {}".format(img_ext))

out_folder = os.path.join(img_path, res_folder_name)
if not os.path.exists(out_folder):
    os.makedirs(out_folder)

print("out_folder   : {}".format(out_folder))
print "Results folder successfully created"

# create fgdb
ws = os.path.join(out_folder, fgdb_name)
if not arcpy.Exists(ws):
    arcpy.CreateFileGDB_management(out_folder, fgdb_name, "CURRENT")
print("ws           : {}".format(ws))

#Saving name for Excess Green mosaic - using basename from the original imagery
exgr_filename = os.path.join(ws, img_name + exgr_postfix)
print("exgr_filename: {}".format(exgr_filename))

#Set workspace to multiband raster, then list rasters to get band names
arcpy.env.workspace = imagery
bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]
print("bands        : {}".format(bands))

#Set workspace to the location were the results will be saved
# arcpy.env.workspace = r'C:\GeoNet\UAV2017\fgdb\results.gdb'

#Excess Green calculation - (2*Ngreen-Nred-Nblue)
Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])
print "Ngreen succesfully calculated"
Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])
print "Nred succesfully calculated"
Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])
print "Nblue succesfully calculated"
exgr = ((2 * Ngreen) - Nred - Nblue)
print "exgr succesfully calculated"

# exgr_filename = r'C:\GeoNet\UAV2017\fgdb\results.gdb\exgr'
exgr.save(exgr_filename)
print "Excess Green successfully calculated and stored..."

#Build pyramids and calculate statistics for each raster in env.worspace folder
arcpy.BuildPyramids_management(exgr_filename)
arcpy.CalculateStatistics_management(exgr_filename)
print "Pyramids and Statistics successful"

end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))

This will print:

img_path     : C:\GeoNet\UAV2017
img_name_ext : Field8_90ft_08012017.tif
img_name     : Field8_90ft_08012017
img_ext      : .tif
out_folder   : C:\GeoNet\UAV2017\testx02
Results folder successfully created
ws           : C:\GeoNet\UAV2017\testx02\ExcessGreen.gdb
exgr_filename: C:\GeoNet\UAV2017\testx02\ExcessGreen.gdb\Field8_90ft_08012017_ExGr
bands        : 
Ngreen succesfully calculated
Nred succesfully calculated
Nblue succesfully calculated
exgr succesfully calculated
Excess Green successfully calculated and stored...
Pyramids and Statistics successful
Duration: 0:06:36.999000
PauloFlores
Emerging Contributor

Xander,

Thanks for the code. I ran it twice on my desktop and it gave me the same error message. Everything works just fine until it reach the point to save the exgr raster. I ran it on both my laptop computer and my Surface Pro 3 and it ran just fine. I am guessing that there is some kind of issue with my desktop computer. Do you thing that reinstalling ArcGIS 10.5 would help with my issues?

Thanks again.

0 Kudos
DanPatterson_Retired
MVP Emeritus

If you can, upgrade to 10.6 and I would just suggest using PRO at this point

DanPatterson_Retired
MVP Emeritus

Paulo... the file you attached is too big (over 1 Gb) and would take a long time to download. Given that is so big and you have been trying it so many times, I am wondering if your temporary processing folders are just getting full at some point.  It may be worth cleaning up your workspaces especially in your 'User' profile area.

0 Kudos
PauloFlores
Emerging Contributor

Dan,

I cleaned some space on my disk last Friday. I have quite a bit of empty space on my C hard drive (around 400 GB) and over 1 TB on my E drive. So, I am not sure that space would be an issue on this case.

Thanks

0 Kudos
PauloFlores
Emerging Contributor

Dan,

Here is a smaller imagery https://filetransfer.ndsu.edu/link/n5DOUyXirQIdAf5nudoene if you still want to play around with it. It is around 350MB in size.

Best,

Paulo

0 Kudos
DanPatterson_Retired
MVP Emeritus

Paulo, I will have a look at it soon.  Glad things worked and the desktop computer gets an upgrade as well

0 Kudos
PauloFlores
Emerging Contributor

Dear Dan and Xander,

Thanks again for your help with my code issues. I have ran Xander's code with using a at least 2 different rasters and it works just fine (Surface Pro 3 tablet). I am still working on my original code since it saves the ExGr raster on the same folder than the RGB raster, and I have another piece of code that runs some analysis on those two rasters. So, it is easier for me if both of them are located on the same folder, I already have the other piece of code. I ran the original code with 6 different rasters using my Surface Pro 3 (Windows 10 and ArcGIS 10.4) and it worked every time, but I can not run neither code version on my desktop computer. I talked with our IT folks and they are getting me ArcGIS 10.6 and ArcGIS Pro. I will try to run the codes again on my desktop with the new software and I will let you guys know how that goes.

Best.

KoryKramer
Esri Community Moderator

I haven't extensively scoured the thread here, but wanted to make sure you were aware of ArcGIS 10.5.1 (Desktop, Engine, Server) Spatial Analyst Zonal Statistics Tool Patch 

Related?

0 Kudos
PauloFlores
Emerging Contributor

I got ArcGIS 10.6 installed on my computer this morning. I tried to run my original code (above) a couple times and I get the same error message than I was getting with ArcGIS 10.5. One thing that I noticed when I get this error message is that it happens really fast (within few seconds). It seems like it is not performing the calculations on lines 35, 36, and 37. If I add print statements under those algebra functions, they are printed within those few seconds before I get the error message. When I run the code on my laptop computer it takes much longer to print each of those messages (like it is taking the time to actually perform the calculation). Any advise or suggestion?

Thanks,

Paulo.

0 Kudos