Hi there,
I am new at Pythin and during the last couple months I have working on writing some Python scripts to automatize some of my drone imagery analysis, such as the code bellow. I have written few scripts and used them on different datasets and they were working just fine. A couple days ago I realize that would be better to save my final raster (yellow) on the same folder that my original RGB raster (blue), so I can use that same path in other scripts for further analysis without having to copy rasters around. Then, I started to get a message ERROR 999998: Unexpected Error, and now does not matter which code I try to run, I get the same message.
I have been looking around for a solution for my issue but I have been unsuccessful on that regard. By reading some material online, it seems that problem is related to memory issues, but I am not quite sure about that. Here are some details about my machine and software.
- desktop computer (Windows 10, version 1511, OS Build 10586.663, Processor Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 3.41 GHz, RAM 32GB, System type 64-bit operating system, x64-based processor) . I have ArcGis 10.5 and I am using PythonWin (PythonWin 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32.) to write my scripts.
- I ran the code bellow on my Surface 3 laptop (Windows 10 Pro, version 10.0.16299 build 16299, system type x64-based PC, processor Intel Core i5-4300 CPU @ 1.90GHz, 2501 Mhz, 2 cores, 4 logical processors; RAM 8GB) this morning and it worked just fine.
-Last night I ran the same code from within ArcGIS and it worked just fine.
-I can not understand how everything was working just fine until a couple days ago, and suddenly the computer does not have enough memory to handle the same task. My RGB file is (around 1.8 GB in size, but I used files larger than that in the past with no problems.
Any help to fix this issue would be greatly appreciated.
Code:
import arcpy, string, os, errno
from arcpy import env
from arcpy.sa import*
from datetime import datetime
arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True
start_time = datetime.now()
imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'
path = os.path.dirname(imagery)
base1 = os.path.basename(imagery)
base = os.path.splitext(base1)[0]
folder = "Results"
Results = os.path.join(path,folder)
if not os.path.exists(Results):
os.makedirs(Results)
print "Results folder successfully created"
exgr_filename = base + "_ExGr.tif"
arcpy.env.workspace = imagery
bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]
print bands
arcpy.env.workspace = Results
Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])
Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])
Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])
exgr = (2 * Ngreen) - Nred - Nblue
exgr.save(exgr_filename)
print "Excess Green successfully calculated"
#Build pyramids and calculate statistics
arcpy.BuildPyramids_management(exgr_filename)
arcpy.BuildPyramids_management(exgr_filename)
print "Pyramids and Statistics successful"
end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))
Interactive Window messages:
E:\UAV2017\PythonField8ExGr
Field8_90ft_08012017
Results folder successfully created
Traceback (most recent call last):
File "C:\Python27\ArcGIS10.5\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 325, in RunScript
exec codeObject in __main__.__dict__
File "C:\Users\Paulo.Flores\Desktop\Python Codes\New versions\ExGr_calc_02072018.py", line 45, in <module>
exgr.save(exgr_filename)
RuntimeError: ERROR 999998: Unexpected Error.
To give you an idea of what could be done:
import arcpy
import os
from datetime import datetime
from arcpy.sa import *
arcpy.CheckOutExtension("spatial")
arcpy.env.overwriteOutput = True
start_time = datetime.now()
# imagery = r'E:\UAV2017\PythonField8ExGr\Field8_90ft_08012017.tif'
imagery = r'C:\GeoNet\UAV2017\Field8_90ft_08012017.tif'
res_folder_name = 'testx02' # 'Results'
fgdb_name = 'ExcessGreen.gdb'
exgr_postfix = '_ExGr'
#Get path information to create Results folder, where NDVI and NDRE will be saved
img_path, img_name_ext = os.path.split(imagery)
img_name, img_ext = os.path.splitext(img_name_ext)
print("img_path : {}".format(img_path))
print("img_name_ext : {}".format(img_name_ext))
print("img_name : {}".format(img_name))
print("img_ext : {}".format(img_ext))
out_folder = os.path.join(img_path, res_folder_name)
if not os.path.exists(out_folder):
os.makedirs(out_folder)
print("out_folder : {}".format(out_folder))
print "Results folder successfully created"
# create fgdb
ws = os.path.join(out_folder, fgdb_name)
if not arcpy.Exists(ws):
arcpy.CreateFileGDB_management(out_folder, fgdb_name, "CURRENT")
print("ws : {}".format(ws))
#Saving name for Excess Green mosaic - using basename from the original imagery
exgr_filename = os.path.join(ws, img_name + exgr_postfix)
print("exgr_filename: {}".format(exgr_filename))
#Set workspace to multiband raster, then list rasters to get band names
arcpy.env.workspace = imagery
bands = [Raster(os.path.join(imagery, b)) for b in arcpy.ListRasters()]
print("bands : {}".format(bands))
#Set workspace to the location were the results will be saved
# arcpy.env.workspace = r'C:\GeoNet\UAV2017\fgdb\results.gdb'
#Excess Green calculation - (2*Ngreen-Nred-Nblue)
Ngreen = (Float(bands[1])) / (Float(bands[0]) + bands[1] + bands[2])
print "Ngreen succesfully calculated"
Nred = (Float(bands[0])) / (Float(bands[0]) + bands[1] + bands[2])
print "Nred succesfully calculated"
Nblue = (Float(bands[2])) / (Float(bands[0]) + bands[1] + bands[2])
print "Nblue succesfully calculated"
exgr = ((2 * Ngreen) - Nred - Nblue)
print "exgr succesfully calculated"
# exgr_filename = r'C:\GeoNet\UAV2017\fgdb\results.gdb\exgr'
exgr.save(exgr_filename)
print "Excess Green successfully calculated and stored..."
#Build pyramids and calculate statistics for each raster in env.worspace folder
arcpy.BuildPyramids_management(exgr_filename)
arcpy.CalculateStatistics_management(exgr_filename)
print "Pyramids and Statistics successful"
end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))
This will print:
img_path : C:\GeoNet\UAV2017
img_name_ext : Field8_90ft_08012017.tif
img_name : Field8_90ft_08012017
img_ext : .tif
out_folder : C:\GeoNet\UAV2017\testx02
Results folder successfully created
ws : C:\GeoNet\UAV2017\testx02\ExcessGreen.gdb
exgr_filename: C:\GeoNet\UAV2017\testx02\ExcessGreen.gdb\Field8_90ft_08012017_ExGr
bands :
Ngreen succesfully calculated
Nred succesfully calculated
Nblue succesfully calculated
exgr succesfully calculated
Excess Green successfully calculated and stored...
Pyramids and Statistics successful
Duration: 0:06:36.999000
Xander,
Thanks for the code. I ran it twice on my desktop and it gave me the same error message. Everything works just fine until it reach the point to save the exgr raster. I ran it on both my laptop computer and my Surface Pro 3 and it ran just fine. I am guessing that there is some kind of issue with my desktop computer. Do you thing that reinstalling ArcGIS 10.5 would help with my issues?
Thanks again.
If you can, upgrade to 10.6 and I would just suggest using PRO at this point
Paulo... the file you attached is too big (over 1 Gb) and would take a long time to download. Given that is so big and you have been trying it so many times, I am wondering if your temporary processing folders are just getting full at some point. It may be worth cleaning up your workspaces especially in your 'User' profile area.
Dan,
I cleaned some space on my disk last Friday. I have quite a bit of empty space on my C hard drive (around 400 GB) and over 1 TB on my E drive. So, I am not sure that space would be an issue on this case.
Thanks
Dan,
Here is a smaller imagery https://filetransfer.ndsu.edu/link/n5DOUyXirQIdAf5nudoene if you still want to play around with it. It is around 350MB in size.
Best,
Paulo
Paulo, I will have a look at it soon. Glad things worked and the desktop computer gets an upgrade as well
Dear Dan and Xander,
Thanks again for your help with my code issues. I have ran Xander's code with using a at least 2 different rasters and it works just fine (Surface Pro 3 tablet). I am still working on my original code since it saves the ExGr raster on the same folder than the RGB raster, and I have another piece of code that runs some analysis on those two rasters. So, it is easier for me if both of them are located on the same folder, I already have the other piece of code. I ran the original code with 6 different rasters using my Surface Pro 3 (Windows 10 and ArcGIS 10.4) and it worked every time, but I can not run neither code version on my desktop computer. I talked with our IT folks and they are getting me ArcGIS 10.6 and ArcGIS Pro. I will try to run the codes again on my desktop with the new software and I will let you guys know how that goes.
Best.
I haven't extensively scoured the thread here, but wanted to make sure you were aware of ArcGIS 10.5.1 (Desktop, Engine, Server) Spatial Analyst Zonal Statistics Tool Patch
Related?
I got ArcGIS 10.6 installed on my computer this morning. I tried to run my original code (above) a couple times and I get the same error message than I was getting with ArcGIS 10.5. One thing that I noticed when I get this error message is that it happens really fast (within few seconds). It seems like it is not performing the calculations on lines 35, 36, and 37. If I add print statements under those algebra functions, they are printed within those few seconds before I get the error message. When I run the code on my laptop computer it takes much longer to print each of those messages (like it is taking the time to actually perform the calculation). Any advise or suggestion?
Thanks,
Paulo.