topic Re: Area Solar Radiation from Python script - error in Python Questions

Area Solar Radiation from Python script - error

toddsams — Wed, 29 Jan 2014 13:47:47 GMT

Hello,

I am attempting to execute Area Solar Radiation in a stand-alone python script (shown below). The basis for the script was provided in the ESRI tool help. I made a few adjustments to this template by adjusting a few of the local variables and adding try/except etc.

The problem is that when I execute the script, it runs for about a minute then fails with an ERROR 999999 (also shown below in quoted script output).

Does anyone have an idea on why this may be occurring?

# Name: SolarRad_MayJuly.py # Description: Derives incoming solar radiation from a raster surface.  #              Outputs a global radiation raster and optional direct, diffuse and direct duration rasters #              for a specified time period. (May to July). #               # Requirements: Spatial Analyst Extension  # Import system modules import arcpy, time from arcpy import env from arcpy.sa import *  # Set environment settings env.workspace = "D:/Projects/Tasks/02_SpatialInputs/SolarRad/work/MayJulyOutput"  # Check out the ArcGIS Spatial Analyst extension license arcpy.CheckOutExtension("Spatial")  # Set local variables inRaster = "D:/Projects/Tasks/02_SpatialInputs/Geomorph/ElevationSlope/elev" latitude = 37.1567 skySize = 200 timeConfig = TimeMultipleDays(2012, 121, 212) dayInterval = 14 hourInterval = 1 zFactor = 1 calcDirections = 32 zenithDivisions = 8 azimuthDivisions = 8 diffuseProp = 0.3 transmittivity = 0.5 outDirectRad = "" outDiffuseRad = "" outDirectDur = ""  print time.asctime(time.localtime()) print "Running Area Solar Radiation..." try:     # Execute AreaSolarRadiation     outGlobalRad = AreaSolarRadiation(inRaster, latitude, skySize, timeConfig,        dayInterval, hourInterval, "NOINTERVAL", zFactor, "FROM_DEM",        calcDirections, zenithDivisions, azimuthDivisions, "UNIFORM_SKY",        diffuseProp, transmittivity, outDirectRad, outDiffuseRad, outDirectDur) except:     print "Error while processing solar radiation"     print (arcpy.GetMessages())  # Save the output  outGlobalRad.save("D:/Projects/Tasks/02_SpatialInputs/SolarRad/work/MayJulyOutput/globrad") print "Finished" print time.asctime(time.localtime())

Tue Jan 28 09:52:16 2014
Running Area Solar Radiation...
Error while processing solar radiation
Executing: AreaSolarRadiation D:/Projects/Tasks/02_SpatialInputs/Geomorph/ElevationSlope/elev D:/Projects/Tasks/02_SpatialInputs/SolarRad/work/MayJulyOutput\AreaSol_elev1 37.1567 200 "MultiDays 2012 121 212" 14 1 NOINTERVAL 1 FROM_DEM 32 8 8 UNIFORM_SKY 0.3 0.5 # # #
Start Time: Tue Jan 28 09:52:16 2014
ERROR 999999: Error executing function.
Failed to open raster dataset
Failed to execute (AreaSolarRadiation).
Failed at Tue Jan 28 09:53:33 2014 (Elapsed Time: 1 minutes 17 seconds)

Re: Area Solar Radiation from Python script - error

MathewCoyle — Wed, 29 Jan 2014 17:01:53 GMT

It seems like something is wrong with your source data. What format is it in? Have you run this from ArcGIS with the same data source without issues?

Re: Area Solar Radiation from Python script - error

toddsams — Wed, 29 Jan 2014 17:10:31 GMT

The source data are in GRID format. I have run Area Solar Radiation on GRID files successfully in ArcGIS in the past.

This particular data set does not successfully complete within ArcGIS. It just runs for about two days, then ArcCatalog quits with no error. I attributed this to the large size of the data set and the complexity of the Solar Radiation analysis. This is what led me to try in a standalone Python script, which aborts (with an error) much quicker than within ArcGIS.

Re: Area Solar Radiation from Python script - error

MathewCoyle — Wed, 29 Jan 2014 17:33:54 GMT

I would first try moving it out of a GRID to a geodatabase raster or ascii. How big is your dataset? Have you tried breaking it up unto smaller sections? If performance is also something you want to gain you can look at multiprocessing. Esri has a blog entry on performing analysis on a large dataset using multithreading that you may find interesting.
http://blogs.esri.com/esri/arcgis/2011/08/29/multiprocessing/

Re: Area Solar Radiation from Python script - error

toddsams — Wed, 29 Jan 2014 17:42:55 GMT

The data set is a 30m resolution with about 20,000 and 25,000 rows and columns (2GB integer GRID).

I have considered breaking into smaller sections, but would get edge effects if the sections do not overlap. I was just trying to give the full dataset a whirl before having to break up.

I will try other raster formats and look into multithreading. Right now, the process gets allocated to only a single core.

Thanks for your thoughts. Will check back if this does not work out.

Re: Area Solar Radiation from Python script - error

toddsams — Wed, 29 Jan 2014 20:42:34 GMT

I tried using TIF, TXT, and a FDGDB raster. None of these worked.

Are there any other suggestions?

Re: Area Solar Radiation from Python script - error

toddsams — Wed, 29 Jan 2014 21:03:39 GMT

I clipped the raster to a smaller area and it works ok. Looks like I will have to split the data up.

Re: Area Solar Radiation from Python script - error

MathewCoyle — Wed, 29 Jan 2014 21:27:22 GMT

What are the specs of the machine you are running this on? You may be running in to memory issues, do you have the 64-bit background geoprocessing module installed?

Re: Area Solar Radiation from Python script - error

toddsams — Wed, 29 Jan 2014 22:03:36 GMT

I have Windows 7 64-bit on a Core i7 with 12GB of RAM.

I had 64-bit geoprocessing installed with 10.0. I am not able to recall if I installed when I upgraded to 10.1. How would I check?

Since the process fails after only a minute, it seems that it does not like the size of the raster as it reads it into memory.

Re: Area Solar Radiation from Python script - error

MathewCoyle — Thu, 30 Jan 2014 13:26:14 GMT

Easiest way is to open IDLE that installed with your version of ArcGIS. At the top it will say whether the default is 32 or 64 bit. As 10.0 and 10.1 use different versions of Python you almost definitely have the 32 bit version, so would have this tag.

...[MSC v.1500 32 bit (Intel)]...

64 bit would have this.

...[MSC v.1500 64 bit (AMD64)]...

Using 64 bit geoprocessing should solve this issue for you so I would look into getting that.

Re: Area Solar Radiation from Python script - error

KevinBell — Thu, 30 Jan 2014 14:16:41 GMT

If you tile the data you'll have edge effects from objects casting shadows across the tile boundary. To prevent that, figure out the height of features that are near your solve tile, buffer the solve tile as a function of the sun angle. Solve the whole buffer, then clip out the original tile, delete the data in the buffer (that now has edge errors) and mosaic the tiles back together. I ran all of Salt Lake County (800 sq miles) on a 32 core server w/ 100 GB Ram and it took a month at 80% capacity. see www.solarsimplified.org

:cool:

Re: Area Solar Radiation from Python script - error

toddsams — Thu, 30 Jan 2014 15:13:00 GMT

I already have 64-bit processioning installed.

It seems I will have to tile the data. I will have to experiment to determine a good tile size.

Thanks for all of your thoughts.