Area Solar Radiation Tool - Large Raster Datasets & Computing Time

[DanielTruessel]

Hi,

I'd like to exchange experiences related to the "Area Solar Radiation" Tool.

We are using the tool to calculate direct and diffuse incoming solar radiation for a valley in the mountains. The cell size of the raster is 1 m x 1 m and it has around 45'000 columns by 42'000 rows - it's quite big.
We try to calculate the radiation for the period of a whole year with a time interval of 14 days and 0.5 hours. The output will be a single file.

The calculation runs on a Windows Server 2012 machine with 72 GB RAM. ArcGIS 10.1, SP1 and the Desktop Background Geoprocessing product are installed. It is running stable since more or less 4 weeks. Most of the time around 9 GB RAM and 10 % of the CPU are in use.

The whole thing is more an experiment to check the possibilities and the power of this tool. We already successfully used it for smaller areas with a lower resolution.

Does someone of you have experience with such large raster datasets used in the Area Solar Radiation Tool? Experience with computing time? Or in general does someone have interesting or helpful information/experiences to share?


Regards

[JimCousins]

I have not used the Area Solar Radiation tool. However, we do perform modelling with large datasets that often take days to run. We reduce the time to results by splitting the analysis into multiple "chunks", having multiple machines (or even multiple cores on the same machine) process the chunks at the same time, and then merging the results. Parallel processing.
Just curious, how many processing cores does your machine have? If it has 10 cores, and 1 core is working this process, then it is maxed out at the 10% shown.
Regards,
Jim

[DanielTruessel]

Dear Jim,

Thanks for your response!

It is an Intel Xeon Processor X5690 with 6 cores. The total CPU usage sometimes quickly rises up to 16 % - 17 %. That is reinforcing/confirming your assumption.

I am not sure if there is even a way to split this large dataset into several parts. The tool is working with hemispherical viewsheds similar to upward-looking hemispherical fisheye photographs (http://resources.arcgis.com/en/help/main/10.1/index.html#//009z000000t9000000), so even - or especially - distant cells matters.


Regards

[KevinBell]

Yes, I've successfully completed an area the same size.  Not for the faint of heart.  Take a look at this map service:
http://maps.slcgov.com/slcisolar/rest/services/solar/SLCO_SolarRadiation/MapServer?f=jsapi

I used a 32 core machine w/ 100 GB Ram, and I ran it parallel at 80% for about a month.

I wouldn't think that'd it be possible to throw your entire area at that tool and have success.

I developed a "variably buffered moving window" (my invention) to tile out the processes without introducing error.  You see, if you cut a chunk of your LiDAR out and solve for that, you'll introduce errors along the borders as there would be shadows cast from outside your solve area that you aren't accounting for.  These errors would be more pronounced in areas with more z variability like downtown where the buildings are big.

My method is a whole bunch of python.  Shoot me an email.  kevin.bell@slcgov.com

http://www.slcgovsolar.com/  this website will be overhauled very soon to be this one that's in beta: http://solarsimplified.org/index.php

May the force be with you.  You'll need it!;)

[DanielTruessel]

Dear Kevin,

Thanks to you too for your interesting post!

It would be really interesting to glance at your solution. Especially in our case with a valley in the alps I'm wondering how it would account for "distant" mountains which are mainly responsible for the shadow.

While running the tool, I encountered another kind of problem. I read about similar effects in connection with Arc Hydro and tools of the Geostatistical Analyst Extension.
In our case, Windows is installed on a relatively space limited solid-state drive. On the other hand the data we are processing are stored on another large hard drive. It seems that temporary/intermediate files are getting written to the Windows user profile (C:\Users\User\AppData\Local\Temp\...). This is quite annoying as the SSD is getting packed with gigabytes of data. It doesn't seem to have to do anything with the definition of the default or the scratch geodatabase in the settings.
Do you (or someone else) know a workaround for this issue? Do I have to play around with the Windows %TEMP% variable or is there an easier way within ArcGIS (what I definitely would prefer).


Otherwise, I'll try to use the force! 😉


Cheers!

[MarkBoucher]

Working with rasters over the network is slow. If you can work on them on your local machine, the processing time is greatly improved. This goes for goeprocessing in spatial analyst; where I've experienced it.

[DanielTruessel]

The SSD and the large hard drive are both located in the same machine.

So I guess what happens is that the files are getting written to the SSD first (C:\Users\...\Temp\...) and then getting transferred back to the other hard drive where also the input data are stored.

[TesfayWoldemariam]

Hi All,

I have been running the Area Solar Radiation Analysis tool for over two weeks but it was interrupted. My question is,  can I re-run/restart the analysis (example from the "results" window), only for the remaining subset of the DEM because, about 50% of it is done. and I can open the output with out problem assuming the iteration was done completely on this part of the dem?

I started the analysis on July 30, 2013 over gmted2010 DEM of size ca.765 MB, format 16 bit signed integer. The settings are  for within a day analysis at 1 ^1/2 interval from 6.30AM to 12.30PM, with geographic coordinate, but z-factor considered. slow but, it has been running and made progress up to 68%, until August 05, 2013. Unfortunately, my ArcGIS10.1 lost license because, our network was down and the silence was on the network server which interrupted the analysis. Due to some networking configuration changes happened within that down time, I could not re-activate the license without rebooting my machine.

Now, checking the "results" window, I see the process status as Not Run", yet, when I opened output products like Solar radiation, direct radiation, diffuse radiation etc; it loaded and it appears that successfully calculated for over more than 50% of the area of the DEM.
My question is, is there a way to rerun the tool/the analysis (example going to the "results" window), but only over the remaining subset of the DEM??

Thanks you in advance,

Tesfay