topic Re: area solar radiation performance in ArcGIS Spatial Analyst Questions

area solar radiation performance

robertkalasek — Sun, 05 Mar 2023 18:51:31 GMT

i am working on a intel core i9-13900k machine with 24 cores, 128gb ram and one of the fastests nvme ssds currently existing. and i am about to do some experiments with a test region of about 4000 x 4500 cells 10x10m each.

the expected reuslult set to "Whole year", all the other settings are left default exept for the "Diffusse proportion" - changed from 0.3 to 0.32 and "Transmittivity" from 0.5 to 0.48.

that calculation takes quite a long time and the most annoying thing is, that the taskmanager shows just one single core doing at least something - at an overall cpu workload of 7 - 9% and a rate of about 1,5 to 5.2 GHz.

but on top of that astonishingly poor performance one older machine with lower performance specs in any dimension is outperforming the modern machine by about 30% (!).

Re: area solar radiation performance

DanPatterson — Sun, 05 Mar 2023 19:51:39 GMT

Yes, it warns you about that in the first paragraph under Usage

Area Solar Radiation (Spatial Analyst)—ArcGIS Pro | Documentation

But did you see the Environment settings that might improve the performance?

Analysis environments and Spatial Analyst—ArcGIS Pro | Documentation

What do you have yours set to?

Re: area solar radiation performance

robertkalasek — Mon, 06 Mar 2023 07:52:02 GMT

thx, of course i have been reading the whole documentation in advance. and i have been experimenting with even smaller (than 4000 x 4000 cells) rasters. especially with the environment parameters for parallel processing - without any significant effect on the performance. and still the first issue is the overall performance of intel's flagship processor but the second is even more concerning - the less advanced machine is on par - both with about 7% workload. which means with more or less just one core working on the calculation job. that concept seems to be pretty outdated to me ... but i am still searching for workarounds and settings to bypass these limitations

Re: area solar radiation performance

DanPatterson — Mon, 06 Mar 2023 11:25:55 GMT

I move the thread to ArcGIS Spatial Analyst Questions

to see if any one from that team knows any specific settings that might be causing the issue

I assume that you have ruled out any dataset specific issues that would affect calculation speed

Re: area solar radiation performance

RyanDeBruyn — Mon, 06 Mar 2023 18:03:57 GMT

Thank you @robertkalasek for your post.

The current solar radiation analysis tools are not designed as multithreaded or honor the parallel processing environment setting. So yes have their known limitations.

The larger the raster (or greater the input resolution) the longer the processing time and are recommended for smaller landscape scales (less than one degree).

This is something we are actively working on to release new versions of the Solar tools hopefully in the next release of ArcGIS Pro. This will include support for larger processing areas, along with performance and algorithm improvements, multithread and GPU support.

We hope to have these new tools in your hands as soon as possible. In the mean time keep the communication open and let us know any specific questions or issues you have.

Kind regards,
-Ryan DeBruyn

Re: area solar radiation performance

AlbertC — Mon, 12 Jun 2023 12:15:28 GMT

Same problem here. We did some calculations back in 2015 for a relatively large area. We worked by iterating an overlapping fishnet of our AOI and it took us about 3 months to process the whole thing.
Now we are doing it again using a new DSM. Our machines are way better now compared to the ones we used in 2015. We are performing first estimates on how long will it take us and it is not much less than 3 months. It's true we have more resolution now (our DSM has 4 times more resolution). But still, one would expect much more improvement having way better CPUs with more threads, cores, RAM, switching from 32 bits to 64, etc.

And yes, you take a look at the taskmanager and it doesn't seem to be doing much, probably uses 5% of resources.
This would be ridiculous in gaming. There have been a lot of improvements in the ESRI ecosystem but geoprocessing power has never been a top priority.

Re: area solar radiation performance

AlbertC — Mon, 12 Jun 2023 12:19:44 GMT

It's great you are working on it. Please keep us posted!

Re: area solar radiation performance

KevinBell1 — Mon, 25 Sep 2023 21:04:44 GMT

I did 800 square miles of Salt Lake County, Utah back in 2013 on 1 meter LiDAR. Yes, it's a heavy lift. I ran it for every month of the year on both Direct Hours and kwh/m2, so 24 individual runs. I was on 32 bit back then, 100 GB RAM, and 32 cores. That took a month running maxed out!

You CAN run "embarrassingly parallel" but you need a Spatial Analyst license for each core you'll run. We have a site license, so I'm not limited by SA licenses. So "embarrassingly parallel" means you write a script for each run, and kick them off separately. Have the result write to separate file locations and merge afterward. Done.

Re: area solar radiation performance

GBacon — Thu, 30 Nov 2023 19:19:02 GMT

I'm having better luck with the new Pro 3.2 raster solar radiation tool that can use GPU and parallel processing - I just don't understand the tool parameters and outputs as much as the old one. The .crf output format seems like it has some nice possibilities, especially if I can publish it as an image service.

CPU:

GPU humming along nicely too.

Re: area solar radiation performance

robertkalasek — Tue, 19 Dec 2023 16:18:55 GMT

@GBacon ... well actually i am not really impressed by an overall 16% utilization-indicator. would be interesting, what the processing-time benefits are. have you tested without parallel processing ?

Re: area solar radiation performance

GBacon — Tue, 19 Dec 2023 16:25:37 GMT

No, but I wasn't getting anywhere with the old tool after running for days. Going to run this again on a new, much more powerful machine when I get it up and running. The whole county ~400 sq. miles at 5-ft DSM resolution took ~72 hours.

Re: area solar radiation performance

RyanDeBruyn — Tue, 19 Dec 2023 20:17:09 GMT

Hi @robertkalasek , thanks for the post.

It is important to note that the original Solar radiation tools are not multi-core / multithread and will run on a single core. So having more (24) cores will not be an advantage here. At what portion during the analysis you are seeing the performance metrics I am not sure. Not sure what the comparison to the older machine is or what the difference is there.

With the new Solar Radiation tools available in ArcGIS Pro 3.2, these tools will use all available cores and if you have an available GPU will benefit significantly.

Even running CPU only mode one would benefit from the new implementation of the tool compared to the original.

Little additional details. In CPU mode only all cores will be occupied, but when in GPU mode only 2-3 cores will be used to handle the processing that is required. So with a good GPU card and the machine above you should have much improved processing performance and scalability.

We are additionally working on performance improvements for the new tools for the next release 3.3 for CPU and GPU processing.

Re: area solar radiation performance

GBacon — Mon, 18 Mar 2024 12:53:58 GMT

@RyanDeBruyn I ran this analysis again against a GPU with 48 GB of dedicated memory without downgrading resolution and it completed in 50 hours as opposed to 72 hours at coarser resolution. I've published image services of the resulting rasters and am very happy with the results.

One question I have though: why does the output of a mean summary statistic from the multidimensional toolbar produce a .crf if different dates are no longer present? The sum statistic does not produce a .crf.

Re: area solar radiation performance

robertkalasek — Thu, 21 Mar 2024 13:56:37 GMT

@GBacon: i would have expected a more impressive advantage by using a GPU. 72 compared to 50 hours obviously is less than 50% increase in performance. so even without using the gpu this increase should be much higher by using up to 32 cores on modern machines compared to just one core with the previous version.

Re: area solar radiation performance

GBacon — Thu, 21 Mar 2024 14:09:19 GMT

Yeah, well. Who knows. I've also encountered a GPU related error along the way even though the processing finished. I've updated my NVIDIA driver to see if that helps.

Re: area solar radiation performance

RyanDeBruyn — Mon, 25 Mar 2024 15:15:14 GMT

Hi @GBacon glad you are having some success.

"Why does the output of a mean summary statistic from the multidimensional toolbar produce a .crf if different dates are no longer present?"

The result of the summary statistics is an aggregated output and we maintain the consistency of the output format as a multidimensional raster, even though as you mentioned the data has only one result (mean values) from all time slices. This does also help to make clear the properties of the data from the analysis. The result .crf should not be any different or affect any subsequent use.

Note: You can run similar results using the GP tool Aggregate Multidimensional Raster (also found in the MD ribbon) where the result may differ in the number (count) of output slices.

Hope this helps.