topic Re: Boost Analysis Speed with VM? in ArcGIS Pro Questions

Boost Analysis Speed with VM?

BB34 — Mon, 13 Apr 2020 21:30:20 GMT

Hello there, hope you are all well.

I'm hoping someone can explain the benefits of using ArcGIS Pro in a Virtual Machine such as Azure. I am a GIS Masters student in my first semester and I am working on a project that is taking a long analysis time. Its not a complicated project, I'm calculating rooftop solar radiation. So I am using the Area Solar Radiation tool and a 0.7m DSM for an entire county. However, I have played with the tool settings and I do not seem to be making any headway on decreasing the analysis time. I feel bad for my poor computer, it hasn't had a moment of rest in days. So I have been researching Virtual Machine Computing, which I have no experience with.

There is a lot of information out there on the subject, but most of it is geared towards ArcGIS Enterprise or 3D rendering. I just wanted to know if a virtual machine could decrease my analysis time, and if so how does one go about doing that?

Thanks.

Re: Boost Analysis Speed with VM?

DanPatterson_Retired — Mon, 13 Apr 2020 21:59:36 GMT

From the area solar radiation tool

... do some test runs with a coarser resolution or subset of your data to ensure the settings are correct before committing a run with the full-resolution data.

What resolution did you find was adequate for your purposes?
Are you trying to derive values for actual roofs? or do you just need values for your latitude?
Country-wide? which country? and did you use a projected coordinate system or did you account for cell size during the projection? Cell Size Projection Method (Environment setting)—Geoprocessing | Documentation

I find it hard to believe that you would have need a 0.7 m raster for a whole country to get the solar potential results that you need.

Perhaps you can clarify your workflow. Simply aggregating or filtering your elevations by a factor of 2 will reduce computations by approximately a factor of 4

Re: Boost Analysis Speed with VM?

BB34 — Mon, 13 Apr 2020 22:23:01 GMT

Thank you for your response. I have run several tests with the tool, using a 10m DEM and decreasing the number of calculation directions, however, when I make the leap to the 0.7m it really increases the time.

Unfortunately I have not been able to find any other DEMs for my county (Rutland County, Vermont), other then the two I mentioned. I am indeed doing a rooftop analysis so its really important to have that 0.7m accuracy for the rooftops so I can get an accurate measurement of slope and aspect which are important later in calculating the potential of each rooftop. However, other then the rooftops I don't need the DEM/DSM to be so accurate, the 10m would be fine. However, I was under the impression that it was impossible to mosaic rasters of different cell sizes together, otherwise I would give that a try. I have everything projected to NAD '83 State Plane.

Re: Boost Analysis Speed with VM?

DanPatterson_Retired — Mon, 13 Apr 2020 23:19:37 GMT

Some further comments, not necessarily in order...

You can model the radiation on a horizontal surface at each location, then extract those values to polygons that represent your actual rooftops and apply slope and aspect corrections using well known equations (eg. Sellers, 1965 plus the subsequent generations of climatologists).
The calculations are ground-based, or have you added building heights to the dem? Shading on a roof vs the ground will be different as you know.
As for the skipping the dem from 10 m to 0.7, just 'filter'/'aggregate' the 0.7 dem to a coarser resolution, and run tests to see if there is any break points.
And finally, if you are interested in existing roof structures, then you can convert a lot of you dem to nodata to cut down on calculation time significantly.

(PS, my undergrad thesis circa 1977, Global Irradiance on Slopes )

Re: Boost Analysis Speed with VM?

George_Thompson — Tue, 14 Apr 2020 11:40:41 GMT

Now I cannot comment on the specific tool but here is some documentation related to using ArcGIS Pro within Azure / AWS.

ArcGIS Pro on Microsoft Azure Cloud—ArcGIS Pro | Documentation

License ArcGIS Pro in a virtualized environment—ArcGIS Pro | Documentation

Running ArcGIS Pro in a virtualized environment—ArcGIS Pro | Documentation

This is where I would start to look for a more powerful machine. Now more powerful does not always mean faster .

Re: Boost Analysis Speed with VM?

BB34 — Tue, 14 Apr 2020 17:11:39 GMT

This is the link to the model I have been following, hopefully this explains my process better.

Estimate Solar Power Potential | Learn ArcGIS

I have not added any building height to the calculation I assumed the tool would take that into account especially it being such a precise DSM raster.
I am not familiar with the filter/aggregate technique to convert the raster to a coarser resolution, this sounds like it might be helpful. While I need precision for the roof details, 0.7m is probably excessive, I would expect something like 2m might be more appropriate and would probably decrease analysis time?
I have considered altering my DSM to only consider the rooftops, however, I am not familiar with the area solar radiation tool enough to know if this would alter my results significantly. My worry is that in a mountainous and hilly area such as Vermont, altering the DSM might change how much sun the tool believes the rooftops are receiving.

Thank you for your help in this.

Re: Boost Analysis Speed with VM?

BB34 — Tue, 14 Apr 2020 17:16:47 GMT

Thank you for the documentation.

Unfortunately the Area Solar Radiation tool does not currently support parallel processing. So as you mentioned I am not sure if a virtualized environment would increase processing speed of not.

Re: Boost Analysis Speed with VM?

George_Thompson — Tue, 14 Apr 2020 17:24:54 GMT

Thanks for the update. You may be correct on not supporting parallel processing and larger machine.

Re: Boost Analysis Speed with VM?

MarcoBoeringa — Tue, 14 Apr 2020 17:30:59 GMT

Definitely don't expect any miracles from a VM. A VM is just another virtual computer, and as such limited by the same constraints that are valid for your own desktop. A virtual processor is still tied to a core somewhere on the globe, and those cores may even be worse in single thread processing than your local PC if you are unlucky. Only when your geoprocessing tool supports parallel processing, could you see benefits of some VM with many virtual cores or GPU based processing. But unless you're willing to pay a significant sum, a truly powerful VM may be out of reach, unless your university offers specialized computing services for students.

Maybe cutting up your raster data set in smaller units, and compute those individually (in parallel), on multiple cores of your current machine, or multiple computers at the same time in your department, is the best bet to get results faster.