ArcGIS and High Performance Computing (HPC)

GeriMiller · ‎09-12-2017

*Updated 2023

We are continuing to see questions about ArcGIS and high-performance computing (HPC), such as:

I am part of a grant involving HPC – how do I run ArcGIS in HPC?
What specific ArcGIS tools work in HPC environment?
How do I license ArcGIS in a HPC environment?
Where do I go for further questions?

Conceptual Summary

Some of the inquiries we’ve received do not distinguish between multicore computers and HPC. This could stem from general fuzziness about what some people mean by HPC, and how the term has evolved in recent years. Most modern desktop computers, including laptops, are “multicore” – meaning they have more than one CPU (central processing unit).

If there is an environment where multiple computers are connected (and each of these may have multiple cores), then you have a compute cluster or distributed compute environment, which could be run in various infrastructures, commercial cloud being the likely place for such an environment. Traditionally HPC, or a distributed compute environment, is described as multiple computers (hundreds) connected and traditionally application software was specifically written for this compute environment.

A lot of the tools in ArcGIS Pro can take advantage of multicore hardware and parallel processing. Some tools do this automatically, some honor a geoprocessing environment, that allows one to specify how many processors one wants to use. Note that not all ArcGIS Pro tools run parallel processing and these tools do not run across multiple machines, therefore a user would need a large single machine to do a big job.

For a full list of the spatial analysis tools that can utilize parallel processing, please refer to this documentation: ArcGIS Pro Parallel Processing

ArcGIS Pro is desktop software which runs on a Windows platform, but what if you are running a Linux HPC environment?

Deployment Options

The vast majority of HPC environments are Linux based, for which a build for ArcGIS Pro does not exist.

For users who have a Linux HPC, there are 2 options available.

Option 1: Deploy ArcGIS Enterprise

ArcGIS Enterprise and the GeoAnalytics Server and Image Server (Raster Analytics) server roles are designed to run in distributed compute environments and can also run across multiples cores of a single machine. These roles are designed to take advantage of distributed storage (not all on one node/disk) and distributed computation (multiple computers).

With a deployment of ArcGIS Enterprise, users can then deploy any of the additional server roles that suit their analytical needs.

These roles are made available via ArcGIS Enterprise but can be accessed from a desktop application, such as ArcGIS Pro, or from a web browser (such as the Map Viewer). Running analysis tools utilizing these server roles can be much faster since the analysis tools have been optimized for the distributed computation across multiple nodes. Users utilizing these tools do not need to do anything additional (coding or writing an application) to take advantage of their distributed compute environment.

For additional guidance, below is documentation related to various configurations for raster analysis and deep learning:

Option 2: Deploy ArcGIS Server

If a user is looking for core geoprocessing tools that are part of ArcGIS Pro (the ArcPy python library), these are found within ArcGIS Server.

By installing and licensing ArcGIS Server, users can then programmatically run the ArcPy geoprocessing tools via scripting environments.

Conclusion

With a Windows HPC environment, users have the option of installing ArcGIS Pro, ArcGIS Enterprise, or ArcGIS Server. In a Linux based cluster, ArcGIS Enterprise or ArcGIS Server provide the best options. Secondly, the GeoAnalytics Server and Image Server (Raster Analytics) server roles utilize distributed computing and can run across multiple machines, or multiple cores of a single machine. A base ArcGIS Enterprise deployment would need to be setup, and these additional server roles enabled.

Further questions

As an academic user, please reach out to highered@esri.com with any Education-related questions.

DawnWright · ‎03-20-2018

Great post and some additional context and helpful information may be in Did You Know? Latest Contributions of Esri R&D and ArcGIS to Science, including how to run free ArcGIS processes on XSEDE (eXtreme Science and Engineering Discovery Environment), the NSF-funded collection of research supercomputer centers in the US.