Why is ArcGIS Pro's Multicore Support Still So Terrible in 2023?

[AlvisDou]

I am writing to express my frustration with the lack of multicore support in ArcGIS Pro. Despite being in 2023, the software's ability to handle data processing on multiple cores and threads is still abysmal.

There are only a handful of tools in ArcGIS Pro that are capable of utilizing multiple cores, and even then, the performance boost is often marginal. For tasks such as DEM filling and flow direction calculations, which are critical in many spatial analysis workflows, the software is still limited to single-threaded processing. This results in excruciatingly slow performance that can make even the simplest analyses take an inordinate amount of time.

As someone who relies on ArcGIS Pro for my daily work, I find it unacceptable that the software is so far behind the curve in terms of multicore support. It is not as if multicore processing is a new concept; it has been around for decades, and most modern software can take advantage of it.

I urge Esri to prioritize improving the multicore support in ArcGIS Pro. This would not only improve the software's performance but also increase productivity for users who depend on it for their work.

[MichaelVolz]

Are you just trying to run one geoprocess and it is taking a long time or are you trying to run simultaneous geoprocesses?

How much RAM does the computer have where you are running your geoprocess(es)?

[AlvisDou]

I am using a workstation with two Xeon Gold CPUs (32 cores) and 128G of RAM. I tried to fill in the pits of the DEM of the whole Tibetan plateau, and of course I divided the whole area into 12 separate parts according to river basins. However, during the processing, arcgis pro calls only one core for CPU and no more than 25G for RAM, which led me to spend a long time and the progress of the processing process was 0.

 

 

[MichaelVolz]

How about running separate geoprocesses simultaneously on separate river basins with different Pro projects open?  I would add another basin to start and if that does not crash Pro then add another and monitor how much total RAM is being used as you add more simultaneous geoprocesses.

Would that achieve a workaround for true multiprocessing.

[AlvisDou]

I understand what you mean. But the problem I'm facing now is that even if only one watershed is filled, Pro's computing efficiency is still very low, like what I've shown in the screenshot, the progress is completely 0. I once spent 5 days to compute an area but the progress is always 0. I had to force kill Pro's process because I don't know if it's really still working properly. I hope Pro can make full use of the machine's performance, even if it's just a processing can also use all the computing.

[MichaelVolz]

So this thread talks about getting computers up to spec to run latest version of Pro, where there is a significant amount of pushback from IT management.  Not sure if this is a problem for you or not as your computer has much higher specs than the computers at my org, so the geoprocesses you are running on your computer would be even more problematic on my org's computers.

How to convince I.T. to buy computers that can han... - Esri Community

Would you know if the geoprocess of creating 1 or 2 meter contours in your individual river basins would be as computationally intensive as the processes you are currently performing of DEM filling and flow direction calculations?  I would be curious to know how long a contour generation process would take compared to your current processes.

[DrewFlater]

There are around 100 of the most commonly used geoprocessing tools in ArcGIS Pro which can leverage parallel processing. Like you alluded to, parallel processing is not a silver bullet or one size fits all solution, so some tools use parallel processing in different ways. For some tools it is enabled by default to use all your machine cores in parallel, some tools use half, and some tools that support parallel processing do not do it by default. You can learn more about parallel processing in each tool reference topic. You pictured and described the Fill tool.

https://pro.arcgis.com/en/pro-app/latest/tool-reference/spatial-analyst/fill.htm

Here is the relevant excerpt. 

This tool supports parallel processing. If your computer has multiple processors or processors with multiple cores, better performance may be achieved, particularly on larger datasets. The Parallel processing with Spatial Analyst help topic includes details about this capability and how to configure it.

When using parallel processing, temporary data will be written to manage the data chunks being processed. ...

By default, this tool will use 50 percent of the available cores. If the input data is smaller than 5,000 by 5,000 cells in size, fewer cores may be used. You can control the number of cores the tool uses with the Parallel processing factor environment.

You can try to set the parallel processing factor to 100 and see if the performance for Fill is improved with this setting. 

[AlvisDou]

thanks for you help. I will refer these information for my next processing.

[AlvisDou]

hello, I set the parallel processing factor to 100 and processed the flow direction, but the performance did not improve very highly. As you can see from the screenshot, only ~5% of CPU was used and only  ~10% RAM was used. Could it improve more?