ArcGIS Pro Please use my RAM and CPU.

17548
18
Jump to solution
02-14-2017 12:24 PM
JohnBrockwell
Occasional Contributor III

I have a new workstation that has 128 GB of RAM and 12 core Intel Xeon CPU. I am trying to run geoprocessing task using ArcGIS Pro 1.4. ArcGIS Pro is not consuming any RAM (7GB) and the CPU usage is non existent.

OS is Windows 7 SP1

What can I do to get ArcGIS Pro 1.4 to utilize my hardware?

1 Solution

Accepted Solutions
DanPatterson_Retired
MVP Emeritus

There is limited options that that tool supports other than current and scratch workspace (check the supported environments way at the bottom of the tool's help).  I thought it might be that it was running slow or you wanted to divide the tasks like in the parallel processing factor

View solution in original post

18 Replies
DanPatterson_Retired
MVP Emeritus

depends on what you are doing in-memory workspaces are useful  and you might want to check the specs anyway, especially your type of windows 7

0 Kudos
JohnBrockwell
Occasional Contributor III

Hi Dan,

I belive my situation is more unique. I'm not trying allocate an allotment of memory. I have 128 gb's of memory. How can I get Pro 1.4 to use 90gb's+ on a geprocessing task. I'm currently running 'Create Space Time Cube'.

Thank you for your feedback,

JB

0 Kudos
DanPatterson_Retired
MVP Emeritus

There is limited options that that tool supports other than current and scratch workspace (check the supported environments way at the bottom of the tool's help).  I thought it might be that it was running slow or you wanted to divide the tasks like in the parallel processing factor

JohnBrockwell
Occasional Contributor III

Finding the Parallel Processing Setting was very useful. I appreciate your feedback.

0 Kudos
MarcoBoeringa
MVP Regular Contributor

As Dan_Patterson‌ already pointed out, this tool does not (yet) support the parallel processing factor environment setting, so it can't take advantage of multi-core CPUs. The fact that your processor has next to none activity, is probably caused by Windows only showing the combined activity of all cores. Since you have 12 cores on a single CPU, the CPU's usage is likely less than 10%, but that single core used for the actual task running at 100%.

This is one of those cases where "less is more". It is likely a moderate 6 core CPU with higher clock frequency, could do the job faster.

Although I haven't had that much need for the particular tools supporting the parallel processing, I have frequently used a single tool that did support it, and yes, all 4 cores of my machine were used to the fullest, so the CPU usage at 100% and each individual core at 25% of that total. Additionally, individually spawned Python processes for each core used up quite some memory.

As to memory: well, most people actually find it desirable if the process uses less RAM. And it may actually be that your particular dataset and process flow, simply doesn't require that amount of RAM to execute, due to dataset characteristics and the way it is programmed.

Lastly, there can be an IO bottleneck, if you have a slow network connection to the location where you access the original data or store the result.

JohnBrockwell
Occasional Contributor III

I have 24 cores and I allocated 18 to ArcGIS Pro 1.4. I understand your points. However, my organization purchased thie beast and I want to see it run. I'll see what happens when I run another geoprocessing tool. Create Space Time Cube as you have stated will not leverage my hardware.

MarcoBoeringa
MVP Regular Contributor

John, unless the tool was specifically designed to take advantage of multiple cores, as witness-able by a "Parallel Processing Factor" environment setting mentioned in the ArcGIS Pro tool Help under the Environments heading, you will never see all cores fly. You also need to be aware that some processes simply can not reasonably be split up, and may never have a simple "Parallel Processing Factor" setting.

Of course, if you are able to cut up your dataset in smaller pieces, and you have ArcMap installed, you could launch multiple ArcMap sessions and have each of those process a partial dataset. Unfortunately, although I think ESRI is working on it as many people complained about this, it is currently not possible to launch multiple ArcGIS Pro sessions, you can only have one instance of ArcGIS Pro.

So, if you want to test your machine to the max, look in the ArcGIS Pro Help for the topics about Parallel Processing, and see which tools support it. It is senseless to try just any tool, because most of the hundreds of tools in ArcGIS won't support it.

Some of the tools supporting it are in the Analysis toolbox (e.g. Clip):

Analysis toolbox history—Help | ArcGIS Desktop 

Most of the tools of the Geostatistical Analyst support it:

Parallel processing with multiple CPUs—ArcGIS Pro | ArcGIS Desktop 

Probably also multiple tools in the Spatial Analyst toolboxes.

And if you have ArcMap installed - because this tool is not yet available for Pro - and have any interest in OpenStreetMap data, the latest toolbox of the ArcGIS Editor for OpenStreetMap has an OSM File Loader (Load only) tool for importing your data into a File Geodatabase, which can take advantage of multiple cores. Don't forget to set the parallel processing factor to the number of available cores though, e.g. in your case 18 under the "Environment" settings. You can download e.g. a country wide extract from Geofabrik that can serve as input for the tool. Make sure to download an .osm.bz2 zipped file, because that contains a zipped OSM XML file that is the type of input needed for the tool. Note that Windows can not unzip these by default, a program like 7-Zip can, and you will need to unzip the file before you can use it as input for the tool.

This last tool will definitely make your machine run for its money... It is the tool I have used frequently with great results and with multiple cores flying. Note that this import tool goes through multiple stages, three lengthy ones of them using multiple cores, inbetween are processes that are necessarily single core (e.g. an Append operation on database tables).

EDIT: Note that this tool also needs the Scratch Workspace from the "Environment" settings properly set. I would definitely recommend you to set the scratch workspace to the fastest SSD storage you have available, especially when trying to import with 18 cores. If you would do that using a traditional hard drive as Scratch, you might run into a serious IO bottleneck (and likely prematurely ruin the disk in the longer run).

EDIT2: Also note that the initial step of the OSM File Loader (Load only) tool is to break up the original OSM XML file in smaller pieces for the processing on each core. This process is single core and can take time on large (e.g. country size) extracts. Only after breaking up the original XML file, will it start doing parallel processing.

GabrielMarcus
New Contributor II

I am in a similar boat.

Running:  32 GB of RAM and a 6 core Intel Xeon ES-1650 obtained last month.

AcGIS Pro 2.01

Why hasn't ESRI made the core geoprocessing tools multi-processor friendly when they created ArcGIS Pro? I ran an intersect process last night on the FEMA Flood GDB. It took 3 hours to intersect 600 points on the flood zone layer. Everything is running locally on a SSD. I set the parallel processing factor to 200%

I started this process again and:

  • The resource monitor and the odd numbered cores were parked, while the even numbered cores had only 10% usage. The overall CPU usage is only 9-10% still!
  • The odd numbered cores are parked! Window is supposed to take cores out of park when they are needed to process something. 

I would appreciate any help. I have read all the pertinent help docs that I could find. 

Thanks!

JohnBrockwell
Occasional Contributor III

200% is impossible. Multiple Core CPU’s still don’t help that much. I have found that one great chip is better than all these multi-core chips that are really designed to support VM environments.

Change your parallel processing factor to 75% (24 GB’s) and build a model using model builder to run the operation and export the model to Python and run. It won’t take 6 hours.

 

Lastly, the Windows OS will not dedicate CPU Cores and RAM with the efficiency we desire for ArcGIS Pro.

Let me know if you find a better solution.

 

Best of Luck,

JB