topic Re: ArcGIS Pro Please use my RAM and CPU. in ArcGIS Pro Questions

ArcGIS Pro Please use my RAM and CPU.

JohnBrockwell — Tue, 14 Feb 2017 20:24:38 GMT

I have a new workstation that has 128 GB of RAM and 12 core Intel Xeon CPU. I am trying to run geoprocessing task using ArcGIS Pro 1.4. ArcGIS Pro is not consuming any RAM (7GB) and the CPU usage is non existent.

OS is Windows 7 SP1

What can I do to get ArcGIS Pro 1.4 to utilize my hardware?

Re: ArcGIS Pro Please use my RAM and CPU.

DanPatterson_Retired — Tue, 14 Feb 2017 20:28:44 GMT

depends on what you are doing in-memory workspaces are useful and you might want to check the specs anyway, especially your type of windows 7

Re: ArcGIS Pro Please use my RAM and CPU.

JohnBrockwell — Tue, 14 Feb 2017 20:36:06 GMT

Hi Dan,

I belive my situation is more unique. I'm not trying allocate an allotment of memory. I have 128 gb's of memory. How can I get Pro 1.4 to use 90gb's+ on a geprocessing task. I'm currently running 'Create Space Time Cube'.

Thank you for your feedback,

Re: ArcGIS Pro Please use my RAM and CPU.

DanPatterson_Retired — Tue, 14 Feb 2017 20:54:53 GMT

There is limited options that that tool supports other than current and scratch workspace (check the supported environments way at the bottom of the tool's help). I thought it might be that it was running slow or you wanted to divide the tasks like in the parallel processing factor

Re: ArcGIS Pro Please use my RAM and CPU.

MarcoBoeringa — Wed, 15 Feb 2017 18:27:50 GMT

As Dan_Patterson‌ already pointed out, this tool does not (yet) support the parallel processing factor environment setting, so it can't take advantage of multi-core CPUs. The fact that your processor has next to none activity, is probably caused by Windows only showing the combined activity of all cores. Since you have 12 cores on a single CPU, the CPU's usage is likely less than 10%, but that single core used for the actual task running at 100%.

This is one of those cases where "less is more". It is likely a moderate 6 core CPU with higher clock frequency, could do the job faster.

Although I haven't had that much need for the particular tools supporting the parallel processing, I have frequently used a single tool that did support it, and yes, all 4 cores of my machine were used to the fullest, so the CPU usage at 100% and each individual core at 25% of that total. Additionally, individually spawned Python processes for each core used up quite some memory.

As to memory: well, most people actually find it desirable if the process uses less RAM. And it may actually be that your particular dataset and process flow, simply doesn't require that amount of RAM to execute, due to dataset characteristics and the way it is programmed.

Lastly, there can be an IO bottleneck, if you have a slow network connection to the location where you access the original data or store the result.

Re: ArcGIS Pro Please use my RAM and CPU.

JohnBrockwell — Wed, 15 Feb 2017 19:59:18 GMT

Finding the Parallel Processing Setting was very useful. I appreciate your feedback.

Re: ArcGIS Pro Please use my RAM and CPU.

JohnBrockwell — Wed, 15 Feb 2017 20:02:04 GMT

I have 24 cores and I allocated 18 to ArcGIS Pro 1.4. I understand your points. However, my organization purchased thie beast and I want to see it run. I'll see what happens when I run another geoprocessing tool. Create Space Time Cube as you have stated will not leverage my hardware.

Re: ArcGIS Pro Please use my RAM and CPU.

MarcoBoeringa — Wed, 15 Feb 2017 20:46:19 GMT

John, unless the tool was specifically designed to take advantage of multiple cores, as witness-able by a "Parallel Processing Factor" environment setting mentioned in the ArcGIS Pro tool Help under the Environments heading, you will never see all cores fly. You also need to be aware that some processes simply can not reasonably be split up, and may never have a simple "Parallel Processing Factor" setting.

Of course, if you are able to cut up your dataset in smaller pieces, and you have ArcMap installed, you could launch multiple ArcMap sessions and have each of those process a partial dataset. Unfortunately, although I think ESRI is working on it as many people complained about this, it is currently not possible to launch multiple ArcGIS Pro sessions, you can only have one instance of ArcGIS Pro.

So, if you want to test your machine to the max, look in the ArcGIS Pro Help for the topics about Parallel Processing, and see which tools support it. It is senseless to try just any tool, because most of the hundreds of tools in ArcGIS won't support it.

Some of the tools supporting it are in the Analysis toolbox (e.g. Clip):

Analysis toolbox history—Help | ArcGIS Desktop

Most of the tools of the Geostatistical Analyst support it:

Parallel processing with multiple CPUs—ArcGIS Pro | ArcGIS Desktop

Probably also multiple tools in the Spatial Analyst toolboxes.

And if you have ArcMap installed - because this tool is not yet available for Pro - and have any interest in OpenStreetMap data, the latest toolbox of the ArcGIS Editor for OpenStreetMap has an OSM File Loader (Load only) tool for importing your data into a File Geodatabase, which can take advantage of multiple cores. Don't forget to set the parallel processing factor to the number of available cores though, e.g. in your case 18 under the "Environment" settings. You can download e.g. a country wide extract from Geofabrik that can serve as input for the tool. Make sure to download an .osm.bz2 zipped file, because that contains a zipped OSM XML file that is the type of input needed for the tool. Note that Windows can not unzip these by default, a program like 7-Zip can, and you will need to unzip the file before you can use it as input for the tool.

This last tool will definitely make your machine run for its money... It is the tool I have used frequently with great results and with multiple cores flying. Note that this import tool goes through multiple stages, three lengthy ones of them using multiple cores, inbetween are processes that are necessarily single core (e.g. an Append operation on database tables).

EDIT: Note that this tool also needs the Scratch Workspace from the "Environment" settings properly set. I would definitely recommend you to set the scratch workspace to the fastest SSD storage you have available, especially when trying to import with 18 cores. If you would do that using a traditional hard drive as Scratch, you might run into a serious IO bottleneck (and likely prematurely ruin the disk in the longer run).

EDIT2: Also note that the initial step of the OSM File Loader (Load only) tool is to break up the original OSM XML file in smaller pieces for the processing on each core. This process is single core and can take time on large (e.g. country size) extracts. Only after breaking up the original XML file, will it start doing parallel processing.

Re: ArcGIS Pro Please use my RAM and CPU.

GabrielMarcus — Sat, 11 Nov 2017 00:58:30 GMT

I am in a similar boat.

Running: 32 GB of RAM and a 6 core Intel Xeon ES-1650 obtained last month.

AcGIS Pro 2.01

Why hasn't ESRI made the core geoprocessing tools multi-processor friendly when they created ArcGIS Pro? I ran an intersect process last night on the FEMA Flood GDB. It took 3 hours to intersect 600 points on the flood zone layer. Everything is running locally on a SSD. I set the parallel processing factor to 200%

I started this process again and:

The resource monitor and the odd numbered cores were parked, while the even numbered cores had only 10% usage. The overall CPU usage is only 9-10% still!
The odd numbered cores are parked! Window is supposed to take cores out of park when they are needed to process something.

I would appreciate any help. I have read all the pertinent help docs that I could find.

Thanks!

Re: ArcGIS Pro Please use my RAM and CPU.

JohnBrockwell — Mon, 13 Nov 2017 14:09:58 GMT

200% is impossible. Multiple Core CPU’s still don’t help that much. I have found that one great chip is better than all these multi-core chips that are really designed to support VM environments.

Change your parallel processing factor to 75% (24 GB’s) and build a model using model builder to run the operation and export the model to Python and run. It won’t take 6 hours.

Lastly, the Windows OS will not dedicate CPU Cores and RAM with the efficiency we desire for ArcGIS Pro.

Let me know if you find a better solution.

Best of Luck,

Re: ArcGIS Pro Please use my RAM and CPU.

GabrielMarcus — Mon, 05 Feb 2018 21:51:27 GMT

Actually 200% isn't impossible, according to the help files, but it isn't what I meant to say. I put 100% in the Parallel Processing Factor field.

Doing the same task in ArcGIS Pro takes less time, because I think it uses parallel processing better.

Re: ArcGIS Pro Please use my RAM and CPU.

JohnBrockwell — Tue, 06 Feb 2018 13:01:49 GMT

I'd be interested in seeing the help files. Do you have a link you could share? Parallel Processing depends on the geoprocessing tool. It really is a tool by tool decision as some tools can't utilize parallel processing.

You may have to use a 100% but your CPU will still have to allocate resources to the Operating System. Hence, you can't use 100% of the CPU.

Good Discussion. I'd like to start a list of tools that can use Parallel Processing. Please let me know what geoprocessing tools you have used.

Re: ArcGIS Pro Please use my RAM and CPU.

Karyn_Kerdolff — Thu, 03 May 2018 21:33:13 GMT

There are a couple ground-level concepts that are useful to remember when it comes to multi-core/multi-thread processors.

First, as many have covered already, not all tasks can be run in parallel. If step 8 relies on the output of step 7, which in turn relies on step 6, throwing more cores at the task won't do anything to help it run faster. You can't run step 6 on one core, step 7 on another, and step 8 on a third (well, you probably could, but you'd still be waiting for step 6 before moving on to step 7). Writing truly parallel code can be rather tricky.

Second, most high-end CPUs now support Symmetric Multi Threading, which takes advantage of CPU architecture wizardry to run two threads on one physical core (Anandtech has a great primer on how this works). Because of SMT, Windows sees a 4-core Intel i7 as actually having 8 cores. I don't have one in front of me (currently running a "lowly" i5 without SMT), but if memory serves these "extra cores" usually show up as the odd-numbered cores in Windows. Again, not all tasks are capable of taking advantage of SMT, so it is not surprising to often see all these cores not being used very heavily. I suspect this is what's happening in Gabriel Marcus' case.

Re: ArcGIS Pro Please use my RAM and CPU.

ThomasColson — Sun, 15 Jul 2018 15:30:47 GMT

Speaking of core parking, there is a registry setting you can change that will permanently unpark them......

Re: ArcGIS Pro Please use my RAM and CPU.

JayJohnson6 — Tue, 02 Oct 2018 19:28:12 GMT

Is this it?

https://www.thewindowsclub.com/enable-disable-core-parking-windows

Re: ArcGIS Pro Please use my RAM and CPU.

ThomasColson — Wed, 03 Oct 2018 19:08:36 GMT

No! Never use an "App" that manages core parking. They're all viruses. There's a registry change somewhere....

Re: ArcGIS Pro Please use my RAM and CPU.

JayJohnson6 — Wed, 03 Oct 2018 19:14:38 GMT

https://www.thewindowsclub.com/intel-extreme-tuning-utility

https://www.thewindowsclub.com/amd-overdrive-utility-helps-you-overclock

Maybe these would be safer.

Re: ArcGIS Pro Please use my RAM and CPU.

JayJohnson6 — Wed, 03 Oct 2018 19:17:03 GMT

Does this seem right?

https://shurikenskribbles.wordpress.com/2018/01/03/unparking-cpu-cores/

I can test it if you think it resembles your last try.

Re: ArcGIS Pro Please use my RAM and CPU.

ThomasColson — Wed, 03 Oct 2018 19:21:54 GMT

I'll look for the registry URL, but my 2 cents is that every single "unpark cpu" app is a virus.....