topic Re: Parallel Processing Factor in Geoprocessing Questions

Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 05:56:00 GMT

I am using Parallel Processing Factor to speed up a process that has been taking too long. I have a question though. I have a workstation computer that has a 16-core processor. Each core has two logical processors. How many processes would it do if arcpy.env.parallelProcessingFactor was set to 50%, 75%, or 100%, given that fact?

Re: Parallel Processing Factor

DanPatterson_Retired — Thu, 28 Mar 2019 06:38:15 GMT

Parallel Processing Factor (Environment setting)—Geoprocessing | ArcGIS Desktop

has the methods used to do the calculation

Re: Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 15:34:56 GMT

I have seen the page, but it does not mention logical processors of which there 32, 2 for each core. How do I take that into account?

Re: Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 16:10:07 GMT

Also, I put it in a user-defined function in Python. What do you think might be wrong with that?

Re: Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 18:09:31 GMT

Also, I am using the Mosaic to New Raster tool.

Re: Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 18:11:19 GMT

Does that tool honor the Parallel Processing Factor?

Re: Parallel Processing Factor

DanPatterson_Retired — Thu, 28 Mar 2019 18:46:26 GMT

every tool... including

Mosaic To New Raster—Data Management toolbox | ArcGIS Desktop

has a section at the bottom of the help topic which states environment factors that are supported with an disclaimers....

Environments

Compression, Extent, NoData, Output CONFIG Keyword, Parallel Processing Factor, Pyramid, Raster Statistics, Resampling Method, Tile Size

Extent

Not valid for enterprise geodatabase

Re: Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 20:30:45 GMT

Yes I noticed. However, I am using ArcGIS 10.6. I do not have ArcGIS Pro, unfortunately. The bad news with that is that Parallel Processing Factor is not listed in the Environments section on the page for Mosaic to New Raster. That environment is only available starting with ArcGIS 10.7 for that tool. Bother.

Re: Parallel Processing Factor

NathanFleischman — Thu, 28 Mar 2019 20:32:08 GMT

However, if I were to get ArcGIS Pro or ArcGIS 10.7, how do I take logical processors into account.

Re: Parallel Processing Factor

DuncanHornby — Fri, 29 Mar 2019 10:15:26 GMT

The link that Dan Patterson gives clearly talks about number of cores ... "processes will be divided between hardware cores (processors) built into the machine". You say you have 16 cores then setting this environment setting to 100% will use all 16 cores. As that page never mentions "logical processors" I would assume it would mean that tools would not distribute work over 32 logical processors.

Re: Parallel Processing Factor

spatialcoder — Thu, 18 Mar 2021 13:45:22 GMT

Why not use the Multiprocessing module that comes with the baseline python libraries? This typically splits things out by "threat" (Logical Core). At least that is the case on the 6 core i& I had that had 12 logical processors. We also use it for multi processing on virtual server environments, which don't normally list logical cores, so your "cpu count" would equate to the cores on the machine.

More so, if your machine displays "logical processors", then with the multiprocessing module, you can call the process to split processes out over each logical core that exists. In the case of my 6 core i7, I had 6 cores, but could split the processes out over the 12 logical processors.