topic Does anyone know why the Build Overviews tool would refuse to use Parallels processing when it has previously done it by default? in Geoprocessing Questions

Does anyone know why the Build Overviews tool would refuse to use Parallels processing when it has previously done it by default?

MDB_GIS — Thu, 19 Oct 2023 19:50:11 GMT

I am working on creating a written workflow for handling a large imagery dataset at my organization and teaching future users how to create them. Quick summary of my workflow:

1. Create Mosaic Dataset

2. Add Rasters to Mosaic Dataset

3. Define Overviews

4. Build Overviews

Step 4 is where I encountered the issue. On my first test, building the overviews took 35 minutes for ~900 imagery tiles. During step 3. I designated that the overviews be stored in the file geodatabase.

Once I finished my documentation, I decided run through it to make sure it was coherent and easy to follow. I did everything the same, except this time, when Defining Overviews in step 3, I did not designate them to be stored in a file goedatabase, but rather a folder on my C drive. When I clicked run, I immediately noticed that the tool was going much slower. When it finally finished it had taken over 5 hours to run.

I decided to comb through the tool logs in the GP history panel and I came across one big discrepancy.

Just before the tool started generating overviews, it said the following:

2023-10-18T13:44:59.883: Distributing mosaic dataset operation across 10 parallel instances on the specified host: [GIS-0123].

When I go to that same spot in the log for the second run, I can clearly see that it did not say anything about parallel processing. Here is a screenshot from the logs (the log on the right is the 35 minute run and the log on the left is the the 5 hour run).

The only thing I did differently was in the defining overview stage. So I can't figure out why the parallel processing was affected. I did not tweak the parallel processing environment settings in either tool. Anyone have any experience with this issue?

Re: Does anyone know why the Build Overviews tool would refuse to use Parallels processing when it has previously done it by default?

DanPatterson — Thu, 19 Oct 2023 21:15:14 GMT

Build Overviews (Data Management)—ArcGIS Pro | Documentation

Parallel Processing factor refers to Build Overviews

Parallel Processing Factor (Environment setting)—ArcGIS Pro | Documentation

However, for cases in which all your processes are I/O bound to a disk or to an enterprise database connection, you may get better performance by specifying more processes than you have cores. For example, the Add Rasters to Mosaic Dataset tool is I/O bound when the mosaic dataset is stored in an enterprise database. Also, the Build Overviews tool is primarily I/O bound to the disk. You can use more processes than your machine has cores by specifying either a percent value greater than 100% or a number of processes greater than the number of cores on your machine. For example, if you have a 4-core machine, specifying 8 or 200% will spread operations over 8 processes.

Re: Does anyone know why the Build Overviews tool would refuse to use Parallels processing when it has previously done it by default?

MDB_GIS — Fri, 20 Oct 2023 14:44:29 GMT

That's very interesting. I know you didn't write the documentation but based off all your helpful posts across the community, I think you know the ins and outs of the software better than many ESRI employees. So do you know what they mean when they say "I/O Bound?" I know that means input/output, but the way I'm reading that, it seems to imply that writing to a file GDB isn't I/O bound.

Isn't pretty much everything on a computer I/O bound at some level? I'm trying to understand the difference between writing to a folder on my local drive and writing to a file geodatabase is in terms of I/O.

Regarding the value you use for parallel processing if you manually input it, do you know what the limit is? Based on the example they provided, are you limited to 2 process per core? So for example, I have an i7 -12700 which has 20 threads (8 p cores and 4 e cores). Which number would I double? I'm guessing the P cores since they are typically the workhorse cores? So I would use 16?

Sorry for throwing all these questions at you! I'm trying to learn this stuff as best I can so I can document it for the future!