Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2'

marcozanieri

Hello,

I have just started to use 'ArcGIS Pro 3.2.2' (before I used 'ArcGIS Pro 3.1') and I'm working with the tool 'Batch Calculate Statistics':

I have used this tool with 4 geotif files:

3880

With the following settings:

Here below all the details and the time that the application needs to complete this task:

Batch Calculate Statistics
=====================
Parameters

Input Raster Datasets \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
X Skip Factor 1
Y Skip Factor 1
Ignore values
Skip Existing
Batch Calculate Statistics Succeeded true
=====================
Messages

Start Time: sabato 27 aprile 2024 11:33:12
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
Succeeded at sabato 27 aprile 2024 11:37:05 (Elapsed Time: 3 minutes 53 seconds)

------------------------

If I try to do the same using 'ArcGIS Pro 3.1' the performances are much much much BETTER:

Batch Calculate Statistics
=====================
Parameters

Input Raster Datasets \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
X Skip Factor 1
Y Skip Factor 1
Ignore values
Skip Existing
Batch Calculate Statistics Succeeded true
=====================
Messages

Start Time: Saturday, 27 April, 2024 11:44:21
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
Succeeded at Saturday, 27 April, 2024 11:44:27 (Elapsed Time: 6.09 seconds)

SO THIS IS THE RESULT:

Software	Geoprocessing tool	Elapsed Time
ArcGIS Pro 3.2.2	Batch Calculate Statistics	3 minutes 53 seconds
ArcGIS Pro 3.1	Batch Calculate Statistics	6.09 seconds

Spoiler

THE TIME IN ArcGIS Pro 3.2.2 IS ABOUT THE 3880% IN MORE THAN IN ArcGIS Pro 3.1

This is a big issue for us, do you know if ESRI is working to improve these performances like ArcGIS Pro 3.1 ?

Thanks a lot for your support!

Regards,

Marco Zanieri

MarcoBoeringa

With such a big difference, and all other things equal, I would definitely recommend contacting ESRI support, they should be able to tell you if this is a known issue, and whether it is scheduled for a fix. It is not the first time ArcGIS suffers a performance regression and may need a fix.

DavidColey

I can't even get past 2 tif files without the either the tool freezing up or my python script just hanging after taking 3 - 5 minutes per file, no matter if I am using a local workspace or across the network. Typically, I run something like:

import sys, os, arcpy, time, smtplib
from datetime import date, datetime, timedelta
from arcpy import env

# Local variables...
message = ""
current_time = time.asctime()
todaysDate = date.today()
priorYear = str(todaysDate - timedelta(days=547)) #547
starting_time = time.time()

arcpy.SetLogHistory(False)
env.configKeyword= "DEFAULTS"
env.overwriteOutput = True

try:
env.workspace = "D:\\Imagery"
#env.workspace = r"E:\\Tiles\\GeoTIFF\\Property"
# Had to go into directory and give account full control - see if works for loop, after 2 files locks up
#env.workspace = r"\\networkshare\\imagery\\2024\\GeoTIFFs"
rasters = arcpy.ListRasters("*", "TIF")
for raster in rasters:
print(raster)
arcpy.CalculateStatistics_management(raster, "1", "1", "", "SKIP_EXISTING", "")
print (raster + " stats calculated")
message = message + "\n\n" + raster + " stats calculated"

except Exception:
# If an error occurred, print line number and error message
import traceback, sys
tb = sys.exc_info()[2]
e = sys.exc_info()[1]
print(e.args[0])
print ("Line %i" % tb.tb_lineno)
message = message + "\n" + "Line %i" % tb.tb_lineno
message = message + "\n" + str(e)

And like I said, it won't get past 2 tif files before just hanging. I should mention I have almost 900 of these - out entire county, 3 band tif images, 4" resolution. I used the same script workflow above to BuildPyramids, that took about 8 hours. Not great, but I can live with that.

@KenBuja - would you happen to have any ideas?

Thanks,

David

RTPL_AU

Link to Idea:
when-releasing-a-new-version-of-pro-benchmark-gp

@SSWoodward @Bud

DavidColey

I moved over to Catalog and was able to batch calc raster statistics for 835 tif files in like 6 minutes. Fortunately I still have an 10.8.2 install on a laptop...