topic Re: Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2' in ArcGIS Pro Questions

Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2'

marcozanieri — Sat, 27 Apr 2024 09:58:39 GMT

Hello,

I have just started to use 'ArcGIS Pro 3.2.2' (before I used 'ArcGIS Pro 3.1') and I'm working with the tool 'Batch Calculate Statistics':

I have used this tool with 4 geotif files:

3880

With the following settings:

Here below all the details and the time that the application needs to complete this task:

Batch Calculate Statistics
=====================
Parameters

Input Raster Datasets \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif;\\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
X Skip Factor 1
Y Skip Factor 1
Ignore values
Skip Existing
Batch Calculate Statistics Succeeded true
=====================
Messages

Start Time: sabato 27 aprile 2024 11:33:12
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
Succeeded at sabato 27 aprile 2024 11:37:05 (Elapsed Time: 3 minutes 53 seconds)

------------------------

If I try to do the same using 'ArcGIS Pro 3.1' the performances are much much much BETTER:

Batch Calculate Statistics
=====================
Parameters

Start Time: Saturday, 27 April, 2024 11:44:21
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_30_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_31_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_32_GRID.tif
Successfully calculated statistics : \\gisdata.eni.pri\GISDATA\WORK_AREA\SYSTEM_TEST\_DatiTest\EnGISPro\XwareLoader\Loader_Dem\dsm_33_GRID.tif
Succeeded at Saturday, 27 April, 2024 11:44:27 (Elapsed Time: 6.09 seconds)

SO THIS IS THE RESULT:

Software	Geoprocessing tool	Elapsed Time
ArcGIS Pro 3.2.2	Batch Calculate Statistics	3 minutes 53 seconds
ArcGIS Pro 3.1	Batch Calculate Statistics	6.09 seconds

THE TIME IN ArcGIS Pro 3.2.2 IS ABOUT THE 3880% IN MORE THAN IN ArcGIS Pro 3.1

This is a big issue for us, do you know if ESRI is working to improve these performances like ArcGIS Pro 3.1 ?

Thanks a lot for your support!

Regards,

Marco Zanieri

Re: Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2'

MarcoBoeringa — Sun, 28 Apr 2024 09:21:48 GMT

With such a big difference, and all other things equal, I would definitely recommend contacting ESRI support, they should be able to tell you if this is a known issue, and whether it is scheduled for a fix. It is not the first time ArcGIS suffers a performance regression and may need a fix.

Re: Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2'

DavidColey — Mon, 29 Apr 2024 14:53:57 GMT

I can't even get past 2 tif files without the either the tool freezing up or my python script just hanging after taking 3 - 5 minutes per file, no matter if I am using a local workspace or across the network. Typically, I run something like:

import sys, os, arcpy, time, smtplib from datetime import date, datetime, timedelta from arcpy import env # Local variables... message = "" current_time = time.asctime() todaysDate = date.today() priorYear = str(todaysDate - timedelta(days=547)) #547 starting_time = time.time() arcpy.SetLogHistory(False) env.configKeyword= "DEFAULTS" env.overwriteOutput = True try: env.workspace = "D:\\Imagery" #env.workspace = r"E:\\Tiles\\GeoTIFF\\Property" # Had to go into directory and give account full control - see if works for loop, after 2 files locks up #env.workspace = r"\\networkshare\\imagery\\2024\\GeoTIFFs" rasters = arcpy.ListRasters("*", "TIF") for raster in rasters: print(raster) arcpy.CalculateStatistics_management(raster, "1", "1", "", "SKIP_EXISTING", "") print (raster + " stats calculated") message = message + "\n\n" + raster + " stats calculated" except Exception: # If an error occurred, print line number and error message import traceback, sys tb = sys.exc_info()[2] e = sys.exc_info()[1] print(e.args[0]) print ("Line %i" % tb.tb_lineno) message = message + "\n" + "Line %i" % tb.tb_lineno message = message + "\n" + str(e)

And like I said, it won't get past 2 tif files before just hanging. I should mention I have almost 900 of these - out entire county, 3 band tif images, 4" resolution. I used the same script workflow above to BuildPyramids, that took about 8 hours. Not great, but I can live with that.

@KenBuja - would you happen to have any ideas?

Thanks,

David

Re: Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2'

RTPL_AU — Tue, 30 Apr 2024 02:03:14 GMT

Link to Idea:
when-releasing-a-new-version-of-pro-benchmark-gp

@SSWoodward @Bud

Re: Low performances using 'Batch Calculate Statistics' in 'ArcGIS Pro 3.2'

DavidColey — Wed, 01 May 2024 20:13:02 GMT

I moved over to Catalog and was able to batch calc raster statistics for 835 tif files in like 6 minutes. Fortunately I still have an 10.8.2 install on a laptop...