Iterate raster and Zonal statistics at table too slow

Question

I am using Iterate raster and Zonal statistics at table to compute 'ALL' statistical attributes of 4000 raster files and then convert output to excel files. The process is quick at the beginning but gets extremely slow midway..... The first half takes like four hours and the remaining half another 24 hours!!!! I have tried remedies online but nothing works. Please Help.... I still have like 11 more batches of 4000 files each

PaulineOnjira · Answer

Dear Dan,

Thank you for the idea........ It took me around 2 hrs to get the code working but am happy it's faster: i would like to share in the following section;

import arcpy, os
from arcpy import env
from arcpy.sa import *
env.workspace = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT"
arcpy.env.overwriteoutput = True

B3 = "D:/GIS/sub_basins/B4H003.shp"
B5 = "D:/GIS/sub_basins/B4H005.shp"
B7 = "D:/GIS/sub_basins/B4H007/B4H007.shp"
B9 = "D:/GIS/sub_basins/B4H009/b4H009.shp"
B10 = "D:/GIS/sub_basins/B4H010/B4H010.shp"
whole = "D:/GIS/sub_basins/whole/whole.shp"

outDir1 = "D:/KIT/ACAD/SMC/STEFAN/Statistics/D_SM_EXT/B4H003/"
outDir2 = "D:/KIT/ACAD/SMC/STEFAN/Statistics/D_SM_EXT/B4H005/"
outDir3 = "D:/KIT/ACAD/SMC/STEFAN/Statistics/D_SM_EXT/B4H007/"
outDir4 = "D:/KIT/ACAD/SMC/STEFAN/Statistics/D_SM_EXT/B4H009/"
outDir5 = "D:/KIT/ACAD/SMC/STEFAN/Statistics/D_SM_EXT/B4H010/"
outDir6 = "D:/KIT/ACAD/SMC/STEFAN/Statistics/D_SM_EXT/whole/"

inDir1 = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT/"
inDir2 = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT/"
inDir3 = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT/"
inDir4 = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT/"
inDir5 = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT/"
inDir6 = "D:/KIT/ACAD/SMC/STEFAN/D_TIFF_SM_EXT/"

rasters = arcpy.ListRasters ()
for Raster in rasters:
 print(Raster)

temp = os.path.splitext(Raster)[0]

outTable1 = temp
 arcpy.sa.ZonalStatisticsAsTable(B3,"DN",Raster,outTable1,"NODATA","ALL")
 in_table1 = inDir1 + temp
 out_xls1 = outDir1 + "B4H003_" + temp + ".xls"
 arcpy.TableToExcel_conversion(in_table1, out_xls1)

outTable2 = temp
 arcpy.sa.ZonalStatisticsAsTable(B5,"DN",Raster,outTable2,"NODATA","ALL")
 in_table2 = inDir2 + temp
 out_xls2 = outDir2 + "B4H005_" + temp + ".xls"
 arcpy.TableToExcel_conversion(in_table2, out_xls2)

outTable3 = temp
 arcpy.sa.ZonalStatisticsAsTable(B7,"DN",Raster,outTable3,"NODATA","ALL")
 in_table3 = inDir3 + temp
 out_xls3 = outDir3 + "B4H007_" + temp + ".xls"
 arcpy.TableToExcel_conversion(in_table3, out_xls3)

outTable4 = temp
 arcpy.sa.ZonalStatisticsAsTable(B9,"DN",Raster,outTable4,"NODATA","ALL")
 in_table4 = inDir4 + temp
 out_xls4 = outDir4 + "B4H009_" + temp + ".xls"
 arcpy.TableToExcel_conversion(in_table4, out_xls4)

outTable5 = temp
 arcpy.sa.ZonalStatisticsAsTable(B10,"DN",Raster,outTable5,"NODATA","ALL")
 in_table5 = inDir5 + temp
 out_xls5 = outDir5 + "B4H010_" + temp + ".xls"
 arcpy.TableToExcel_conversion(in_table5, out_xls5)

outTable6 = temp
 arcpy.sa.ZonalStatisticsAsTable(whole,"DN",Raster,outTable6,"NODATA","ALL")
 in_table6 = inDir6 + temp
 out_xls6 = outDir6 + "whole_" + temp + ".xls"
 arcpy.TableToExcel_conversion(in_table6, out_xls6)

DanPatterson_Retired · Answer

my next question would be have you tried smaller batches?

How long does 250 rasters take? 500? 1000?

I think your batches are too large for your system and the limitations of arcmap and you might find that numerous smaller batches would get the job done faster.

Second suggestion... write your code in python, you can use the ZonalStatisticsAsTable tool in the code and the TableToExcel within a for loop, looping through your rasters.  Iterators currently don't translate well from modelbuilder, but if you post your model and the exported script I am sure someone might have a quick solution.

PaulineOnjira · Answer

Hello Peterson,

Thanks for responding. How would I do this in a Model?

My Model had two steps only:

Iterate rasters -----> Zonal Statistics at Table -----> Table to Excel

I only save the output of Table to Excel.

I have read about using scratch space to Output intermediate data, but when I do this it doesn't execute the last step (Table to excel)

Kindly help