topic Re: Calculate daily average of raster files based on their names in python in Python Questions

Calculate daily average of raster files based on their names in python

AmenehTavakol — Sat, 11 Dec 2021 19:49:26 GMT

I have hundreds of SMAP soil moisture data and need to calculate the daily average of them. For every day there are 8 files (every 3 hours) and I need a code to calculate the average based on their names.

I have a code but i am interested to know how can I define a code that average data based on their names?

# Import arcpy module
import arcpy


# Local variables:
SMAP_L4_SM_gph_20150331T043000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c15_tif = "SMAP_L4_SM_gph_20150331T043000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c15.tif"
SMAP_L4_SM_gph_20150331T073000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c0d_tif = "SMAP_L4_SM_gph_20150331T073000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c0d.tif"
SMAP_L4_SM_gph_20150331T103000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c14_tif = "SMAP_L4_SM_gph_20150331T103000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c14.tif"
SMAP_L4_SM_gph_20150331T133000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf5_tif = "SMAP_L4_SM_gph_20150331T133000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf5.tif"
SMAP_L4_SM_gph_20150331T163000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf3_tif = "SMAP_L4_SM_gph_20150331T163000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf3.tif"
SMAP_L4_SM_gph_20150331T193000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf4_tif = "SMAP_L4_SM_gph_20150331T193000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf4.tif"
SMAP_L4_SM_gph_20150331T223000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c0f_tif = "SMAP_L4_SM_gph_20150331T223000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c0f.tif"
SMAP_L4_SM_gph_20150331T013000_Vv2030_001_Geophysical_Data_sm_surface_bdea9eb8_tif = "SMAP_L4_SM_gph_20150331T013000_Vv2030_001_Geophysical_Data_sm_surface_bdea9eb8.tif"
v_name_ = "D:\\NASA\\Data\\SMAP\\2014\\Processed\\%name%.tif"

# Process: Raster Calculator
arcpy.gp.RasterCalculator_sa("( \"%SMAP_L4_SM_gph_20150331T043000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c15.tif%\" + \"%SMAP_L4_SM_gph_20150331T073000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c0d.tif%\" + \"%SMAP_L4_SM_gph_20150331T103000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c14.tif%\" +\"%SMAP_L4_SM_gph_20150331T133000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf5.tif%\" + \"%SMAP_L4_SM_gph_20150331T163000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf3.tif%\" + \"%SMAP_L4_SM_gph_20150331T193000_Vv2030_001_Geophysical_Data_sm_surface_bdea9bf4.tif%\" + \"%SMAP_L4_SM_gph_20150331T223000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c0f.tif%\" + \"%SMAP_L4_SM_gph_20150331T013000_Vv2030_001_Geophysical_Data_sm_surface_bdea9eb8.tif%\") / 8", v_name_)


Your help is really appreciated.

Re: Calculate daily average of raster files based on their names in python

BlakeTerhune — Tue, 13 Jun 2017 22:21:24 GMT

Would it be easier to use the MEAN raster property for each raster?

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Tue, 13 Jun 2017 22:37:48 GMT

Actually I need a raster file for every day (instead of 8 raster files for every day) which every pixel contains the average of soil moisture of the same pixel in 8 different time spells. By now, I can not define how arcpy identify files based on their name which contains the year month and year.

Re: Calculate daily average of raster files based on their names in python

BlakeTerhune — Tue, 13 Jun 2017 22:43:32 GMT

Can you use raster properties to average the average of every raster for the day? In other words, loop over each raster for the day to grab its average and put it in a list, then average the list; like numpy.mean()

Or are you also asking how to look at all these raster files and identify which ones are from the same day with the file name?

Re: Calculate daily average of raster files based on their names in python

curtvprice — Tue, 13 Jun 2017 22:43:56 GMT

I think you want to look into the Cell Statistics tool. The order doesn't matter, you could use Model Builder‌ or Python (arcpy.ListRasters() or arcpy.da.Walk) to create your list of rasters and pass it to the tool.

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Tue, 13 Jun 2017 22:46:54 GMT

Exactly my question is how to look at all these raster files and identify which ones are from the same day with the file name? and write a code that can define data by days and average the according to every day.

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Tue, 13 Jun 2017 22:49:01 GMT

Do you have an example of the code. I am very new in python so your help is really appreciated.

Re: Calculate daily average of raster files based on their names in python

BlakeTerhune — Tue, 13 Jun 2017 22:51:55 GMT

Is the prefix and suffix of the filenames (the stuff before and after the date) always the same? If not, is the length the same? What are the rules on how those are generated?

Re: Calculate daily average of raster files based on their names in python

curtvprice — Tue, 13 Jun 2017 22:54:31 GMT

There are code samples in the help

ListFeatureClasses—ArcPy Functions | ArcGIS Desktop

Walk—Help | ArcGIS Desktop

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Sat, 11 Dec 2021 19:49:29 GMT

These are samples of the data.

SMAP_L4_SM_gph_20150331T043000_Vv2030_001_Geophysical_Data_sm_surface_bdea9c15.tif 
SMAP_L4_SM_gph_20161231T223000_Vv2030_001_Geophysical_Data_sm_surface_352ca87d.tif
SMAP_L4_SM_gph_20170104T043000_Vv2030_001_Geophysical_Data_sm_surface_616d799c.tif

In addition to the date (year, month, day) which are different, the letters in the last part of the name (before .tif) are different.

Re: Calculate daily average of raster files based on their names in python

BlakeTerhune — Sat, 11 Dec 2021 19:49:32 GMT

Assuming the characters before the date in the filename are always the same length and all of your rasters are together in the same folder, see if this prints out the raster file names for each day that exists in the filename dates.

import arcpy
from collections import defaultdict

def main():
    raster_dir = r"D:\NASA\Data\SMAP\2014\Processed"
    arcpy.env.workspace = raster_dir
    raster_days = defaultdict(list)
    for r in arcpy.ListRasters():
        raster_days[r[15:23]].append(r)

    for date, rasters in raster_days.items():
        print(date)
        for r in rasters:
            print("\t{}".format(r))

if __name__ == '__main__':
    main()
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This creates a dictionary where the key is a date string (YYYYMMDD) and the value is a list of raster filenames that all have that date. If that works, you should be able to use this dictionary to calculate your statistics however you want.

EDIT:

Fixed typo on raster_days variable name in for loop.

Re: Calculate daily average of raster files based on their names in python

curtvprice — Wed, 14 Jun 2017 16:47:17 GMT

defaultDict is pretty fancy for a beginner, but what a useful approach! Thanks for sharing that.

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Sat, 11 Dec 2021 19:49:34 GMT

import arcpy
from collections import defaultdict

def main():
    raster_dir = r"D:\NASA\Data\SMAP\OutPut"
    arcpy.env.workspace = raster_dir
    raster_days = defaultdict(list)
    for r in arcpy.ListRasters():
        raster_days[r[15:23]].append(r)

    for date, rasters in raster_days.items():
        print(date)
        for r in rasters:
            print("\t{}".format(r))

if __name__ == '__main__':
    main()

This Code exactly classified my data into day groups. Thank you so much for your help. I have just changes "day_rasters.items()" in your code into "raster_days.items()"

Thank you so much.

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Sat, 11 Dec 2021 19:49:37 GMT

Right now I have to calculate the average for every cluster which is in this shape,

20150429
SMAP_L4_SM_gph_20150429T013000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec829.tif
SMAP_L4_SM_gph_20150429T043000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec80f.tif
SMAP_L4_SM_gph_20150429T073000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec7e9.tif
SMAP_L4_SM_gph_20150429T103000_Vv2030_001_Geophysical_Data_sm_surface_bf1ecad1.tif
SMAP_L4_SM_gph_20150429T133000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec808.tif
SMAP_L4_SM_gph_20150429T163000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec82d.tif
SMAP_L4_SM_gph_20150429T193000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec7eb.tif
SMAP_L4_SM_gph_20150429T223000_Vv2030_001_Geophysical_Data_sm_surface_bf1ec82a.tif

But I could not do that through getproperties and this code:

for rasters in main():
    Ave = np.mean()

So how can i define the average for every cluster?

Re: Calculate daily average of raster files based on their names in python

BlakeTerhune — Sat, 11 Dec 2021 19:49:40 GMT

Check out the Cell Statistics tool that Curtis Price‌ mentioned. It takes a list of raster filenames and calculates an output raster with whatever summary statistic type you choose. Using my file grouping code above, you can do something like...

for date, rasters in raster_days.items():
    # Execute CellStatistics
    outCellStatistics = arcpy.sa.CellStatistics(rasters, "MEAN")

    # Save the output
    outCellStatistics.save("C:/example/cellstats_output")‍‍‍‍‍‍‍‍‍‍‍‍

Note that it looks like you do need the spatial analyst extension for Cell Statistics.

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")‍‍

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Sat, 11 Dec 2021 19:49:43 GMT

import arcpy
import numpy as np

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")

from collections import defaultdict

def main():
    raster_dir = r"D:\NASA\Data\SMAP\OutPut"
    arcpy.env.workspace = raster_dir
    raster_days = defaultdict(list)
    for r in arcpy.ListRasters():
        raster_days[r[15:23]].append(r)

    for date, rasters in raster_days.items():
        print(date)
        for r in rasters:
            print("\t{}".format(r))

if __name__ == '__main__':
    main()

for date, rasters in raster_days.item():
        # Execute CellStatistics
        outCellStatistics = arcpy.sa.CellStatistics(rasters, "MEAN")
        # Save the output
        outCellStatistics.save("D:/NASA\Data\SMAP\OutPut/Processed")

I tried this one but I received an error "for date, rasters in raster_days.item():
NameError: name 'raster_days' is not defined"

Re: Calculate daily average of raster files based on their names in python

curtvprice — Tue, 20 Jun 2017 18:10:09 GMT

Blake there is a typo in your code above I think

    for date, rasters in day_rasters.items():

should be

    for date, rasters in raster_days.items():

Re: Calculate daily average of raster files based on their names in python

AmenehTavakol — Tue, 20 Jun 2017 18:39:41 GMT

They are "for date, rasters in raster_days.item():"

But still I receive this error!! that "NameError: name 'raster_days' is not defined".

Re: Calculate daily average of raster files based on their names in python

curtvprice — Tue, 20 Jun 2017 18:44:01 GMT

That's because you put that code in the wrong place -- outside of the main() function.

The variable raster_days is not defined outside of the function.

Re: Calculate daily average of raster files based on their names in python

BlakeTerhune — Wed, 21 Jun 2017 15:21:12 GMT

Yes, I see it now, thanks. I'll update my code.