topic Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics in Python Questions

Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Sun, 19 Nov 2017 15:00:42 GMT

Hi everyone, I have folder with thousands of average daily relative humidity raster files that I would like to use cell statistics to calculate the mean in 8 day intervals. The data has dates in the format YYYYMMDD. I've looked at a similar code written after a similar question was asked but it doesnt do the 8 day averaging. I've included the link to the other answered question and a picture of what the data look like. Your help is greatly appreciated.Calculate daily average of raster files based on their names in python xander_bakker

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

XanderBakker — Sun, 19 Nov 2017 15:13:28 GMT

Hi Larry Biodun ,

First thing I notice is the "+" sign in the name of the img files. This may generate problems when you use them in any mathematical operation in ArcGIS / Python. So it might be necessary to start with renaming the rasters and shorten the names. This can be done with Python. Would this be an option?

Edit:

I also notice that half of the raster have a "T" after the "+" sign. And 2 rasters will reference the same date. Should the 8 day calculation include 16 rasters?

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Sun, 19 Nov 2017 15:38:11 GMT

Hi Xander, thanks for your quick response. That was an error in that sample list i sent. Thanks for picking that up. The 8 day calculation should only include 8 rasters. I have included another pix of the actual correct data. I am happy to shorten the names. Infact anything after the first 14 characters dont matter.

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

DanPatterson_Retired — Sun, 19 Nov 2017 16:22:41 GMT

there is a thread here which Xander Bakker‌ led the charge on. https://community.esri.com/message/584750?commentID=584750#comment-584750

Then a parallel incarnation here https://community.esri.com/thread/173806 shoul

you require more statistical style calculations in either a block or running style calculation

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

XanderBakker — Sun, 12 Dec 2021 00:16:46 GMT

The first step would be to apply some proper naming for the files. See below a snippet of code that could do this:

def main():
    from shutil import copyfile, move
    import os

    # list of extensions you want to copy or rename
    lst_ext = ['.img']

    input_folder = r'C:\GeoNet\Average8Days\input_files'
    output_folder = r'C:\GeoNet\Average8Days\output_files'

    for path, dirs, files in os.walk(input_folder):
        for filename in files:
            name, ext = os.path.splitext(filename)
            if ext in lst_ext:
                in_file = os.path.join(path, filename)
                print in_file
                out_name = name[:12] + ext
                out_file = os.path.join(output_folder, out_name)
                print out_file

                # decide to copy
                copyfile(in_file, out_file)

                # or move:
                move(in_file, out_file)

                # or rename:
                os.rename(in_file, out_file)


if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

On line 6, you will have to include the extensions of the files you want to copy, move or rename.

On line 8 and 9 specify the paths of the input and output folder

Line 21 to 28, decide if you want to copy, which is safest, but you will need to additional disk space, or move or rename. If you have any dependencies on the current file name and locations I recommend you to copy the data.

Once you have the files in with usable names, you are ready for the next step.

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Sun, 19 Nov 2017 22:57:52 GMT

Thanks Xander. The renaming is done. I chose the copy part of the code. I'll wait on the averaging code. Here is a snapshot of the copied files.

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

DanPatterson_Retired — Sun, 19 Nov 2017 23:03:59 GMT

You can prep by reading Xander's post here, where the shell is outlined

https://community.esri.com/thread/171685?commentID=584750#comment-584750

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

XanderBakker — Sun, 19 Nov 2017 23:29:17 GMT

Can you check if the renamed files open properly in ArcGIS (just open one), since only the .img file was copied. It will probably recreate the aux and xml files, but just to be sure that the data is valid.

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Sun, 19 Nov 2017 23:41:20 GMT

Thanks for that Dan

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Sun, 19 Nov 2017 23:43:24 GMT

Yes they do open in ArcGIS properly. The associated files were created when I opened it in ArcGIS.

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

XanderBakker — Sun, 12 Dec 2021 00:16:49 GMT

It is good to hear that the files open correctly. In case the dates are consecutive and there will always be 8 elements for each calculation, I guess the chunk method mentioned in the thread referenced by Dan should work. I just ran this snippet:

def main():
    import os

    my_list = ["calc200601{0}.img".format("%02d" % (a,)) for a in range(1, 31)]
    out_ext = ".tif"

    for my_list_chunk in chunks(my_list, 8):
        first_name = my_list_chunk[0]
        name, ext = os.path.splitext(first_name)
        first_date = name[-8:]
        out_ras_name = "mean" + first_date + out_ext
        print out_ras_name, my_list_chunk

def chunks(l, n):
    """ Yield successive n-sized chunks from l."""
    for i in xrange(0, len(l), n):
        yield l[i:i+n]

if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Which yielded:

mean20060101.tif ['calc20060101.img', 'calc20060102.img', 'calc20060103.img', 'calc20060104.img', 'calc20060105.img', 'calc20060106.img', 'calc20060107.img', 'calc20060108.img']
mean20060109.tif ['calc20060109.img', 'calc20060110.img', 'calc20060111.img', 'calc20060112.img', 'calc20060113.img', 'calc20060114.img', 'calc20060115.img', 'calc20060116.img']
mean20060117.tif ['calc20060117.img', 'calc20060118.img', 'calc20060119.img', 'calc20060120.img', 'calc20060121.img', 'calc20060122.img', 'calc20060123.img', 'calc20060124.img']
mean20060125.tif ['calc20060125.img', 'calc20060126.img', 'calc20060127.img', 'calc20060128.img', 'calc20060129.img', 'calc20060130.img']‍‍‍‍

So, each chunk contains 8 rasters and the output raster takes the date of the first raster. In this case I added the .tif extension, but you could leave that extension off to create a Esri grid in a folder or point to an output fgdb.

To validate is this shell works, could you run the snippet below and see if this splits the list of raster up into the chunks you want?

def main():
    import arcpy
    import os

    # change this folder to the folder with the renamed img files
    input_folder = r'C:\GeoNet\Average8Days\output_files'
    out_ext = ".tif"

    arcpy.env.workspace = input_folder
    my_list = arcpy.ListRasters()

    for my_list_chunk in chunks(my_list, 8):
        first_name = my_list_chunk[0]
        name, ext = os.path.splitext(first_name)
        first_date = name[-8:]
        out_ras_name = "mean" + first_date + out_ext
        print out_ras_name, my_list_chunk

def chunks(l, n):
    """ Yield successive n-sized chunks from l."""
    for i in xrange(0, len(l), n):
        yield l[i:i+n]

if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Mon, 20 Nov 2017 06:56:36 GMT

Hi Xander, works just like you said! However, as I really am a newbie to this scripting thing, I do not know where to insert the other code to run the actual cell statistics as I realise the above code just prints out the batches of 8 files. Could you please help that little bit further with where to insert the cell statistics code.

Thanks

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

XanderBakker — Sun, 12 Dec 2021 00:16:52 GMT

Hi larry32 , can you try this code?

Change:

input folder with .img files on line 6
output folder or fgdb on line 9 and the corresponding extension on line 10

def main():
    import arcpy
    import os

    # change this folder to the folder with the renamed img files
    input_folder = r'C:\GeoNet\Average8Days\output_files'

    # define the output workspace and extension
    output_ws = r'C:\GeoNet\Average8Days\gdb\myFileGeoDB.gdb'
    out_ext = ""

    # create list of rasters in input workspace
    arcpy.env.workspace = input_folder
    my_list = arcpy.ListRasters()

    # checkout a Spatial Analyst license
    arcpy.CheckExtension("Spatial")

    # loop through chunks of 8 rasters
    for my_list_chunk in chunks(my_list, 8):
        # define the output name
        first_name = my_list_chunk[0]
        name, ext = os.path.splitext(first_name)
        first_date = name[-8:]
        out_ras_name = "mean" + first_date + out_ext
        print out_ras_name, my_list_chunk

        # now we have:
        # - an output raster filename
        # - the list of rasters for calculating the mean value

        # perform the cellstatistics with MEAN
        cellstat = arcpy.sa.CellStatistics(my_list_chunk, "MEAN", "DATA")

        # define file path of output raster and save cell statistics result
        outname = os.path.join(output_ws, out_ras_name)
        cellstat.save(outname)


def chunks(l, n):
    """ Yield successive n-sized chunks from l."""
    for i in xrange(0, len(l), n):
        yield l[i:i+n]

if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Calculate mean in 8 day intervals for thousand of daily raster files using cell statistics

LarryBiodun — Tue, 21 Nov 2017 10:54:23 GMT

Thanks Xander! Works like a dream. Very quick too.

Thanks Dan for the useful comments. I appreciate it all.