topic Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy in Python Questions

Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sat, 11 Dec 2021 16:36:00 GMT

I have five subfolders with Rasters name with the same parameter. I am trying to figure out a way to Loop average of all raster and save output raster in outfolder . For instance

Subfolder1   Subfolder2   Subfolder3   Subfolder4    Subfolder5   Outfolder
r001_NPP.tif r001_npp.tif r001_NPP.tif r_001_NPP.tif r001_NPP.tif r001_MEAN_NPP.tif‍‍‍‍‍‍

‍‍‍‍‍‍‍

I have tried to write code for this, Below i post my code what i have so far

import arcpy, os, glob
arcpy.CheckOutExtension("Spatial")

ws1 = glob.glob(r'D:\subfolder1\*.tif')
ws2 = glob.glob(r'D:\subfolder2\*.tif')
ws3 = glob.glob(r'D:\subfolder3\*.tif')
ws4 = glob.glob(r'D:\subfolder4\*.tif')
ws5 = glob.glob(r'D:\subfolder5\*.tif')
outws = r'D:\outfolder'

for r in ws1, ws2, ws3, ws4:
    basename = os.path.basename(r).split("_")[0]
    r1 = arcpy.sa.Raster(os.path.join(ws1, basename + "r_NPP.tif"))
    r2 = arcpy.sa.Raster(os.path.join(ws2, basename + "r_NPP.tif"))
    r3 = arcpy.sa.Raster(os.path.join(ws3, basename + "r_NPP.tif"))
    r4 = arcpy.sa.Raster(os.path.join(ws4, basename + "r_NPP.tif"))
    r5 = arcpy.sa.Raster(os.path.join(ws5, basename + "r_NPP.tif"))
    

    result = (r1 + r2 + r3 + r4 + r5) / 5
    outname = basename + "r_Mean_NPP.tif"

    result.save(os.path.join(outws, outname))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

curtvprice — Sat, 11 Dec 2021 16:36:03 GMT

I would look into arcpy.da.walk() to generate the list of rasters, and use the Spatial Analyst Cell Statistics tool on the raster list.

Walk—Help | ArcGIS for Desktop

Cell Statistics—Help | ArcGIS for Desktop

Cell Statistics is much more efficient, especially with long lists of rasters -- and it also has a very handy DATA/NODATA option so null is treated as zero in the cell by cell summation.

Untested code:

import arcpy
arcpy.CheckOutExtension("spatial")
from arcpy.sa import *

# generate list of rasters
workspace = r"D:\"
rasters = []
walk = arcpy.da.Walk(workspace, datatype="RasterDataset", type="TIF")
for dirpath, dirnames, filenames in walk:
    for filename in filenames:
    rasters.append(os.path.join(dirpath, filename))

# calculate mean
sum = CellStatistics(rasters, "MEAN")
sum.save(os.path.join(out_workspace, "npp_sum.tif")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sat, 11 Dec 2021 16:36:05 GMT

Thank you. I have updated my suggested by you, problem still persist, i am not understanding how to iterate through each folder, take all series of 001_NPP from all folder and calculate average oh them and save it as 001_NPP_MEAN, below is my updated code and along with code i have attached reference raster also which i am using for my raster calculation,

import arcpy
arcpy.CheckOutExtension("spatial")
from arcpy.sa import *

# generate list of rasters
workspace = r"D:\TEST"
out_workplace = "D:\TEST\MEAN_NPP"

rasters = []
walk = arcpy.da.Walk(workspace, datatype="RasterDataset", type="TIF")
for dirpath, dirnames, filenames in walk:
    for filename in filenames:
    rasters.append(os.path.join(dirpath, filename))

# calculate mean
sum = CellStatistics(rasters, "MEAN")
sum.save(os.path.join(out_workspace, "npp_sum.tif")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

DanPatterson_Retired — Sat, 08 Oct 2016 05:54:24 GMT

your code is only minimally updated from what Neil provided and it was untested.

What was the error message?

issues to get you started...

Line 13 is not indented properly
Walk—Help | ArcGIS for Desktop you need to check the correct format
datatype and type in Walk are incorrect
you shouldn't use from X import *

You need to provide some input to the solution and to do that providing commentary as to what went wrong and where would go a long way

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sat, 11 Dec 2021 16:36:08 GMT

Dan . thanks , you are right. I didn't post the error massage. Line no 13 i am getting the Indentation Error massage. I tried to fix it but not solved yet. error massage below

File "<module1>", line 13
    rasters.append(os.path.join(dirpath, filename))
          ^
IndentationError: expected an indented block
‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sat, 08 Oct 2016 08:36:13 GMT

xander_bakker‌, We need your cooperation to solve above problem. Kindly cooperate with us.

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

DanPatterson_Retired — Sat, 08 Oct 2016 09:13:28 GMT

Ok.... add four (4) more spaces to the beginning of line 13 so it is indented properly

Here is the python style guide recommendations PEP 8 -- Style Guide for Python Code | Python.org

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sat, 11 Dec 2021 16:36:11 GMT

Dan , thanks . I have solved the indentation error, and i updated my code but still i could not solve the problem, updated code below

error masage

ile "<module1>", line 18
    sum.save(os.path.join(out_workspace, 'r{0}_MEAN_NPP.TIF'.format(ras_num)
                                                                           ^
SyntaxError: invalid syntax‍‍‍‍

Updated Code

import arcpy
arcpy.CheckOutExtension("spatial")
from arcpy.sa import *

# generate list of rasters
workspace = r"D:\TEST"
out_workplace = "D:\TEST\MEAN_NPP"

for ras_num in [str(i).zfill(3) for i in range(1, 12)]:
 rasters = []
walk = arcpy.da.Walk(workspace, datatype="RasterDataset", type="TIF")
for dirpath, dirnames, filenames in walk:
    for filename in filenames:
     rasters.append(os.path.join(dirpath, filename))

# calculate mean
sum = CellStatistics(rasters, "MEAN")
sum.save(os.path.join(out_workspace, 'r{0}_MEAN_NPP.TIF'.format(ras_num)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

RebeccaStrauch__GISP — Sat, 08 Oct 2016 12:59:09 GMT

Change line 7 to

out_workplace = r"D:\TEST\MEAN_NPP"

You missed the "r" in that path.

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

DanPatterson_Retired — Sat, 11 Dec 2021 16:36:13 GMT

Here is how you test things... when you get an error, narrow down the code and test it, like in the following example to ensure that you output file names are correct..

>>> import os
>>> out_workplace = 'D:/Test/MEAN_NPP'  # note I am using os independent path specs
>>> num = 1   # obviously if you had this in a loop 'num' would be changing
>>> p = 'r{0}_MEAN_NPP.TIF'.format(num)   # this is the path I am trying to make
>>> x = os.path.join(out_workplace, p)  # joining them together
>>> x
'D:/Test/MEAN_NPP/r1_MEAN_NPP.TIF'      # this is what we get
‍‍‍‍‍‍‍

Now... is that right?

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

XanderBakker — Sat, 08 Oct 2016 15:19:28 GMT

In addition, line 18 is missing 2 closing brackets "))" at the end of the line. Perhaps you didn't copy the entire line.

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

XanderBakker — Sat, 11 Dec 2021 16:36:16 GMT

I'm looking at the code and trying to figure what you want to do exactly, since the indentation is not correct.

First of all I assume you want to perform this calculation for each month (001, .. , 012). To do so, you will need increase the range to (1, 13) (see line 9 below).

I also assume that you do not want to have the same output results for each month and that the list of rasters should be filtered by the month value. Therefore I included line 14.

Please be aware that in you original code you create the raster r001_MEAN_NPP.TIF and since that raster is stored in a subfolder of the input workspace, it will be included in the result for the second month. I don't think that is what you want. Also remember your closing brackets on line 19 (there were two missing in your code)

import arcpy
arcpy.CheckOutExtension("spatial")
from arcpy.sa import *

# generate list of rasters
workspace = r"D:\TEST"
out_workplace = "D:\TEST\MEAN_NPP"

for ras_num in [str(i).zfill(3) for i in range(1, 13)]:
    rasters = []
    walk = arcpy.da.Walk(workspace, datatype="RasterDataset", type="TIF")
    for dirpath, dirnames, filenames in walk:
        for filename in filenames:
            if ras_num in filename:
                rasters.append(os.path.join(dirpath, filename))

    # calculate mean
    ras_mean = CellStatistics(rasters, "MEAN")
    ras_mean.save(os.path.join(out_workspace, 'r{0}_MEAN_NPP.TIF'.format(ras_num)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

DanPatterson_Retired — Sat, 11 Dec 2021 16:36:18 GMT

Xander, thanks.... line 7, perhaps...

now before people get sloppy and forget to use 'raw' or unix based folder formatting... lets get those little 'r' s in there, especially if people get sloppy and/or use lower case, or upper case gets translated to lower case.... sooo. a 'case' in point...

>>> w = 'D:\Test\Folder'
>>> print(w)
D:\Test\Folder
>>>
>>> # now let's get sloppy
>>>
>>> w = 'D:\test\folder'
>>> print(w)
D:     est 
older‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

XanderBakker — Sat, 08 Oct 2016 17:09:26 GMT

Absolutely Dan, you are right!

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sat, 11 Dec 2021 16:36:21 GMT

Thank you very much all of you for your kind cooperation.

Xander Bakker I ran the above code , but i am getting error. I think not having problem with code. Might be we need more explanation what i am trying to do, please follow the below

Message     File Name     Line     Position     
Traceback                    
    <module>     <module1>     20          
    CellStatistics     C:\Program Files\ArcGIS\Desktop10.3\ArcPy\arcpy\sa\Functions.py     2951          
    swapper     C:\Program Files\ArcGIS\Desktop10.3\ArcPy\arcpy\sa\Utils.py     53          
    Wrapper     C:\Program Files\ArcGIS\Desktop10.3\ArcPy\arcpy\sa\Functions.py     2947          
RuntimeError: ERROR 999998: Unexpected Error.                    
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I am giving here what I am trying to do, I have three sub-folder (Name : NPP_2008, NPP_2009, NPP_2010),

NPP_2008      NPP_2009       NPP_2010       MEAN_NPP (output Folder)
r001_NPP.tif  r001_npp.tif   r001_NPP.tif   r001_MEAN_NPP.tif (mean of all r001_NPP)
r002_NPP.tif r002_npp.tif    r002_NPP.tif   r002_MEAN_NPP.tif (mean of all r002_NPP)
So on ..............

Every sub-folder hold monthly raster data for different year with the name of e.g r001_NPP, r002_NPP……..r012_NPP. So I want calculate mean of those raster which is r001_NPP series in every sub-folder, similarly program will loop through till r012_NPP series of all raster to calculate mean and save it another subfolder (Name : Mean_NPP) with output name 001_MEAN_NPP(which is mean of all r001_NPP), 002_MEAN NPP…..012_MEAN_NPP.

Here with i have attached reference raster which i am using for my raster calculation

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

curtvprice — Sun, 09 Oct 2016 05:56:09 GMT

Hey no fair. 😉

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

XanderBakker — Sat, 11 Dec 2021 16:36:24 GMT

I can't reproduce the error you mentioned. I did encounter some other errors that should be corrected like:

include import os at beginning of script and
change "out_workplace" into "out_workspace"

import arcpy
import os
arcpy.CheckOutExtension("spatial")

# generate list of rasters
##workspace = r"D:\TEST"
##out_workplace = r"D:\TEST\MEAN_NPP"
workspace = r"C:\GeoNet\MeanRasters\Case20161009"
out_workspace = r"C:\GeoNet\MeanRasters\Case20161009\MEAN_NPP"

for ras_num in [str(i).zfill(3) for i in range(1, 13)]:
    print ras_num
    rasters = []
    walk = arcpy.da.Walk(workspace, datatype="RasterDataset", type="TIF")
    for dirpath, dirnames, filenames in walk:
        for filename in filenames:
            if ras_num in filename:
                rasters.append(os.path.join(dirpath, filename))

    # calculate mean
    if len(rasters) > 0:
        print "Calculating mean of rasters: ", ', '.join(rasters)
        ras_mean = arcpy.sa.CellStatistics(rasters, "MEAN")
        ras_mean.save(os.path.join(out_workspace, 'r{0}_MEAN_NPP.TIF'.format(ras_num)))
    else:
        print "no rasters for ras_num", ras_num‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This created the mean raster for ras_num 001 to 005, see attached ZIP with resulting rasters

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Sun, 09 Oct 2016 19:19:16 GMT

xander_bakker‌. Thank you very much. Above Code running perfectly.

Regarding error issue which you got different error massage, because i corrected that line before running script.

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

XanderBakker — Sun, 09 Oct 2016 19:32:24 GMT

I'm glad it's working now...

Re: Perform Raster Calculation from Multiple Sub-folder Using ArcPy

ShouvikJha — Tue, 11 Oct 2016 08:48:03 GMT

Thank you. I am not getting your points. i am trying plot two variable of raster like this ...