Calculate monthly average AOD based on the Julian day wise data.

1567
13
Jump to solution
09-30-2018 07:47 PM
BIJOYGAYEN
New Contributor II
import datetime
start = datetime.datetime.now()
print 'start run: %s\n' % (start)
import arcpy, os
from arcpy import sa
from arcpy import env
import glob
arcpy.env.overwriteOutput = True
#outdir=r"G:\\ANFIS_FINAL\\AOD\\3KM_AOD\\"
di = r"G:\\ANFIS_FINAL\\AOD\\3KM_AOD\\201-18"
lstrasters = glob.glob(di + os.sep + "*Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth_Land.tif")
lstrasters.sort()


M01 =[]
M02 =[]
M03 =[]
M04 =[]
M05 =[]
M06 =[]
M07 =[]
M08 =[]
M09 =[]
M10 =[]
M11 =[]
M12 =[]

yer=2017
for ras in lstrasters:
    yrr=str(ras[49:(len(ras)-116)])
    year=int(yrr)
    #print year
    if year == yer:
     jdate=str(ras[53:(len(ras)-113)])
     day=int(jdate)
     #print day
     sa.Raster(ras)
     if day in range     (1,32):
        M01.append(ras)
     elif day in range  (32,60):
        M02.append(ras)
     elif day in range  (60,91):
        M03.append(ras)
     elif day in range (91,121):
        M04.append(ras)
     elif day in range(121,152):
        M05.append(ras)
     elif day in range(152,182):
        M06.append(ras)
     elif day in range(182,213):
        M07.append(ras)
     elif day in range(213,244):
        M08.append(ras)
     elif day in range(244,274):
        M09.append(ras)
     elif day in range(274,305):
        M10.append(ras)
     elif day in range(305,335):
        M11.append(ras)
     elif day in range (335,366):
        M12.append(ras)

     else:
        print "Jdate out of range"
    else:
        print ("No available Data in this year:"+ str(yer))
        #yer+=0001



print M01
print M02
print M03
print M04
print M05
print M06
print M07
print M08
print M09
print M10
print M11
print M12



def Null_sc_mean(input,output):
 SCM00_000=[]
 if len(input)!= 0:
             for ds in input:
               print ds
               dat=str(ds[59:(len(ds)-113)])
               setnull =arcpy.gp.SetNull_sa(ds,ds, "in_memory/dat", "\"Value\" < 0")
               ras=arcpy.Raster(setnull)
               scalefactor=float(0.001)
               scale=scalefactor*ras
               SCM00_000.append(scale)
               if len(SCM00_000)!=0:
                   arcpy.gp.CellStatistics_sa(SCM00_000,output,"MEAN", "DATA")

extra=1
m3 ="G:\\ANFIS_FINAL\\AOD\\3KM_AOD\\01_18\\"
for ls in M01,M02,M03,M04,M05,M06,M07,M08,M09,M10,M11,M12:
  out1= m3+'{0}.img'.format("Month"+ str(extra)+str(yer))
  mon1= Null_sc_mean(ls,out1)
  extra += 1
  if extra==13:
    yer+=0001




print arcpy.GetMessages()
print 'finished run: %s\n\n' % (datetime.datetime.now() - start)





I have Julian day wise AOD data 2017/06 to 2018/05. Every day(...2017152...) have three files(...Land1, Land2, Land3), here we'll take only the land2 data after that I want to calculate the monthly average value from month wise data and save it in raster format with month wise by the loop. Before going to the average function we have some additional calculation is that every reading raster file first removes or deleting specific value ( less than 0 to -9999) and multiplied with 0.001 value, and then going to the Average calculation function. 

When  I am running this code then there is some anomaly. like when I am processing the data specifying subsequent months of a year or years then it saves the output files haphazardly. For example, my desired years are 2017 and 2018. It picks all the file for the specified years, but it saves the output file for 2018 in 2017 file and along with its own output file.

kindly give me a resolution or any other alternative way, and also I want to set this code leap year also.

I have attached some sample data.

0 Kudos
13 Replies
BIJOYGAYEN
New Contributor II

I mean, I want to do another process that will be day wise system and save those every raster file by their year, month, date, name.

0 Kudos
RandyBurton
MVP Alum

Currently the grouping is by year then month.  Instead of grouping by month, you could grab the day and group by day, so you would be able to combine Jan 1, Feb 1, March 1, etc. into a file.  You would structure your dictionary with 1 to 31 for days of the month instead of 1 to 12 for months of the year.

If you just want to rename/copy files with a YMD in the name, you could just loop through your directory list and extract the date from the Julian day for your rename.  This wouldn't involve a dictionary.

This code still groups by month, but it may give you some ideas on how to do what you want.  It uses the tuple idea I mentioned previously and follows the flow of an earlier example. 

from datetime import datetime

def extractJulian(filename):
    jdate = filename.split(".")[1][1:] # get date portion, splitting on periods
    dt = datetime.strptime(jdate, '%Y%j').date()
    return(filename, dt.year, dt.month, dt.day, jdate[4:]) # return a tuple

def someFunction(aTupleList):
    for aTuple in aTupleList: # print each tuple in list of tuples
        print
        print "Old file: {}".format(aTuple[0]) # filename
        print "Newname_{}_{}_{}_{}.tif".format(aTuple[1],aTuple[2],aTuple[3],aTuple[4]) # Year, Month, Day, DayNumber
    
    return None


# lstrasters = glob.glob(di + os.sep + "*Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth_Land.tif")
lstrasters = [  # list from your directory search 
    'MOD04_3K.A20161.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A20162.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A2017152.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A2017153.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A2017254.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A2017255.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A2018356.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif',
    'MOD04_3K.A2018357.mosaic.061.2017271115004.pssgmcrpgs_000501268215.Corrected_Optical_Depth_Land_2-Corrected_Optical_Depth.Land.tif'
    ]  # these rasters are for months 6 and 9 in 2017, and 12 in 2018

rasters = { 2017: {1:[],2:[],3:[],4:[],5:[],6:[],7:[],8:[],9:[],10:[],11:[],12:[]},
            2018: {1:[],2:[],3:[],4:[],5:[],6:[],7:[],8:[],9:[],10:[],11:[],12:[]}
            } # starting dictionary

for r in lstrasters:
    ej = extractJulian(r)
    # ej[0] = filename (same as r), ej[1] = year, ej[2] = month, ej[3] = day
    if ej[1] in rasters.keys():
        rasters[ej[1]][ej[2]].append(ej)
    else:
        print "ERROR year {}: {}\n".format(ej[1], r)

print rasters  # take a look at the dictionary

for k1,v1 in rasters.iteritems():
    year =k1
    for k2,v2 in v1.iteritems():
        if len(v2):
            month =k2
            print "\n\nYear: {} Month: {}".format(year, month)
            rtn = someFunction(v2)  # function prints v2 (tuples in a list) and returns None‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
BIJOYGAYEN
New Contributor II

Code working well. Thank you so much, sir, for giving me the time to solve this problem.

0 Kudos
RandyBurton
MVP Alum

You're welcome.  Glad to help.

0 Kudos