topic Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D in Python Questions

Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

BennyIstanto — Sun, 12 Dec 2021 05:38:46 GMT

Hi, with limited knowledge on python and arcpy, I tried to modified script from former staff in my office with no success.

The existing script is:

Try to calculate number of consecutive days without rainfall (rainfall < 1mm) based on daily CHIRPS data with naming convention "chirps-v2.0.YYYY.MM.DD.tif"
The calculation required number of days without rain from the previous day. If this data not available, it will skipped to next data.
Let say, the data start from 1 Jan 1981, then the script will skip this date. And continue the process for the next day (2 Jan 1981) with following process:
- IF(chirps-v2.0.1981.01.01.tif <1, 1, 0) and save as cli_chirps_dslr_19810102.tif
- IF(chirps-v2.0.1981.01.02.tif <1, cli_chirps_dslr_19810102.tif + 1, cli_chirps_dslr_19810102.tif = 0) and save as cli_chirps_dslr_19810103.tif
- and so on.

And now I would like to use the script and modified some part to calculate similar approach with rainfall > 50mm, but using Dekad CHIRPS data with naming convention

chirps-v2.0.1981.01.1.tif

chirps-v2.0.1981.01.2.tif

chirps-v2.0.1981.01.3.tif

chirps-v2.0.1981.02.1.tif

chirps-v2.0.1981.02.2.tif

chirps-v2.0.2020.04.3.tif

I got an error because the script try to read based on Date information, after finished process chirps-v2.0.1981.01.03.tif the scipt will continue to find chirps-v2.0.1981.01.04.tif and couldn't find the data then error.

Any reference or example on how to make it work?

Below is the error, script and few example of the data (subset from area with smaller size).

....
found chirps-v2.0.2001.01.2.tif in the dekad rainfall folder
found chirps-v2.0.2016.07.2.tif in the dekad rainfall folder
found chirps-v2.0.1994.11.2.tif in the dekad rainfall folder
found chirps-v2.0.2000.01.2.tif in the dekad rainfall folder
found chirps-v2.0.2017.07.2.tif in the dekad rainfall folder
found chirps-v2.0.1995.11.2.tif in the dekad rainfall folder
execute first rainy data from date 1981011
creating precipgt50 data 1981-01-02 from dekad rainfall data from 1981-01-01
1981-01-02
Processing chirps_precipgt50_1981012.tif
file chirps_precipgt50_1981012.tif is created
start reading existing PRECIPgt50 Dataset....
looking for file with naming chirps_precipgt50_YYYYMMD
found chirps_precipgt50_1981012.tif in the PRECIPgt50 folder
PRECIPgt50 Data found. Looking for latest PRECIPgt50 Data...
next dekad data is available...
start processing next PRECIPgt50...
Processing PRECIPgt50 from Z:\Temp\DryWetSeason\Output\chirps_precipgt50_1981012.tif and Z:\Temp\DryWetSeason\Dekad\chirps-v2.0.1981.01.2.tif
PRECIPgt50 File chirps_precipgt50_1981013.tif is created
next dekad data is available...
start processing next PRECIPgt50...
Processing PRECIPgt50 from Z:\Temp\DryWetSeason\Output\chirps_precipgt50_1981013.tif and Z:\Temp\DryWetSeason\Dekad\chirps-v2.0.1981.01.3.tif
PRECIPgt50 File chirps_precipgt50_1981014.tif is created
next dekad data is not available. Exit...
next dekad data is not available. Exit...
next dekad data is not available. Exit...
next dekad data is not available. Exit...
next dekad data is not available. Exit...
next dekad data is not available. Exit...
next dekad data is available...
start processing next PRECIPgt50...
Processing PRECIPgt50 from Z:\Temp\DryWetSeason\Output\chirps_precipgt50_19810110.tif and Z:\Temp\DryWetSeason\Dekad\chirps-v2.0.1981.01.1.tif
Traceback (most recent call last):
File "PRECIPgt50.py", line 118, in <module>
list = create_PRECIPgt50('Z:\\Temp\\DryWetSeason\\Output','Z:\\Temp\\DryWetSeason\\Dekad', 50)
File "PRECIPgt50.py", line 107, in create_PRECIPgt50
execute_PRECIPgt50(last_date, _tiffolder, _PRECIPgt50_folder, threshold)
File "PRECIPgt50.py", line 50, in execute_PRECIPgt50
outPRECIPgt50Con = Con(Raster(next_dekaddata) < int(threshold), Raster(last_precipgt50file)+1, 0)
RuntimeError: ERROR 000732: Input Raster: Dataset Z:\Temp\DryWetSeason\Output\chirps_precipgt50_19810110.tif does not exist or is not supported
Z:\Temp\DryWetSeason>

import os
import arcpy
from arcpy.sa import *
from datetime import date, timedelta

# to execute first PRECIPgt50 data
def execute_first_PRECIPgt50(_tiffolder, _PRECIPgt50Folder, threshold):
    sr = arcpy.SpatialReference(4326)
    print("looking at the first dekad rainfall data in tif folder...")
    dekad_list = create_dekad_List(_tiffolder)
    first_date = min(dekad_list)
    print("execute first rainy data from date "+first_date)
    first_data_name = 'chirps-v2.0.{0}.{1}.{2}.tif'.format(first_date[0:4], first_date[4:6], first_date[6:7])
    first_dekad_data = os.path.join(_tiffolder, first_data_name)
    dekad_Date = date(int(first_date[0:4]), int(first_date[4:6]), int(first_date[6:7]))
    precipgt50_date = dekad_Date + timedelta(days=1)
    print("creating precipgt50 data "+str(precipgt50_date)+ " from dekad rainfall data from "+str(dekad_Date))
    PRECIPgt50Year = str(precipgt50_date.year)
    PRECIPgt50month = str(precipgt50_date.month)
    PRECIPgt50day = str(precipgt50_date.day)
    print(str(precipgt50_date))
    PRECIPgt50Filename = 'chirps_precipgt50_{0}{1}{2}.tif'.format(PRECIPgt50Year.zfill(4), PRECIPgt50month.zfill(2), PRECIPgt50day.zfill(1))
    print("Processing "+PRECIPgt50Filename)
    arcpy.CheckOutExtension("spatial")
    outCon = Con(Raster(first_dekad_data) < int(threshold), 1, 0)
    outCon.save(os.path.join(_PRECIPgt50Folder, PRECIPgt50Filename))
    arcpy.DefineProjection_management(os.path.join(_PRECIPgt50Folder, PRECIPgt50Filename), sr)
    arcpy.CheckInExtension("spatial")
    print("file " + PRECIPgt50Filename + " is created")


# to execute next PRECIPgt50 data
def execute_PRECIPgt50(_lastdate, _tiffolder, _PRECIPgt50_folder, threshold):
    sr = arcpy.SpatialReference(4326)
    date_formatted = date(int(_lastdate[0:4]), int(_lastdate[4:6]), int(_lastdate[6:7]))
    last_precipgt50name = 'chirps_precipgt50_{0}'.format(_lastdate)
    last_precipgt50file = os.path.join(_PRECIPgt50_folder, last_precipgt50name)
    next_dekadname = 'chirps-v2.0.{0}.{1}.{2}.tif'.format(_lastdate[0:4], _lastdate[4:6], _lastdate[6:7])
    next_dekaddata = os.path.join(_tiffolder, next_dekadname)
    if arcpy.Exists(next_dekaddata):
        print("next dekad data is available...")
        print("start processing next PRECIPgt50...")
        new_precipgt50_date = date_formatted + timedelta(days=1)
        PRECIPgt50Year1 = str(new_precipgt50_date.year)
        PRECIPgt50month1 = str(new_precipgt50_date.month)
        PRECIPgt50day1 = str(new_precipgt50_date.day)
        new_precipgt50_name = 'chirps_precipgt50_{0}{1}{2}.tif'.format(PRECIPgt50Year1.zfill(4), PRECIPgt50month1.zfill(2), PRECIPgt50day1.zfill(1))
        print("Processing PRECIPgt50 from "+last_precipgt50file+" and "+next_dekaddata)
        arcpy.CheckOutExtension("spatial")
        outPRECIPgt50Con = Con(Raster(next_dekaddata) < int(threshold), Raster(last_precipgt50file)+1, 0)
        outPRECIPgt50Con.save(os.path.join(_PRECIPgt50_folder, new_precipgt50_name))
        arcpy.DefineProjection_management(os.path.join(_PRECIPgt50_folder, new_precipgt50_name), sr)
        arcpy.CheckInExtension("spatial")
        print("PRECIPgt50 File "+new_precipgt50_name+" is created")
    else:
        print("next dekad data is not available. Exit...")

# to check if there is PRECIPgt50 data in output folder
def create_PRECIPgt50_List(_PRECIPgt50_folder):
    print("start reading existing PRECIPgt50 Dataset....")
    print("looking for file with naming chirps_precipgt50_YYYYMMD")
    PRECIPgt50_Date_List=[]
    for PRECIPgt50_Data in os.listdir(_PRECIPgt50_folder):
        if PRECIPgt50_Data.endswith(".tif") or PRECIPgt50_Data.endswith(".tiff"):
            print("found " + PRECIPgt50_Data + " in the PRECIPgt50 folder")
            parse_String = PRECIPgt50_Data.split('_')
            PRECIPgt50_Data_Date = parse_String[2]
            PRECIPgt50_Date_List.append(PRECIPgt50_Data_Date)
    return PRECIPgt50_Date_List

# to check input data
def create_dekad_List(_tif_folder):
    print("start reading list of dekad rainfall data....")
    print("looking for file with naming chirps-v2.0.YYYY.MM.DD")
    Dekad_Date_List=[]
    for Dekad_Data in os.listdir(_tif_folder):
        if Dekad_Data.endswith(".tif") or Dekad_Data.endswith(".tiff"):
            print("found " + Dekad_Data+ " in the dekad rainfall folder")
            parse_String = Dekad_Data.split('.')
            Dekad_Data_Date = parse_String[2]+parse_String[3]+parse_String[4]
            Dekad_Date_List.append(Dekad_Data_Date)
    return Dekad_Date_List

# to run the script
def create_PRECIPgt50(_PRECIPgt50_folder, _tiffolder, threshold):

    PRECIPgt50_Date_List = create_PRECIPgt50_List(_PRECIPgt50_folder)
    Dekad_list = create_dekad_List(_tiffolder)
    # if there is no PRECIPgt50 data, creating new PRECIPgt50 data
    if len(PRECIPgt50_Date_List)==0:
        print("No PRECIPgt50 Data found...")
        print("creating first PRECIPgt50 data...")
        execute_first_PRECIPgt50(_tiffolder, _PRECIPgt50_folder, threshold)
        PRECIPgt50_Date_List = create_PRECIPgt50_List(_PRECIPgt50_folder)
    # if there is PRECIPgt50 data
    print("PRECIPgt50 Data found. Looking for latest PRECIPgt50 Data...")
    #Check last PRECIPgt50 available
    last_date = max(PRECIPgt50_Date_List)

    #Check last dekad data availabke
    max_dekad_date = max(Dekad_list)
    last_PRECIPgt50_date = date(int(last_date[0:4]), int(last_date[4:6]), int(last_date[6:7]))
    last_dekad_date = date(int(max_dekad_date[0:4]), int(max_dekad_date[4:6]), int(max_dekad_date[6:7]))
    # process PRECIPgt50 to every dekad data available after last PRECIPgt50 data
    while last_dekad_date + timedelta(days=1) > last_PRECIPgt50_date:

        execute_PRECIPgt50(last_date, _tiffolder, _PRECIPgt50_folder, threshold)

        last_PRECIPgt50_date=last_PRECIPgt50_date+timedelta(days=1)
        PRECIPgt50Year2 = str(last_PRECIPgt50_date.year)
        PRECIPgt50month2 = str(last_PRECIPgt50_date.month)
        PRECIPgt50day2 = str(last_PRECIPgt50_date.day)
        last_date='{0}{1}{2}.tif'.format(PRECIPgt50Year2.zfill(4), PRECIPgt50month2.zfill(2), PRECIPgt50day2.zfill(1))

    print("All PRECIPgt50 data is available")

# run the function (output folder, input folder, threshold)
list = create_PRECIPgt50('Z:\\Temp\\DryWetSeason\\Output','Z:\\Temp\\DryWetSeason\\Dekad', 50)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

DanPatterson — Tue, 02 Jun 2020 23:42:00 GMT

The exact error message is needed to track down the problem

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

BennyIstanto — Tue, 02 Jun 2020 23:51:21 GMT

Hi Dan, thanks. I have updated my question and include a screenshot from cmd window.

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

DanPatterson — Wed, 03 Jun 2020 00:04:51 GMT

outPRECIPgt50Con = Con(Raster(next_dekaddata) < int(threshold), Raster(last_precipgt50file)+1, 0)

Could you put a print statement before this line to see what the values are prior to line 50?

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

BennyIstanto — Wed, 03 Jun 2020 00:13:45 GMT

There is print statement line 48. Or do you mean another print statement?

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

DanPatterson — Wed, 03 Jun 2020 00:40:23 GMT

The print statement is cutoff in the screen grab

Anyways

RuntimeError: ERROR 000732: Input Raster: Dataset Z:\Temp\DryWetSeason\Output\chirps_precipgt50_19810110.tif does not exist or is not supported

Z:\Temp\DryWetSeason>

Error: 000732: Dataset does not exist or is not supported

and its causes.

The root is whether the input raster exists. You should confirm it ... arcpy.Exists ... prior to continuing

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

BennyIstanto — Wed, 03 Jun 2020 00:45:01 GMT

Yes correct, because I am not using Daily data, but Dekad or 10 days data with naming convention

chirps-v2.0.1981.01.1.tif

chirps-v2.0.1981.01.2.tif

chirps-v2.0.1981.01.3.tif

chirps-v2.0.1981.02.1.tif

chirps-v2.0.1981.02.2.tif

chirps-v2.0.2020.04.3.tif

Dekad 1 in chirps-v2.0.1981.01.1.tif means rainfall accumulation from Date 1 to 10, and Dekad 2 from 11 to 20, Dekad 3 from 21 to 28/29/30/31

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

DanPatterson — Wed, 03 Jun 2020 01:06:19 GMT

punctuation in filenames ( . and -) can cause issues. Did the previous useage of the script follow that naming convention?

The root error exists on that line, so you will have to check what goes into the line to see if it exists and if it is a tif, it should be going to a folder with the *.tif extension on the next line.

outPRECIPgt50Con = Con(Raster(next_dekaddata) < int(threshold), Raster(last_precipgt50file)+1, 0)

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

BennyIstanto — Wed, 03 Jun 2020 01:45:10 GMT

Yes, the previous script to calculate consecutive dry days works well using Daily data with similar naming convention

chirps-v2.0.1981.01.01.tif

chirps-v2.0.1981.01.02.tif

chirps-v2.0.1981.01.03.tif

chirps-v2.0.1981.01.04.tif

chirps-v2.0.2020.05.31.tif

punctuation in filenames already eliminated in line 80. I think the problem because the script read input file with date YYYYMMDD information, and the input is just YYYYMMD => the D is not Date but Dekad with value 1, 2 and 3.

I dont know how to modified that

Re: Looping on data process with Dekad (D = 1 or 2 or 3) date information YYYY.MM.D

CarlosAlbertoDuarteCarranza — Sat, 06 Jan 2024 01:04:09 GMT

Dear Benny,

I´m also new to Python and ArcPy. I´m struggling to do exactly what you describe as the "existing script" in this post, to calculate the number of days without rainfall based on CHIRPS daily data. Is it possible for you to share the code of your "existing script". That would helpl me a lot to get to understand how to implement what I need to do.

Thanks in advance!

Carlos