Convert multiple csv (specific date range) to shp using ArcPy

[ShouvikJha]

I have multiples CSV files. Every CSV file contains 365 days of different weather parameters for different location. I want to take specific month from every csv column and export it to point shp file using ArcPy. Can you help me to do it using ArcPy. In csv date format is (MM DD YY)

Below i have attached my python code. This code giving me error while i running date specific date range. Below i have attached my csv file also.

import arcpy,os

... shpworkspace = r"D:\TRY-Python"

... arcpy.env.workspace = shpworkspace

... arcpy.env.overwriteOutput = True

...

... csvlist = arcpy.ListFiles("*.csv")

...

... for csvfile in csvlist:

...     print csvfile

...     outlayer = "CSVEventLayer"

...     spatialreference = "GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119522E-09;0.001;0.001;IsHighPrecision"

...     arcpy.MakeXYEventLayer_management(csvfile,"Longitude","Latitude",outlayer,spatialreference,"#")

...     arcpy.SelectLayerByAttribute_management(outlayer, Date 1/2/2013-1/31/2013, "#")

...     shpfile = os.path.splitext(csvfile)[0].replace('-','_')       

...     arcpy.CopyFeatures_management(outlayer,shpfile)

...

...     del outlayer

[ShouvikJha]

Thank you Jake Skinner‌. Using PyScripter i  ran the code throughout the CSVs. Thank you very much for your guidance and cooperation . 

[ShouvikJha]

Jake Skinner‌. I have one more query on above program, I would like to add some more step with above program.

Step 1: Above program performing well for converting all CSVs to Point shape file.

 After conversion of each CSVs file to Point shape file, I would like to Merge the different month wise all point shapefile (Belongs to same month) into one datebase and save it in existing  months wise different folder.

(From this step we will get the month wise single database contain point vector e.g. 001_MERGE, 002_MERGE......012_MERGE)

Step: 2 Using those marge points from different month ( got it from Step: 1 e.g. 001_MERGE, 002_MERGE......012_MERGE) want to perform IDW interpolation for selected parameters from Merge point database (e.g. Max_Temper, Min_Temper,Precipitation, Relative humidity, Wind Speed, Solar etc. )  for creating raster map based on defined mask file (Vector File), cell Size (defined raster), Geo-statistical analysis: Coincident Points (Mean), SEARCH RADIUS : (FIXED Distance : 0.118646 ; MINIMUM NO OF POINTS : 2) . output raster map pass it to as on different month folder (Output raster name for January e.g.  001_Max_Temper, 001_Min_Temper, 001_Precipitation, 001_Wind, 001_RH, 001_Radiation: FEBRUARY e.g.  002_Max_Temper, 002_Min_Temper, 002_Precipitation, 002_Wind, 002_RH, 002_Radiation......till December ). 

Step : 3 Whatever monthly raster only for Solar Radiation (e.g 001_Radiation) we got it from Step 2 , We want perform some calculation, (e.g. 001_Radiation  * 30) * 0.5 and save it in different month folder, output name ( e.g 001_sr, 002_sr.....012_sr).

Below  i have attached the defined mask file and cell size raster.

[ShouvikJha]

Jake Skinner‌. Above mentioned query i have written the code upto Merging multiple feature for each month Class. Sorry to inconvenience. 

For next step i have put a new question to this community along with  my code. Kindly guide me to do the task. Thank you very much  

[ShouvikJha]

Hi Jake, Earlier Whatever CSV we converted to point feature class and later we merged them into  one feature class month wise (Mereged_001, (Mereged_002...(Mereged_012 ), Now i want to perform Universal Kriging (LINEARDRIFT). I have written a code for that but its not running , Below is my code along with reference data. Any suggestion  would be great. Thank you  

import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.env.overwriteOutput = True

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")

env.workspace = r"D:\SWAT-WEATHER-DATA"

# Set local variables
inPointFeatures = "D:\SWAT-WEATHER-DATA\SELECTED-EX-MERGE.shp"

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]

#Kriging Veriable
cellSize = 0.002298707671
lagSize = 0.5780481172534
majorRange = 6
partialSill = 3.304292110
nugget = 0.002701348
kRadius = RadiusFixed(20000, 1)

#Mask region of interest
mask="D:\Gujarta Shape file\GUJARATSTATE.shp"

# Execute Kriging
for zField in zFields:
    kModelUniversalObj = KrigingModelUniversal("LINEARDRIFT", lagSize, majorRange, partialSill, nugget)

    OutKriging = Kriging(inPointFeatures, zField, kModelUniversalObj, 20000, kRadius)
    # Execute Mask
    #IDWMASk = ExtractByMask(outIDW, mask)
    KrigMask = ExtractByMask(OutKriging, mask)

    if not zField == "Solar":
    # Save output, except Solar Radiation raster
     KrigMask.save("012_{}.tif".format(zField))

    else:
     #Only for Solar Radiation raster
     PAR = (KrigMask * 30) * 0.5
     #Save output, Only for Solar Radiation raster
     PAR.save("012_{}.tif".format(zField))

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[JakeSkinner]

Shouvik,

You have your cell size in the Kriging function set to 20,000.  I updated the cellSize variable to a larger cell size (0.05) so that it would not take as long to process, and then updated this function with this variable.  Also, when specifying paths using a single backslash ("D:\data\etc"), you will want to make sure you have 'r' in front of the path (r"D:\data\etc").  Try the following:

import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.env.overwriteOutput = True

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")

env.workspace = r"D:\Temp\Python\UserData\Shape_File"

# Set local variables
inPointFeatures = r"D:\Temp\Python\UserData\Shape_File\Merged_004.shp"

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]

#Kriging Veriable
cellSize = 0.05
lagSize = 0.5780481172534
majorRange = 6
partialSill = 3.304292110
nugget = 0.002701348
kRadius = RadiusFixed(20000, 1)

#Mask region of interest
mask = r"D:\Temp\Python\UserData\Shape_File\Boundary.shp"

# Execute Kriging
for zField in zFields:
    kModelUniversalObj = KrigingModelUniversal("LINEARDRIFT", lagSize, majorRange, partialSill, nugget)

    OutKriging = Kriging(inPointFeatures, zField, kModelUniversalObj, cellSize, kRadius)

    # Execute Mask
    #IDWMASk = ExtractByMask(outIDW, mask)
    KrigMask = ExtractByMask(OutKriging, mask)

    if not zField == "Solar":
        # Save output, except Solar Radiation raster
         KrigMask.save("012_{}.tif".format(zField))

    else:
         #Only for Solar Radiation raster
         PAR = (KrigMask * 30) * 0.5
         #Save output, Only for Solar Radiation raster
         PAR.save("012_{}.tif".format(zField))

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[ShouvikJha]

jskinner-esristaff‌. Thank you very much. Code working perfectly as you changed the cell value. But output from Kriging seems to be not accurate due to some  limitation with kriging. I keep continue my research on this subject. 

[ShouvikJha]

Hi Jake,I am trying to modify above code, as keep it IDW function same, But instead of giving path for every month shape file , I just trying to use loop function that program can automatic take shape file from different month folder (Name of point shape file in different month folder : JANUARY: Merged_001, FEBRUARY: Merged_002.....DECEMBER: Merged_012 ) and save as it same folder along with month number and Z field name (e.g for JANUARY : 001_Max_Temper, 001_Min_Temper, 001_Precipitat, FEBRUARY : 002_Max_Temper, 002_Min_Temper, 002_Precipitatetc.) . Below is my working code. But its showing error. 

import arcpy, os, calendar
arcpy.CheckOutExtension("Spatial")
topWorkspace = r'D:\NPP_GUJARAT\WEATHER_DATA\2011'
arcpy.env.workspace = topWorkspace

 # Get dict of months and month-number (i.e. January = 001, March = 003 etc.)
months = {calendar.month_name[i].upper(): str(i).zfill(3) for i in range(1, 13)} # Get dict of months and month number (i.e. January = 001, March = 003 etc.)

 # Step through list of all folders
for folderPath in arcpy.ListWorkspaces():
    baseName = os.path.basename(folderPath).upper()
    if baseName in months: # Test that subfolder is a month name
        monthNumber = months[baseName] # Get month-number for use in output filename
        arcpy.env.workspace = folderPath

        #Loop every month folder ta take shape file from each month (e.g. Merged_001 (JANUARY), Merged_002 (FEBRUARY)....Merged_012 (DECEMBER))
        shapeFiles = arcpy.ListFeatureClasses('*Merged_{}.shp')

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]
# IDW variable
cellSize = 0.002298707671
power = 2
searchRadius = RadiusVariable(10, 150000)

 #Mask region of interest
mask="D:\Gujarta Shape file\GUJARATSTATE.shp"

for shape_num in shapeFiles:
 Zfield_num = arcpy.ListFeatureClasses(os.path.join(folderPath, shape_num))

 # Execute IDW
 for zField in zFields:
  outIDW = Idw(shapeFiles, zField, cellSize, power, searchRadius)

 # Execute Mask
 IDWMASk = ExtractByMask(outIDW, mask)

 if not zField == "Solar":
        # Save output, except Solar Radiation raster (e.g for JANUARY:001_Max_Temper, FEBRUARY:002_Max_Temper)
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField_num))
        IDWMASk.save(out_name_IDW)
 else:
        #Only for Solar Radiation raster
        PAR = (IDWMASk * 30) * 0.5
        #Save output, Only for Solar Radiation raster
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField))
        PAR.save(out_name_IDW)

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[ShouvikJha]

Jake Skinner‌, I am trying to solve the above script problem, but i am getting continue error. Error displaying

IndentationError
 unexpected indent (IDW_PAR_Year_WISE.py, line 10) ‍‍

How to resolve the issue. Thanks 

[JakeSkinner]

This error indicates that you have an indent error.  Either the line is indented when it shouldn't be, or vice versa.  I would recommend using an IDE such as PyScripter.  Within this IDE you can click on the 'Run' menu and choose the option 'Check Syntax'.  This will report any syntax errors.

[ShouvikJha]

Hi Jake, Thank you . I am running the code on Pyscripter. I have updated my code, there is nothing error with my code, but  i am not getting any output raster from IDW, below is my updated code. Whats the possible error 

import arcpy, os, calendar
from arcpy import env
from arcpy.sa import *
arcpy.CheckOutExtension("Spatial")
topWorkspace = r'D:\NPP_GUJARAT\WEATHER_DATA1\2011'
arcpy.env.workspace = topWorkspace

 # Get dict of months and month-number (i.e. January = 001, March = 003 etc.)
months = {calendar.month_name[i].upper(): str(i).zfill(3) for i in range(1, 13)} # Get dict of months and month number (i.e. January = 001, March = 003 etc.)

 # Step through list of all folders
for folderPath in arcpy.ListWorkspaces():
       baseName = os.path.basename(folderPath).upper()
       if baseName in months: # Test that subfolder is a month name
        monthNumber = months[baseName] # Get month-number for use in output filename
       arcpy.env.workspace = folderPath

        #Loop every month folder ta take shape file from each month (e.g. Merged_001 (JANUARY), Merged_002 (FEBRUARY)....Merged_012 (DECEMBER))
       shapeFiles = arcpy.ListFeatureClasses('Merged_{}*.shp')

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]
# IDW variable
cellSize = 0.002298707671
power = 2
searchRadius = RadiusVariable(10, 150000)

 #Mask region of interest
mask="D:\Gujarta Shape file\GUJARATSTATE.shp"

for shape_num in shapeFiles:
 Zfield_num = arcpy.ListFeatureClasses(os.path.join(folderPath, shape_num))

 # Execute IDW
 for zField in zFields:
  outIDW = Idw(shapeFiles, zField, cellSize, power, searchRadius)

 # Execute Mask
 IDWMASk = ExtractByMask(outIDW, mask)

 if not zField == "Solar":
        # Save output, except Solar Radiation raster (e.g for JANUARY:001_Max_Temper, FEBRUARY:002_Max_Temper)
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField_num))
        IDWMASk.save(out_name_IDW)
 else:
        #Only for Solar Radiation raster
        PAR = (IDWMASk * 30) * 0.5
        #Save output, Only for Solar Radiation raster
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField))
        PAR.save(out_name_IDW)

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍