topic Re: Convert multiple csv (specific date range) to shp using ArcPy in Python Questions

Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Mon, 01 Aug 2016 16:40:12 GMT

I have multiples CSV files. Every CSV file contains 365 days of different weather parameters for different location. I want to take specific month from every csv column and export it to point shp file using ArcPy. Can you help me to do it using ArcPy. In csv date format is (MM DD YY)

Below i have attached my python code. This code giving me error while i running date specific date range. Below i have attached my csv file also.

import arcpy,os

... shpworkspace = r"D:\TRY-Python"

... arcpy.env.workspace = shpworkspace

... arcpy.env.overwriteOutput = True

...

... csvlist = arcpy.ListFiles("*.csv")

...

... for csvfile in csvlist:

... print csvfile

... outlayer = "CSVEventLayer"

... spatialreference = "GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119522E-09;0.001;0.001;IsHighPrecision"

... arcpy.MakeXYEventLayer_management(csvfile,"Longitude","Latitude",outlayer,spatialreference,"#")

... arcpy.SelectLayerByAttribute_management(outlayer, Date 1/2/2013-1/31/2013, "#")

... shpfile = os.path.splitext(csvfile)[0].replace('-','_')

... arcpy.CopyFeatures_management(outlayer,shpfile)

...

... del outlayer

Re: Convert multiple csv (specific date range) to shp using ArcPy

JakeSkinner — Sat, 11 Dec 2021 17:06:31 GMT

Hi Shouvik,

You will need an OBJECTID field in order to perform the Select Layer By Attribute. I updated the code to import the CSV file into a scratch geodatabase since the CSV does not have a unique ID field. Doing so will add the OBJECTID field. Also, I updated the syntax to query the date. See below:

import arcpy,os

shpworkspace = r"D:\TRY-Python"
arcpy.env.workspace = shpworkspace
arcpy.env.overwriteOutput = True
scratchGDB = arcpy.env.scratchGDB

csvlist = arcpy.ListFiles("*.csv")


for csvfile in csvlist:
    print csvfile
    arcpy.TableToTable_conversion(csvfile, scratchGDB, "CSV")
    outLayer = "CSVEventLayer"
    spatialreference = arcpy.SpatialReference(4326)
    arcpy.MakeXYEventLayer_management(os.path.join(scratchGDB, "CSV"),"Longitude","Latitude",outLayer,spatialreference,"#")
    arcpy.SelectLayerByAttribute_management(outLayer, "NEW_SELECTION", "Date > date '2013-01-02' AND Date < date '2013-01-31'")
    shpfile = os.path.splitext(csvfile)[0].replace('-','_')
    arcpy.CopyFeatures_management(outLayer,shpfile)

del outLayer

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Mon, 01 Aug 2016 18:18:06 GMT

Thank you for your suggestion. I have run the code. There is still error

Runtime error

Traceback (most recent call last):

File "<string>", line 11, in <module>

TypeError: 'NoneType' object is not iterable

Re: Convert multiple csv (specific date range) to shp using ArcPy

JakeSkinner — Mon, 01 Aug 2016 18:21:31 GMT

This error is usually an indication that there are no CSV files in the following directory: "D:\TRY-Python". I would double-check to make sure there is a CSV file located there.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Mon, 01 Aug 2016 18:39:23 GMT

@Jake Skinner Thank you very much sir. Your code is working fine. Once again thank you very much.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Tue, 16 Aug 2016 17:55:14 GMT

jskinner-esristaff Can we modify above code for extracting different months separately (e.g Above code generating single feature class based on input date range but instead of generate single feature class i want generate month wise different feature class and save it into different month folder created by python script using same above input data). I have to analyze monthly profile of weather data therefor i want to create separate feature class for each month.

Re: Convert multiple csv (specific date range) to shp using ArcPy

JakeSkinner — Sat, 11 Dec 2021 17:06:34 GMT

Try the following:

import arcpy,os

shpworkspace = r"D:\temp\python\userdata"
arcpy.env.workspace = shpworkspace
arcpy.env.overwriteOutput = True
scratchGDB = arcpy.env.scratchGDB

csvlist = arcpy.ListFiles("*.csv")

# Create dictionary of months
dict = {"JANUARY": "01", "FEBRUARY": "02", "MARCH": "03", "APRIL": "04", "MAY": "05", "JUNE": "06", "JULY": "07", "AUGUST": "08", "SEPTEMBER": "09", "OCTOBER": "10", "NOVEMBER": "11", "DECEMBER": "12"}

for csvfile in csvlist:
    print csvfile
    arcpy.TableToTable_conversion(csvfile, scratchGDB, "CSV")
    outLayer = "CSVEventLayer"
    spatialreference = arcpy.SpatialReference(4326)
    # Create XY Event Layer
    arcpy.MakeXYEventLayer_management(os.path.join(scratchGDB, "CSV"),"Longitude","Latitude",outLayer,spatialreference,"#")
    # Iterate through dictionary
    for key in dict:
        # Create workspace for each month
        workspace = os.path.join(shpworkspace, key)
        if not os.path.exists(workspace):
            os.makedirs(workspace)

        month = str(dict[key])

        # Create empty tuple
        OIDs = ()

        # Obtain each OID of all features for each month
        with arcpy.da.SearchCursor(os.path.join(scratchGDB, "CSV"), ["OID@", "Date"]) as cursor:
            for row in cursor:
                stringDate = str(row[1])
                month = stringDate.split("-")[1]
                if str(dict[key]) in month:
                    # Add each OID to tuple
                    OIDs = OIDs + (row[0],)

        # Select features using tuple
        arcpy.SelectLayerByAttribute_management(outLayer, "NEW_SELECTION", "OBJECTID IN " + str(OIDs))

        # Copy to shapefile
        csvFileName =  os.path.splitext(csvfile)[0].replace('-','_')
        shpfile = os.path.join(workspace, "Weather_" + csvFileName + ".shp")
        arcpy.CopyFeatures_management(outLayer,shpfile)

del outLayer‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Thu, 18 Aug 2016 16:37:54 GMT

Thanks Jake Skinner‌. Code is working perfectly. Once again thank you very much for the cooperation.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Fri, 19 Aug 2016 12:15:52 GMT

Jake Skinner I have run the code, there is nothing error with this code while i run it on single CSV file but when i run it on 514 CSVs file then Arc GIS suddenly gets shut down (''Arcgis desktop has encountered a serious application error and is unable to continue'') while the time of program execution into 45 number CSVs file.

I have total 514 CSVs file. Earlier which CSV file I had sent for reference purpose that is one of my total CSV file. So upto 44 CSVs program run correctly but when program finished 44 number CSVs conversion than automatically Arc GIS gets shut down. I separately run 45 number CSVs file, but I did not find any error with this file. But when I execute 514 CSVs file at same time then after converting 44 CSVs Arc GIS automatically shut down.

I am using Arc GIS 10.1, on windows 7 32 bit platform. Below I have attached a video file where i pointed the CSV file of error occurrence and Arc GIS error massage and Kindly find attached CSV file also.

I have checked this code on another computers also but there also same issue arising.

Re: Convert multiple csv (specific date range) to shp using ArcPy

JakeSkinner — Fri, 19 Aug 2016 12:41:37 GMT

Try running the script outside of ArcMap using an IDE such as PyScripter. I ran the script using this with ArcGIS Desktop 10.4.1 successfully.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Fri, 19 Aug 2016 16:05:17 GMT

Thank you Jake Skinner‌. Using PyScripter i ran the code throughout the CSVs. Thank you very much for your guidance and cooperation .

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Fri, 19 Aug 2016 18:36:22 GMT

Jake Skinner‌. I have one more query on above program, I would like to add some more step with above program.

Step 1: Above program performing well for converting all CSVs to Point shape file.

After conversion of each CSVs file to Point shape file, I would like to Merge the different month wise all point shapefile (Belongs to same month) into one datebase and save it in existing months wise different folder.

(From this step we will get the month wise single database contain point vector e.g. 001_MERGE, 002_MERGE......012_MERGE)

Step: 2 Using those marge points from different month ( got it from Step: 1 e.g. 001_MERGE, 002_MERGE......012_MERGE) want to perform IDW interpolation for selected parameters from Merge point database (e.g. Max_Temper, Min_Temper,Precipitation, Relative humidity, Wind Speed, Solar etc. ) for creating raster map based on defined mask file (Vector File), cell Size (defined raster), Geo-statistical analysis: Coincident Points (Mean), SEARCH RADIUS : (FIXED Distance : 0.118646 ; MINIMUM NO OF POINTS : 2) . output raster map pass it to as on different month folder (Output raster name for January e.g. 001_Max_Temper, 001_Min_Temper, 001_Precipitation, 001_Wind, 001_RH, 001_Radiation: FEBRUARY e.g. 002_Max_Temper, 002_Min_Temper, 002_Precipitation, 002_Wind, 002_RH, 002_Radiation......till December ).

Step : 3 Whatever monthly raster only for Solar Radiation (e.g 001_Radiation) we got it from Step 2 , We want perform some calculation, (e.g. 001_Radiation * 30) * 0.5 and save it in different month folder, output name ( e.g 001_sr, 002_sr.....012_sr).

Below i have attached the defined mask file and cell size raster.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Sun, 21 Aug 2016 17:22:54 GMT

Jake Skinner‌. Above mentioned query i have written the code upto Merging multiple feature for each month Class. Sorry to inconvenience.

For next step i have put a new question to this community along with my code. Kindly guide me to do the task. Thank you very much

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Sat, 11 Dec 2021 17:06:37 GMT

Hi Jake, Earlier Whatever CSV we converted to point feature class and later we merged them into one feature class month wise (Mereged_001, (Mereged_002...(Mereged_012 ), Now i want to perform Universal Kriging (LINEARDRIFT). I have written a code for that but its not running , Below is my code along with reference data. Any suggestion would be great. Thank you

import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.env.overwriteOutput = True

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")

env.workspace = r"D:\SWAT-WEATHER-DATA"

# Set local variables
inPointFeatures = "D:\SWAT-WEATHER-DATA\SELECTED-EX-MERGE.shp"

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]

#Kriging Veriable
cellSize = 0.002298707671
lagSize = 0.5780481172534
majorRange = 6
partialSill = 3.304292110
nugget = 0.002701348
kRadius = RadiusFixed(20000, 1)

#Mask region of interest
mask="D:\Gujarta Shape file\GUJARATSTATE.shp"

# Execute Kriging
for zField in zFields:
    kModelUniversalObj = KrigingModelUniversal("LINEARDRIFT", lagSize, majorRange, partialSill, nugget)

    OutKriging = Kriging(inPointFeatures, zField, kModelUniversalObj, 20000, kRadius)
    # Execute Mask
    #IDWMASk = ExtractByMask(outIDW, mask)
    KrigMask = ExtractByMask(OutKriging, mask)

    if not zField == "Solar":
    # Save output, except Solar Radiation raster
     KrigMask.save("012_{}.tif".format(zField))

    else:
     #Only for Solar Radiation raster
     PAR = (KrigMask * 30) * 0.5
     #Save output, Only for Solar Radiation raster
     PAR.save("012_{}.tif".format(zField))

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Convert multiple csv (specific date range) to shp using ArcPy

JakeSkinner — Sat, 11 Dec 2021 17:06:40 GMT

Shouvik,

You have your cell size in the Kriging function set to 20,000. I updated the cellSize variable to a larger cell size (0.05) so that it would not take as long to process, and then updated this function with this variable. Also, when specifying paths using a single backslash ("D:\data\etc"), you will want to make sure you have 'r' in front of the path (r"D:\data\etc"). Try the following:

import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.env.overwriteOutput = True

# Check out the ArcGIS Spatial Analyst extension license
arcpy.CheckOutExtension("Spatial")

env.workspace = r"D:\Temp\Python\UserData\Shape_File"

# Set local variables
inPointFeatures = r"D:\Temp\Python\UserData\Shape_File\Merged_004.shp"

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]

#Kriging Veriable
cellSize = 0.05
lagSize = 0.5780481172534
majorRange = 6
partialSill = 3.304292110
nugget = 0.002701348
kRadius = RadiusFixed(20000, 1)

#Mask region of interest
mask = r"D:\Temp\Python\UserData\Shape_File\Boundary.shp"

# Execute Kriging
for zField in zFields:
    kModelUniversalObj = KrigingModelUniversal("LINEARDRIFT", lagSize, majorRange, partialSill, nugget)

    OutKriging = Kriging(inPointFeatures, zField, kModelUniversalObj, cellSize, kRadius)

    # Execute Mask
    #IDWMASk = ExtractByMask(outIDW, mask)
    KrigMask = ExtractByMask(OutKriging, mask)

    if not zField == "Solar":
        # Save output, except Solar Radiation raster
         KrigMask.save("012_{}.tif".format(zField))

    else:
         #Only for Solar Radiation raster
         PAR = (KrigMask * 30) * 0.5
         #Save output, Only for Solar Radiation raster
         PAR.save("012_{}.tif".format(zField))

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Wed, 31 Aug 2016 02:50:09 GMT

jskinner-esristaff‌. Thank you very much. Code working perfectly as you changed the cell value. But output from Kriging seems to be not accurate due to some limitation with kriging. I keep continue my research on this subject.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Sat, 11 Dec 2021 17:06:43 GMT

Hi Jake,I am trying to modify above code, as keep it IDW function same, But instead of giving path for every month shape file , I just trying to use loop function that program can automatic take shape file from different month folder (Name of point shape file in different month folder : JANUARY: Merged_001, FEBRUARY: Merged_002.....DECEMBER: Merged_012 ) and save as it same folder along with month number and Z field name (e.g for JANUARY : 001_Max_Temper, 001_Min_Temper, 001_Precipitat, FEBRUARY : 002_Max_Temper, 002_Min_Temper, 002_Precipitatetc.) . Below is my working code. But its showing error.

import arcpy, os, calendar
arcpy.CheckOutExtension("Spatial")
topWorkspace = r'D:\NPP_GUJARAT\WEATHER_DATA\2011'
arcpy.env.workspace = topWorkspace

 # Get dict of months and month-number (i.e. January = 001, March = 003 etc.)
months = {calendar.month_name[i].upper(): str(i).zfill(3) for i in range(1, 13)} # Get dict of months and month number (i.e. January = 001, March = 003 etc.)

 # Step through list of all folders
for folderPath in arcpy.ListWorkspaces():
    baseName = os.path.basename(folderPath).upper()
    if baseName in months: # Test that subfolder is a month name
        monthNumber = months[baseName] # Get month-number for use in output filename
        arcpy.env.workspace = folderPath

        #Loop every month folder ta take shape file from each month (e.g. Merged_001 (JANUARY), Merged_002 (FEBRUARY)....Merged_012 (DECEMBER))
        shapeFiles = arcpy.ListFeatureClasses('*Merged_{}.shp')

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]
# IDW variable
cellSize = 0.002298707671
power = 2
searchRadius = RadiusVariable(10, 150000)

 #Mask region of interest
mask="D:\Gujarta Shape file\GUJARATSTATE.shp"

for shape_num in shapeFiles:
 Zfield_num = arcpy.ListFeatureClasses(os.path.join(folderPath, shape_num))

 # Execute IDW
 for zField in zFields:
  outIDW = Idw(shapeFiles, zField, cellSize, power, searchRadius)

 # Execute Mask
 IDWMASk = ExtractByMask(outIDW, mask)

 if not zField == "Solar":
        # Save output, except Solar Radiation raster (e.g for JANUARY:001_Max_Temper, FEBRUARY:002_Max_Temper)
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField_num))
        IDWMASk.save(out_name_IDW)
 else:
        #Only for Solar Radiation raster
        PAR = (IDWMASk * 30) * 0.5
        #Save output, Only for Solar Radiation raster
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField))
        PAR.save(out_name_IDW)

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Sat, 11 Dec 2021 17:06:46 GMT

Jake Skinner‌, I am trying to solve the above script problem, but i am getting continue error. Error displaying

IndentationError
 unexpected indent (IDW_PAR_Year_WISE.py, line 10) ‍‍

How to resolve the issue. Thanks

Re: Convert multiple csv (specific date range) to shp using ArcPy

JakeSkinner — Thu, 01 Sep 2016 19:09:29 GMT

This error indicates that you have an indent error. Either the line is indented when it shouldn't be, or vice versa. I would recommend using an IDE such as PyScripter. Within this IDE you can click on the 'Run' menu and choose the option 'Check Syntax'. This will report any syntax errors.

Re: Convert multiple csv (specific date range) to shp using ArcPy

ShouvikJha — Sat, 11 Dec 2021 17:06:49 GMT

Hi Jake, Thank you . I am running the code on Pyscripter. I have updated my code, there is nothing error with my code, but i am not getting any output raster from IDW, below is my updated code. Whats the possible error

import arcpy, os, calendar
from arcpy import env
from arcpy.sa import *
arcpy.CheckOutExtension("Spatial")
topWorkspace = r'D:\NPP_GUJARAT\WEATHER_DATA1\2011'
arcpy.env.workspace = topWorkspace

 # Get dict of months and month-number (i.e. January = 001, March = 003 etc.)
months = {calendar.month_name[i].upper(): str(i).zfill(3) for i in range(1, 13)} # Get dict of months and month number (i.e. January = 001, March = 003 etc.)

 # Step through list of all folders
for folderPath in arcpy.ListWorkspaces():
       baseName = os.path.basename(folderPath).upper()
       if baseName in months: # Test that subfolder is a month name
        monthNumber = months[baseName] # Get month-number for use in output filename
       arcpy.env.workspace = folderPath

        #Loop every month folder ta take shape file from each month (e.g. Merged_001 (JANUARY), Merged_002 (FEBRUARY)....Merged_012 (DECEMBER))
       shapeFiles = arcpy.ListFeatureClasses('Merged_{}*.shp')

zFields = ["Max_Temper", "Min_Temper", "Precipitat", "Wind", "Relative_H", "Solar"]
# IDW variable
cellSize = 0.002298707671
power = 2
searchRadius = RadiusVariable(10, 150000)

 #Mask region of interest
mask="D:\Gujarta Shape file\GUJARATSTATE.shp"

for shape_num in shapeFiles:
 Zfield_num = arcpy.ListFeatureClasses(os.path.join(folderPath, shape_num))

 # Execute IDW
 for zField in zFields:
  outIDW = Idw(shapeFiles, zField, cellSize, power, searchRadius)

 # Execute Mask
 IDWMASk = ExtractByMask(outIDW, mask)

 if not zField == "Solar":
        # Save output, except Solar Radiation raster (e.g for JANUARY:001_Max_Temper, FEBRUARY:002_Max_Temper)
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField_num))
        IDWMASk.save(out_name_IDW)
 else:
        #Only for Solar Radiation raster
        PAR = (IDWMASk * 30) * 0.5
        #Save output, Only for Solar Radiation raster
        out_name_IDW = os.path.join(topWorkspace, '(months)_.TIF'.format(zField))
        PAR.save(out_name_IDW)

#print done
print 'done'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍