topic Re: Batch Processing for Make NetCDF Raster Layer in Python Questions

Batch Processing for Make NetCDF Raster Layer

fabefabe — Sat, 23 Apr 2011 12:52:06 GMT

Hi all,

I have hundreds of netcdf files and I need to convert them into grid format permanently. Instead of going through each netcdf files and converting them into grid and making them permanent, I am trying to batch this process. But, due to limited knowledge in python, I am having hard time in achieving this target. Has anyone run into this problem and come up with the solution?. I will really appreciate your help.

Thanks,
Sami

Re: Batch Processing for Make NetCDF Raster Layer

SarahBurns — Fri, 20 May 2011 03:21:53 GMT

I would like help with this too, let me know if you find a way.
Thanks

Re: Batch Processing for Make NetCDF Raster Layer

RafaGutierrez — Wed, 01 Jun 2011 17:27:11 GMT

Have you tried right-clicking the tool and choose the 'Batch' option?

Re: Batch Processing for Make NetCDF Raster Layer

PamFroemke — Thu, 24 May 2012 20:50:35 GMT

Have you tried right-clicking the tool and choose the 'Batch' option?

It looks like you still have to enter each netcdf file separately? Not very useful for 500 of them.

Re: Batch Processing for Make NetCDF Raster Layer

PhilMorefield — Fri, 10 Dec 2021 23:15:41 GMT

This type of task is very common in GIS work and, fortunately, requires only a very simple script. Your first step should be to read the documentation and make sure you understand what information you need to run the MakeNetCDFRasterLayer tool. Then start with this:

import arcpy as ap
import glob

# this will give you a Python list object that you can use to batch process all of your files
# just insert the path to your folder holding the netCDF files
cdfList = glob.glob('C:\\examplefolder\\*.nc')

# now you can loop through your list and process each file one at a time
for cdf in cdfList:
    print "Now processing: " + cdf
    ap.md.MakeNetCDFRasterLayer(cdf, [fill in the other variables here])
print "Done!"

Note that this is only the start of the script. For example, you'll need to make sure each output raster is given a unique, meaningful name. But we should leave some of the fun stuff for you. 🙂

Re: Batch Processing for Make NetCDF Raster Layer

RPatarasuk — Fri, 10 Dec 2021 23:15:43 GMT

This is my code. I got it to loop through every file in a folder. The problem is that it only reads the first value. I haven't figured out how to loop through each individual value in the netCDF file yet. I have been trying to modify the code from this forum: http://forums.arcgis.com/threads/32361-using-arcpy-to-export-separate-netcdf-dimension-values-as-a-raster so it reads individual value.

# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *

# Input data source
arcpy.env.workspace = "S:/PM_data/Climate_Idaho/Raw/pr"
arcpy.env.overwriteOutput = True

# Set output folder
OutputFolder = "S:/PM_data/Climate_Idaho/Output/pr_raster"


# Loop through a list of files in the workspace
NCfiles = arcpy.ListFiles("*.nc")

for filename in NCfiles:
    print("Processing: " + filename)
    inNCfiles = arcpy.env.workspace + "/" + filename
    fileroot = filename[0:(len(filename)-3)]
    TempLayerFile = "precipitation_amount"
    outRaster = OutputFolder + "/" + fileroot

    # Process: Make NetCDF Raster Layer
    arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile, "", "", "BY_VALUE")

    # Process: Copy Raster
    arcpy.CopyRaster_management(TempLayerFile, outRaster + ".tif", "", "", "", "NONE", "NONE", "")
   

print "***DONE!!!"
print arcpy.GetMessages()

Re: Batch Processing for Make NetCDF Raster Layer

PhilMorefield — Fri, 10 Dec 2021 23:15:46 GMT

# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *

# Input data source
arcpy.env.workspace = "S:/PM_data/Climate_Idaho/Raw/pr"
arcpy.env.overwriteOutput = True

# Set output folder
OutputFolder = "S:/PM_data/Climate_Idaho/Output/pr_raster"


# Loop through a list of files in the workspace
NCfiles = arcpy.ListFiles("*.nc")

for filename in NCfiles:
    print("Processing: " + filename)
    inNCfiles = arcpy.env.workspace + "/" + filename
    fileroot = filename[0:(len(filename)-3)]
    TempLayerFile = "precipitation_amount"
    outRaster = OutputFolder + "/" + fileroot

    # Process: Make NetCDF Raster Layer
    arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile, "", "", "BY_VALUE")

    # Process: Copy Raster
    arcpy.CopyRaster_management(TempLayerFile, outRaster + ".tif", "", "", "", "NONE", "NONE", "")
   

print "***DONE!!!"
print arcpy.GetMessages()

Note that you need to call arcpy.MakeNetCDFRasterLayer_md() for each time step in each netCDF file. You're looping through your files fine, but you need to add another 'inner' loop to go through an individual file. I would try to use "BY_INDEX" instead of "BY_VALUE". Let's assume that your data are in time series, so the "dimension value" is "TIME":

# this will convert the FIRST time step in your netCDF file to a raster layer
arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile1, "#", "TIME 0", "BY_INDEX")

# this will convert the SECOND time step in your netCDF file to a raster layer
arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile2, "#", "TIME 1", "BY_INDEX")

# this will convert the THIRD time step in your netCDF file to a raster layer
arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile3, "#", "TIME 2", "BY_INDEX")

#and so on...

Obviously you can't just copy-and-paste this code, but this should give you an idea of how to set it up.

Re: Batch Processing for Make NetCDF Raster Layer

PamFroemke — Fri, 10 Dec 2021 23:15:49 GMT

This type of task is very common in GIS work and, fortunately, requires only a very simple script. Your first step should be to read the documentation and make sure you understand what information you need to run the MakeNetCDFRasterLayer tool. Then start with this:
import arcpy as ap
import glob

# this will give you a Python list object that you can use to batch process all of your files
# just insert the path to your folder holding the netCDF files
cdfList = glob.glob('C:\\examplefolder\\*.nc')

# now you can loop through your list and process each file one at a time
for cdf in cdfList:
    print "Now processing: " + cdf
    ap.md.MakeNetCDFRasterLayer(cdf, [fill in the other variables here])
print "Done!"
Note that this is only the start of the script. For example, you'll need to make sure each output raster is given a unique, meaningful name. But we should leave some of the fun stuff for you. 🙂

Thank you! I meant to write earlier that this script idea worked like a charm. I used an inner loop with Select by Dimension to evaluate each month in each netcdf file. I noticed you had a different method elsewhere in this thread. I'll try that next time.
Thanks again,
Pam
PS - I can't find the 'green checkmark' that is supposed to indicate this answered my question so I'll just say, "This answered my question!".

Re: Batch Processing for Make NetCDF Raster Layer

PhilMorefield — Wed, 20 Jun 2012 14:25:53 GMT

Thank you! I meant to write earlier that this script idea worked like a charm. I used an inner loop with Select by Dimension to evaluate each month in each netcdf file. I noticed you had a different method elsewhere in this thread. I'll try that next time.
Thanks again,
Pam
PS - I can't find the 'green checkmark' that is supposed to indicate this answered my question so I'll just say, "This answered my question!".

Great! I'll also mention that, compared to third-party Python packages, processing netCDF data in ArcGIS is very, very slow (sorry ESRI). For big jobs, or for folks working with netCDF data regularly, I would recommend ScientificPython. That way you can work with each step in the 'time' dimension as a numpy array, do whatever averaging or calculation is needed, and just use ArcGIS at the end to produce your GIS-friendly output.

Re: Batch Processing for Make NetCDF Raster Layer

RPatarasuk — Fri, 10 Dec 2021 23:15:51 GMT

I got a better code now but it's not really perfect yet

#Make a separate raster *.img file for each dimension value (time) in a netcdf file

import arcpy, os, time, datetime, calendar
from arcpy import env
from arcpy.sa import *

# Input data source

#########Change folder
arcpy.env.workspace = "S:/PM_data/Climate_Idaho/Raw/tmmx" # Change folder++
#########

arcpy.env.overwriteOutput = True
arcpy.env.scratchWorkspace = "S:/Work/Risa/Trials_Errors/ScratchWorkSpace"

# Loop through a list of files in the workspace
NetCDFfiles = arcpy.ListFiles("*.nc")

for filename in NetCDFfiles:
    print("Processing: " + filename)
        
    # Set local variables
    inNetCDF = arcpy.env.workspace + "/" + filename

#########Change the variable
    variable = "air_temperature" 
#########
    
    XDimension = "lon"
    YDimension = "lat"
    bandDimension = ""
    dimensionValues = "day "
    valueSelectionMethod = "BY_VALUE"

#########Change the variable ###
    InMemory_netcdf_raster = "air_temperature" 
#########
    
#########Change folder    
    outRasterFolder = "S:/PM_data/Climate_Idaho/Output/tmmx_raster"
#########
    
    #Global Attribute:
    #ncFP = arcpy.NetCDFFileProperties(inNetCDF)
    #ncDim = ncFP.getDimensions()

    #Get Global Attribues
    #ncAttributeNames = ncFP.getAttributeNames("")
    #for ncAttributeName in ncAttributeNames:
        #print " "
        #print "Attribute Name: %s " % ncAttributeName
        #print ncFP.getAttributeValue("", ncAttributeName)
        #print " "
        
    #Year info

#######change the lengths
    fileroot = filename[5:(len(filename)-3)] #change the name lengths too
#######
    
    Yr = int(fileroot) 
    print "Processing --> " + str(Yr)
    
    #Month info
    allmnths = range(1,13)
    for mnths in allmnths:
        Lastday = calendar.monthrange(Yr, mnths)[1]
        MRange = range(1,Lastday+1)
        for dyy in MRange:
            dyys =int(dyy)
            mnthsss = int(mnths)
            
            a = str(mnthsss)+ "/" + str(dyys) +"/" + str(Yr)

            #IF Days or Months less than 10, we want to put zero at the front so it is easier to sort.

            if mnthsss < 10: 
                b = str(Yr)+ "_" + "0" + str(mnthsss)+ "_" + "0" + str(dyys)
            else:
                b = str(Yr)+ "_" + str(mnthsss)+ "_" + str(dyys)

            if dyys < 10:
                b = str(Yr)+ "_" + "0" + str(mnthsss)+ "_" + "0" + str(dyys)                   
            else:
                b = str(Yr)+ "_" + "0" + str(mnthsss)+ "_" + str(dyys)
                
        ##############  ####### ####### ####### ####### ####### ####### #######
        #the OUTPUT has zero in front of October, Nov, and DEc (month value >10)
        #NEED TO FIX THIS ***
        ##############  ####### ####### ####### ####### ####### ####### #######
                
            dimensionValues = "day " + a
            
##########CHANGE the word too##            
                                        #################
            outTIFF = outRasterFolder + "/" + "tmmx_" + b + ".tif"
                                        #################

            
            # Execute MakeNetCDFRasterLayer
            arcpy.MakeNetCDFRasterLayer_md(inNetCDF, variable, XDimension, YDimension, InMemory_netcdf_raster, bandDimension, dimensionValues, valueSelectionMethod)
            #print "Created NetCDF Layer for " + a
            
            # Process: Copy Raster
            arcpy.CopyRaster_management(InMemory_netcdf_raster, outTIFF, "", "", "", "NONE", "NONE", "")
            print "Created Raster for _tmmx_" + b
                                #################
            
print "\nEnd of processing ... YAHOO!!!"

Re: Batch Processing for Make NetCDF Raster Layer

vegv — Thu, 18 Apr 2013 10:10:22 GMT

hi
im trying to do the conversion discussed above but im completely new to python scripting. i copy pasted the script u posted on python idle and saved the file. then i created a script in a toolbox and tried to run it. and it didnt run.

the netcdf data that i have is a monthly avg.i created a folder for each year and stored 12 files, 1 for each month. so i dont think i have the time dimension you mentioned

Regards,
Veg