nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Batch Processing for Make NetCDF Raster Layer

fabefabe

Hi all,

I have hundreds of netcdf files and I need to convert them into grid format permanently. Instead of going through each netcdf files and converting them into grid and making them permanent, I am trying to batch this process. But, due to limited knowledge in python, I am having hard time in achieving this target. Has anyone run into this problem and come up with the solution?. I will really appreciate your help.

Thanks,
Sami

Trouver plus de publications taguées avec

Accepted answers

All comments

vegv

hi
im trying to do the conversion discussed above but im completely new to python scripting. i copy pasted the script u posted on python idle and saved the file. then i created a script in a toolbox and tried to run it. and it didnt run.

the netcdf data that i have is a monthly avg.i created a folder for each year and stored 12 files, 1 for each month. so i dont think i have the time dimension you mentioned

Regards,
Veg

RPatarasuk

I got a better code now but it's not really perfect yet

#Make a separate raster *.img file for each dimension value (time) in a netcdf file

import arcpy, os, time, datetime, calendar
from arcpy import env
from arcpy.sa import *

# Input data source

#########Change folder
arcpy.env.workspace = "S:/PM_data/Climate_Idaho/Raw/tmmx" # Change folder++
#########

arcpy.env.overwriteOutput = True
arcpy.env.scratchWorkspace = "S:/Work/Risa/Trials_Errors/ScratchWorkSpace"

# Loop through a list of files in the workspace
NetCDFfiles = arcpy.ListFiles("*.nc")

for filename in NetCDFfiles:
    print("Processing: " + filename)
        
    # Set local variables
    inNetCDF = arcpy.env.workspace + "/" + filename

#########Change the variable
    variable = "air_temperature" 
#########
    
    XDimension = "lon"
    YDimension = "lat"
    bandDimension = ""
    dimensionValues = "day "
    valueSelectionMethod = "BY_VALUE"

#########Change the variable ###
    InMemory_netcdf_raster = "air_temperature" 
#########
    
#########Change folder    
    outRasterFolder = "S:/PM_data/Climate_Idaho/Output/tmmx_raster"
#########
    
    #Global Attribute:
    #ncFP = arcpy.NetCDFFileProperties(inNetCDF)
    #ncDim = ncFP.getDimensions()

    #Get Global Attribues
    #ncAttributeNames = ncFP.getAttributeNames("")
    #for ncAttributeName in ncAttributeNames:
        #print " "
        #print "Attribute Name: %s " % ncAttributeName
        #print ncFP.getAttributeValue("", ncAttributeName)
        #print " "
        
    #Year info

#######change the lengths
    fileroot = filename[5:(len(filename)-3)] #change the name lengths too
#######
    
    Yr = int(fileroot) 
    print "Processing --> " + str(Yr)
    
    #Month info
    allmnths = range(1,13)
    for mnths in allmnths:
        Lastday = calendar.monthrange(Yr, mnths)[1]
        MRange = range(1,Lastday+1)
        for dyy in MRange:
            dyys =int(dyy)
            mnthsss = int(mnths)
            
            a = str(mnthsss)+ "/" + str(dyys) +"/" + str(Yr)

            #IF Days or Months less than 10, we want to put zero at the front so it is easier to sort.

            if mnthsss < 10: 
                b = str(Yr)+ "_" + "0" + str(mnthsss)+ "_" + "0" + str(dyys)
            else:
                b = str(Yr)+ "_" + str(mnthsss)+ "_" + str(dyys)

            if dyys < 10:
                b = str(Yr)+ "_" + "0" + str(mnthsss)+ "_" + "0" + str(dyys)                   
            else:
                b = str(Yr)+ "_" + "0" + str(mnthsss)+ "_" + str(dyys)
                
        ##############  ####### ####### ####### ####### ####### ####### #######
        #the OUTPUT has zero in front of October, Nov, and DEc (month value >10)
        #NEED TO FIX THIS ***
        ##############  ####### ####### ####### ####### ####### ####### #######
                
            dimensionValues = "day " + a
            
##########CHANGE the word too##            
                                        #################
            outTIFF = outRasterFolder + "/" + "tmmx_" + b + ".tif"
                                        #################

            
            # Execute MakeNetCDFRasterLayer
            arcpy.MakeNetCDFRasterLayer_md(inNetCDF, variable, XDimension, YDimension, InMemory_netcdf_raster, bandDimension, dimensionValues, valueSelectionMethod)
            #print "Created NetCDF Layer for " + a
            
            # Process: Copy Raster
            arcpy.CopyRaster_management(InMemory_netcdf_raster, outTIFF, "", "", "", "NONE", "NONE", "")
            print "Created Raster for _tmmx_" + b
                                #################
            
print "\nEnd of processing ... YAHOO!!!"

PhilMorefield

Thank you! I meant to write earlier that this script idea worked like a charm. I used an inner loop with Select by Dimension to evaluate each month in each netcdf file. I noticed you had a different method elsewhere in this thread. I'll try that next time.
Thanks again,
Pam
PS - I can't find the 'green checkmark' that is supposed to indicate this answered my question so I'll just say, "This answered my question!".

Great! I'll also mention that, compared to third-party Python packages, processing netCDF data in ArcGIS is very, very slow (sorry ESRI). For big jobs, or for folks working with netCDF data regularly, I would recommend ScientificPython. That way you can work with each step in the 'time' dimension as a numpy array, do whatever averaging or calculation is needed, and just use ArcGIS at the end to produce your GIS-friendly output.

PamFroemke

This type of task is very common in GIS work and, fortunately, requires only a very simple script. Your first step should be to read the documentation and make sure you understand what information you need to run the MakeNetCDFRasterLayer tool. Then start with this:
import arcpy as ap
import glob

# this will give you a Python list object that you can use to batch process all of your files
# just insert the path to your folder holding the netCDF files
cdfList = glob.glob('C:\\examplefolder\\*.nc')

# now you can loop through your list and process each file one at a time
for cdf in cdfList:
    print "Now processing: " + cdf
    ap.md.MakeNetCDFRasterLayer(cdf, [fill in the other variables here])
print "Done!"
Note that this is only the start of the script. For example, you'll need to make sure each output raster is given a unique, meaningful name. But we should leave some of the fun stuff for you. 🙂

Thank you! I meant to write earlier that this script idea worked like a charm. I used an inner loop with Select by Dimension to evaluate each month in each netcdf file. I noticed you had a different method elsewhere in this thread. I'll try that next time.
Thanks again,
Pam
PS - I can't find the 'green checkmark' that is supposed to indicate this answered my question so I'll just say, "This answered my question!".

PhilMorefield

This is my code. I got it to loop through every file in a folder. The problem is that it only reads the first value. I haven't figured out how to loop through each individual value in the netCDF file yet. I have been trying to modify the code from this forum: http://forums.arcgis.com/threads/32361-using-arcpy-to-export-separate-netcdf-dimension-values-as-a-raster so it reads individual value.

# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *

# Input data source
arcpy.env.workspace = "S:/PM_data/Climate_Idaho/Raw/pr"
arcpy.env.overwriteOutput = True

# Set output folder
OutputFolder = "S:/PM_data/Climate_Idaho/Output/pr_raster"


# Loop through a list of files in the workspace
NCfiles = arcpy.ListFiles("*.nc")

for filename in NCfiles:
    print("Processing: " + filename)
    inNCfiles = arcpy.env.workspace + "/" + filename
    fileroot = filename[0:(len(filename)-3)]
    TempLayerFile = "precipitation_amount"
    outRaster = OutputFolder + "/" + fileroot

    # Process: Make NetCDF Raster Layer
    arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile, "", "", "BY_VALUE")

    # Process: Copy Raster
    arcpy.CopyRaster_management(TempLayerFile, outRaster + ".tif", "", "", "", "NONE", "NONE", "")
   

print "***DONE!!!"
print arcpy.GetMessages()

Note that you need to call arcpy.MakeNetCDFRasterLayer_md() for each time step in each netCDF file. You're looping through your files fine, but you need to add another 'inner' loop to go through an individual file. I would try to use "BY_INDEX" instead of "BY_VALUE". Let's assume that your data are in time series, so the "dimension value" is "TIME":

# this will convert the FIRST time step in your netCDF file to a raster layer
arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile1, "#", "TIME 0", "BY_INDEX")

# this will convert the SECOND time step in your netCDF file to a raster layer
arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile2, "#", "TIME 1", "BY_INDEX")

# this will convert the THIRD time step in your netCDF file to a raster layer
arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile3, "#", "TIME 2", "BY_INDEX")

#and so on...

Obviously you can't just copy-and-paste this code, but this should give you an idea of how to set it up.

RPatarasuk

# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *

# Input data source
arcpy.env.workspace = "S:/PM_data/Climate_Idaho/Raw/pr"
arcpy.env.overwriteOutput = True

# Set output folder
OutputFolder = "S:/PM_data/Climate_Idaho/Output/pr_raster"


# Loop through a list of files in the workspace
NCfiles = arcpy.ListFiles("*.nc")

for filename in NCfiles:
    print("Processing: " + filename)
    inNCfiles = arcpy.env.workspace + "/" + filename
    fileroot = filename[0:(len(filename)-3)]
    TempLayerFile = "precipitation_amount"
    outRaster = OutputFolder + "/" + fileroot

    # Process: Make NetCDF Raster Layer
    arcpy.MakeNetCDFRasterLayer_md(inNCfiles, "precipitation_amount", "lon", "lat", TempLayerFile, "", "", "BY_VALUE")

    # Process: Copy Raster
    arcpy.CopyRaster_management(TempLayerFile, outRaster + ".tif", "", "", "", "NONE", "NONE", "")
   

print "***DONE!!!"
print arcpy.GetMessages()

PhilMorefield

This type of task is very common in GIS work and, fortunately, requires only a very simple script. Your first step should be to read the documentation and make sure you understand what information you need to run the MakeNetCDFRasterLayer tool. Then start with this:

import arcpy as ap
import glob

# this will give you a Python list object that you can use to batch process all of your files
# just insert the path to your folder holding the netCDF files
cdfList = glob.glob('C:\\examplefolder\\*.nc')

# now you can loop through your list and process each file one at a time
for cdf in cdfList:
    print "Now processing: " + cdf
    ap.md.MakeNetCDFRasterLayer(cdf, [fill in the other variables here])
print "Done!"

Note that this is only the start of the script. For example, you'll need to make sure each output raster is given a unique, meaningful name. But we should leave some of the fun stuff for you. 🙂

PamFroemke

Have you tried right-clicking the tool and choose the 'Batch' option?

It looks like you still have to enter each netcdf file separately? Not very useful for 500 of them.

RafaGutierrez

Have you tried right-clicking the tool and choose the 'Batch' option?

SarahBurns

I would like help with this too, let me know if you find a way.
Thanks

Rejoindre la conversation

Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.

Liens rapides

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.