topic Re: Export time dimension NetCDF to raster format using ArcPy in Python Questions

Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Sat, 11 Dec 2021 13:48:53 GMT

I have single NetCDF file which is time dimension NetCDF file, the netCDF file contains the monthly weather data for 20 years. I want to convert the NetCDF to raster tiff format and output of the raster as same the month and year name. below is code but it's producing the error in line number 29. Time format of the data is YY-MM-DD 2013-01-01 12:00, 2013-02-01 12:00, so on. How to loop all the time dimension.

updated download link for input files: 2013.nc - Google Drive

I tried to extract the layer manually using ArcGIS tool, it's successfully extracted. I have copied the code from ArcGIS

arcpy.MakeNetCDFRasterLayer_md(in_netCDF_file="E:/Weather/2002.nc",
variable="slhf",
x_dimension="longitude",
y_dimension="latitude",
out_raster_layer="sshf_Layer",
band_dimension="",
dimension_values="time '01-01-2013 12:00:00'",
value_selection_method="BY_VALUE")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *

#Workspace
outLoc = r"E:\Weather/"
inNetCDF = r"E:\Weather\2002.nc"
#Veriable
variable = "slhf"
x_dimension = "lon"
y_dimension = "lat"
band_dimension = ""
dimension = "time"
valueSelectionMethod = "BY_VALUE"

nc_FP = arcpy.NetCDFFileProperties(inNetCDF)
nc_Dim = nc_FP.getDimensions()

for dimension in nc_Dim:
        if dimension == "time":
            top = nc_FP.getDimensionSize(dimension)
            for i in range(0, top):

                dimension_values = nc_FP.getDimensionValue(dimension, i)
                nowFile = str(dimension_values)
                nowFile = nowFile.translate(None, '/')
                # I needed after 2013
                if int(nowFile.split()[0].split('-')[-1])==2013:

                    dv1 = ["time", dimension_values]
                    dimension_values = [dv1]

                    arcpy.MakeNetCDFRasterLayer_md(inNetCDF, variable, x_dimension, y_dimension, nowFile, band_dimension, dimension_values, valueSelectionMethod)
                    print "success"
                    outname = outLoc + nowFile

                    arcpy.CopyRaster_management(nowFile, outname,"", "", "", "NONE", "NONE", "")


                else: print "DATA OUT OF RANGE"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Error message

ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000237: One or more dimensions are invalid
ERROR 000237: One or more dimensions are invalid
Failed to execute (MakeNetCDFRasterLayer).‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Export time dimension NetCDF to raster format using ArcPy

DanPatterson_Retired — Mon, 11 Jun 2018 12:18:26 GMT

/blogs/dan_patterson/2016/08/14/script-formatting It is hard to read and your for loops look they are suffering from a bad copy paste into the code formatter

What are your dimensions? Why not print them out since at least one of them is apparently wrong.

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Mon, 11 Jun 2018 13:03:04 GMT

i have updated the code formatting and chosen Python format.
Regarding dimension, I have time dimension NetCDF file, based on each time file i am trying to export it as a raster.

Re: Export time dimension NetCDF to raster format using ArcPy

DanPatterson_Retired — Mon, 11 Jun 2018 14:53:26 GMT

print the dimensions as you get them in and around line 17-18 since one of your dimensions is wrong.

your indentation is wrong... just changing the parser to Python after the fact won't correct the script, you have to recopy and repaste

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Tue, 12 Jun 2018 05:05:47 GMT

Dan Patterson‌, Thank you. I have added the print statement but the print statement now showing any dimension of NetCDF. I have tried to convert the single layer of NetCDF file to raster using ArcGIS, its successful run. I have copied the code from ArcGIS.

arcpy.MakeNetCDFRasterLayer_md(in_netCDF_file="E:/Weather/2002.nc", variable="slhf", x_dimension="longitude", y_dimension="latitude", out_raster_layer="sshf_Layer", band_dimension="", dimension_values="time '01-01-2013 12:00:00'", value_selection_method="BY_VALUE")

but i have huge layer in NetCDF file, how to loop through all layer and convert it to raster

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Tue, 12 Jun 2018 05:24:06 GMT

Xander Bakker‌, please can you look at the code, why its generating error message,

Re: Export time dimension NetCDF to raster format using ArcPy

DanPatterson_Retired — Tue, 12 Jun 2018 07:47:14 GMT

Can you show the updated code with the print statements and what it is actually printing (not the 'success' print) since one of the dimension names is invalid if the error is still the same)

For instance

x_dimension = "lon"
y_dimension = "lat"

should have been longitude and latitude as in you correct run

also

dv1 = ["time", dimension_values]
dimension_values = [dv1]

why did you make this a list of a list ? since dv1 is already a list

Re: Export time dimension NetCDF to raster format using ArcPy

XanderBakker — Tue, 12 Jun 2018 12:01:29 GMT

I would really like to know what the variable "nowFile" is filled with (if it gets there). Adding some print statements would really help and sharing the complete error message (including the line numbers) would help to understand where things go wrong.

Is it possible to attach the nc file?

Re: Export time dimension NetCDF to raster format using ArcPy

DanPatterson_Retired — Sat, 11 Dec 2021 13:48:55 GMT

Can you run this from your IDE's command line as well to get a listing of the dimensions

import netCDF4

f = r"C:\path\your_netcdf.nc"  # change to match one of your nc files
dataset = netCDF4.Dataset(f)
dataset.dimensions


# ----- report the list that is returned‍‍‍‍‍‍‍‍

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Tue, 12 Jun 2018 13:45:08 GMT

Xander Bakker‌ Thank you. I have added the download link with my question for my input file. Kindly check it.

Re: Export time dimension NetCDF to raster format using ArcPy

XanderBakker — Sat, 11 Dec 2021 13:48:58 GMT

I just changed some code and did a test and it does produce several rasters. See code below:

# Import system modules
import arcpy
import os

#Workspace
outLoc = r"C:\GeoNet\NetCDF"
inNetCDF = r"C:\GeoNet\NetCDF\2013.nc"
#Veriable
variable = "slhf"
x_dimension = "longitude"  # Changed!
y_dimension = "latitude"  # Changed!
band_dimension = ""
dimension = "time"
valueSelectionMethod = "BY_VALUE"

nc_FP = arcpy.NetCDFFileProperties(inNetCDF)
print "nc_FP:", nc_FP
nc_Dim = nc_FP.getDimensions()
print "nc_Dim:", nc_Dim

for dimension in nc_Dim:
    print "dimension:", dimension
    if dimension == "time":
        top = nc_FP.getDimensionSize(dimension)
        print "top:", top
        for i in range(0, top):
            print "i", i
            dimension_values = nc_FP.getDimensionValue(dimension, i)
            print "dimension_values", dimension_values
            nowFile = str(dimension_values)
            print "nowFile (1):", nowFile
            nowFile = nowFile.translate(None, '/')
            print "nowFile (2):", nowFile

            # I needed after 2013
            test = int(nowFile.split()[0].split('-')[-1])
            print "test", test
            if int(nowFile.split()[0].split('-')[-1])==4021900:  # 2013
                dv1 = "time '{}'".format(dimension_values)
                # dv1 = ["time", dimension_values]
                # dimension_values = [dv1]
                out_ras_lay = "sshf_Layer_{}".format(i)

                arcpy.MakeNetCDFRasterLayer_md(inNetCDF, variable, x_dimension, y_dimension, out_ras_lay, band_dimension, dv1, valueSelectionMethod)
                print "success"
                out_name =  "{}.tif".format(out_ras_lay)
                out_ras = os.path.join(outLoc, out_name)
                print "out_name:", out_name
                print "out_ras:", out_ras

                arcpy.CopyRaster_management(out_ras_lay, out_ras,"", "", "", "NONE", "NONE", "")
                print "CopyRaster OK..."

            else:
                print "DATA OUT OF RANGE"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

A number of things that caused problems, for instance:

dimensions X and Y did not have the correct names on lines 10 and 11
On line 38 (in my code) the validation was against 2013, however the year of the dates in the provided NC file was 1900. It never reached the part that converts to raster
The dimension values were not correctly formatted (I'm using line 39)
I changed the output raster layer name to a different unique name, which does not generate problems
I also changed the output rasters names to avoid problems.

A resulting raster (12 were created):

Re: Export time dimension NetCDF to raster format using ArcPy

DanPatterson_Retired — Tue, 12 Jun 2018 23:40:33 GMT

Xander, can you run the snippet that I requested information for.

Good snag on confirming the dimensions provided were wrong

Re: Export time dimension NetCDF to raster format using ArcPy

XanderBakker — Wed, 13 Jun 2018 02:20:35 GMT

Hi Dan, sure, will do that tomorrow

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Sat, 11 Dec 2021 13:49:01 GMT

Xander Bakker‌, Thank you. I ran the code but its showing data out of range. Which NC file I provided you, its contain the monthly data for 2013, but I don't understand how it's showing the year of 1900. maybe a problem with my data however I am providing you the updated link for data download, Kindly look at the updated file of NetCDF

Updated Download link - 2013.nc - Google Drive

After running the code, its showing

dimension_values 01-12-2013 12:00:00
nowFile (1): 01-12-2013 12:00:00
nowFile (2): 01-12-2013 12:00:00
test 2013
DATA OUT OF RANGE‍‍‍‍‍‍‍‍‍‍

I have taken a snapshot of the data from ArcGIS, the time spam of NC looks

Re: Export time dimension NetCDF to raster format using ArcPy

XanderBakker — Wed, 13 Jun 2018 18:15:22 GMT

Hi Shouvik Jha , the data is in a different format. Initially you removed the forward slashed in code from the date and this date format has dashes. That is what is different.

Re: Export time dimension NetCDF to raster format using ArcPy

XanderBakker — Wed, 13 Jun 2018 18:16:41 GMT

It returns an ordered dictionary:

OrderedDict([(u'longitude', <netCDF4.Dimension object at 0x01809C90>), (u'latitude', <netCDF4.Dimension object at 0x0A086690>), (u'time', <netCDF4.Dimension object at 0x0A0866C0>)])‍

Re: Export time dimension NetCDF to raster format using ArcPy

XanderBakker — Sat, 11 Dec 2021 13:49:04 GMT

There is something funky going on here... With the new file I'm still getting the year 1900 as you can see below:

nc_FP: <geoprocessing NetCDF Workspace object object at 0x06640230>
nc_Dim: [u'latitude', u'longitude', u'time']
dimension: latitude
dimension: longitude
dimension: time
top: 12
i 0
dimension_values 4/02/1900 9:28:12 a. m.
nowFile (1): 4/02/1900 9:28:12 a. m.
nowFile (2): 4021900 9:28:12 a. m.
test 4021900
success
out_name: sshf_lay_0.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_0.tif
CopyRaster OK...
i 1
dimension_values 4/02/1900 10:05:24 a. m.
nowFile (1): 4/02/1900 10:05:24 a. m.
nowFile (2): 4021900 10:05:24 a. m.
test 4021900
success
out_name: sshf_lay_1.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_1.tif
CopyRaster OK...
i 2
dimension_values 4/02/1900 10:39:00 a. m.
nowFile (1): 4/02/1900 10:39:00 a. m.
nowFile (2): 4021900 10:39:00 a. m.
test 4021900
success
out_name: sshf_lay_2.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_2.tif
CopyRaster OK...
i 3
dimension_values 4/02/1900 11:16:12 a. m.
nowFile (1): 4/02/1900 11:16:12 a. m.
nowFile (2): 4021900 11:16:12 a. m.
test 4021900
success
out_name: sshf_lay_3.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_3.tif
CopyRaster OK...
i 4
dimension_values 4/02/1900 11:52:12 a. m.
nowFile (1): 4/02/1900 11:52:12 a. m.
nowFile (2): 4021900 11:52:12 a. m.
test 4021900
success
out_name: sshf_lay_4.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_4.tif
CopyRaster OK...
i 5
dimension_values 4/02/1900 12:29:24 p. m.
nowFile (1): 4/02/1900 12:29:24 p. m.
nowFile (2): 4021900 12:29:24 p. m.
test 4021900
success
out_name: sshf_lay_5.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_5.tif
CopyRaster OK...
i 6
dimension_values 4/02/1900 1:05:24 p. m.
nowFile (1): 4/02/1900 1:05:24 p. m.
nowFile (2): 4021900 1:05:24 p. m.
test 4021900
success
out_name: sshf_lay_6.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_6.tif
CopyRaster OK...
i 7
dimension_values 4/02/1900 1:42:36 p. m.
nowFile (1): 4/02/1900 1:42:36 p. m.
nowFile (2): 4021900 1:42:36 p. m.
test 4021900
success
out_name: sshf_lay_7.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_7.tif
CopyRaster OK...
i 8
dimension_values 4/02/1900 2:19:48 p. m.
nowFile (1): 4/02/1900 2:19:48 p. m.
nowFile (2): 4021900 2:19:48 p. m.
test 4021900
success
out_name: sshf_lay_8.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_8.tif
CopyRaster OK...
i 9
dimension_values 4/02/1900 2:55:48 p. m.
nowFile (1): 4/02/1900 2:55:48 p. m.
nowFile (2): 4021900 2:55:48 p. m.
test 4021900
success
out_name: sshf_lay_9.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_9.tif
CopyRaster OK...
i 10
dimension_values 4/02/1900 3:33:00 p. m.
nowFile (1): 4/02/1900 3:33:00 p. m.
nowFile (2): 4021900 3:33:00 p. m.
test 4021900
success
out_name: sshf_lay_10.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_10.tif
CopyRaster OK...
i 11
dimension_values 4/02/1900 4:09:00 p. m.
nowFile (1): 4/02/1900 4:09:00 p. m.
nowFile (2): 4021900 4:09:00 p. m.
test 4021900
success
out_name: sshf_lay_11.tif
out_ras: C:\GeoNet\NetCDF\sshf_lay_11.tif
CopyRaster OK...‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Could this be something related to regional settings?

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Wed, 13 Jun 2018 19:09:34 GMT

Xander Bakker‌ same data i am running on my PC with the same code provided by u but i am getting data out of range.

Please could you clarify why it is happening and when i opening the same data in ArcGIS , the time spam showing 2013 which i shown in attached image, but your code displays 1900 .

I am getting confused

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Wed, 13 Jun 2018 19:11:36 GMT

Xander Bakker‌, please could you open the data in ArcGIS to cross-check the input file with my attached image from ArcGIS

Re: Export time dimension NetCDF to raster format using ArcPy

ShouvikJha — Sat, 11 Dec 2021 13:49:06 GMT

Xander Bakker‌ the same data which I provided you and same code, I ran on my system and it is showing

nc_FP: <geoprocessing NetCDF Workspace object object at 0x0295B530>
nc_Dim: [u'longitude', u'latitude', u'time']
dimension: longitude
dimension: latitude
dimension: time
top: 12
i 0
dimension_values 1/1/2013 12:00:00 PM
nowFile (1): 1/1/2013 12:00:00 PM
nowFile (2): 112013 12:00:00 PM
test 112013
DATA OUT OF RANGE
i 1
dimension_values 2/1/2013 12:00:00 PM
nowFile (1): 2/1/2013 12:00:00 PM
nowFile (2): 212013 12:00:00 PM
test 212013
DATA OUT OF RANGE
i 2
dimension_values 3/1/2013 12:00:00 PM
nowFile (1): 3/1/2013 12:00:00 PM
nowFile (2): 312013 12:00:00 PM
test 312013
DATA OUT OF RANGE
i 3
dimension_values 4/1/2013 12:00:00 PM
nowFile (1): 4/1/2013 12:00:00 PM
nowFile (2): 412013 12:00:00 PM
test 412013
DATA OUT OF RANGE
i 4
dimension_values 5/1/2013 12:00:00 PM
nowFile (1): 5/1/2013 12:00:00 PM
nowFile (2): 512013 12:00:00 PM
test 512013
DATA OUT OF RANGE
i 5
dimension_values 6/1/2013 12:00:00 PM
nowFile (1): 6/1/2013 12:00:00 PM
nowFile (2): 612013 12:00:00 PM
test 612013
DATA OUT OF RANGE
i 6
dimension_values 7/1/2013 12:00:00 PM
nowFile (1): 7/1/2013 12:00:00 PM
nowFile (2): 712013 12:00:00 PM
test 712013
DATA OUT OF RANGE
i 7
dimension_values 8/1/2013 12:00:00 PM
nowFile (1): 8/1/2013 12:00:00 PM
nowFile (2): 812013 12:00:00 PM
test 812013
DATA OUT OF RANGE
i 8
dimension_values 9/1/2013 12:00:00 PM
nowFile (1): 9/1/2013 12:00:00 PM
nowFile (2): 912013 12:00:00 PM
test 912013
DATA OUT OF RANGE
i 9
dimension_values 10/1/2013 12:00:00 PM
nowFile (1): 10/1/2013 12:00:00 PM
nowFile (2): 1012013 12:00:00 PM
test 1012013
DATA OUT OF RANGE
i 10
dimension_values 11/1/2013 12:00:00 PM
nowFile (1): 11/1/2013 12:00:00 PM
nowFile (2): 1112013 12:00:00 PM
test 1112013
DATA OUT OF RANGE
i 11
dimension_values 12/1/2013 12:00:00 PM
nowFile (1): 12/1/2013 12:00:00 PM
nowFile (2): 1212013 12:00:00 PM
test 1212013
DATA OUT OF RANGE‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍