netCDF4 and ArcGIS

StephenFricke · ‎03-19-2013

I am having issues using the netCDF4 module within my python script for one of my ArcGIS tools. The weird thing is, the tool runs fine the first time, but then when I run the tool a second time I am getting an error. The tool is no longer able to read from the netCDF file, and gives the error when I try to read the latitude values of the file. I then have to restart ArcGIS, and the tool then runs successfully once, but then if I run the tool again I get the same error and need to restart again. Does anyone have any idea what is going on here?

Thank you !

PhilMorefield · ‎03-25-2013

Here is a condensed version of my code. You probably won't be able to access this data, but if you know of any other opendap data with a latitude variable, you could substitute that in the data line.
from netCDF4 import Dataset
data= 'http://cloud.insideidaho.org:8001/reacch/CMIP5/maca_huss_BNU-ESM_historical_1950_1959_WUSA.nc'
MyData=Dataset(data)
lt = MyData.variables['lat'][:]
arcpy.AddMessage(lt)
When I run as a script in an ArcGIS toolbox, it runs successfully the first time, and prints off all of the latitudes, the second time however, I get this error message:

Traceback (most recent call last):
File "C:\netcdf4 trouble\netcdf4problem.py", line 5, in <module>
lt = MyData.variables['lat'][:]
File "netCDF4.pyx", line 2631, in netCDF4.Variable.__getitem__ (netCDF4.c:32455)
File "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", line 127, in _StartCountStride
if np.iterable(elem):
AttributeError: 'NoneType' object has no attribute 'iterable'

Failed to execute (netcdf4problem).

Nothing is jumping out at me. I can see that the climate variable has 3,650 time steps. Is your script designed to repeat itself 3,650 times in order to extract the climate variable and lat/lons for each time step?

StephenFricke · ‎03-25-2013

Nothing is jumping out at me. I can see that the climate variable has 3,650 time steps. Is your script designed to repeat itself 3,650 times in order to extract the climate variable and lat/lons for each time step?

Well the tool is designed so that it can extract a particular climate variable for a particular spatial and temporal extent from the netCDF file. There is 3650 time steps because it is daily data for an entire decade(no leap years). If someone wanted data for every day, I have a tool that can extract data for each time step. I just used this very simple example of the tool because I am having the same issue with this small little snippet of code as I am with my longer more complex code. For some reason I am able to run the tool fine when I first open ArcGIS, but after that when I run it again, for whatever reason, I am getting this error.

PhilMorefield · ‎03-26-2013

Something isn't making sense. The line that is throwing the error is doing nothing except checking to see if "elem" is an iterable. If you open up this file: "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", it looks like the import of numpy as np (on line 1) is only being executed successfully the first time you run the tool. Subsequent runs fail and only return a NoneType object. I have no idea why.

Hopefully someone else can enlighten us both.

StephenFricke · ‎03-26-2013

Something isn't making sense. The line that is throwing the error is doing nothing except checking to see if "elem" is an iterable. If you open up this file: "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", it looks like the import of numpy as np (on line 1) is only being executed successfully the first time you run the tool. Subsequent runs fail and only return a NoneType object. I have no idea why.

Hopefully someone else can enlighten us both.

Yes, very frustrating! I appreciate you taking a look though. So does the ScientificPython module not read opendap? I was thinking that may be a decent alternative if this issue can't be worked out, but I get an invalid argument error when I try and put in a url for the netcdf dataset.

PhilMorefield · ‎03-26-2013

Yes, very frustrating! I appreciate you taking a look though. So does the ScientificPython module not read opendap? I was thinking that may be a decent alternative if this issue can't be worked out, but I get an invalid argument error when I try and put in a url for the netcdf dataset.

I think netCDF4 is the only package with built-in OPeNDAP support. I've used Pydap, but there are no Windows installers that I know of. You can install it using pip or easy_install. And it isn't as clean as the netCDF4 syntax:

from pydap.client import open_url

daymetfile = open_url(r'http://daymet.ornl.gov/thredds/dodsC/allcf/2011/12484_2011/tmax.nc')
var_raw = daymetfile['tmax'].array[100,:,:]  # this slices out data for the 100th day

# then I had to do some reshaping of the array and convert it to a raster using arcpy
# maybe there is a better way to do this?

I would say that netCDF4 is the preferred alternative. The error you're getting doesn't make much sense.

Are you running the condensed code you posted earlier out of a Custom Toolbox or Python Toolbox?

StephenFricke · ‎03-26-2013

I think netCDF4 is the only package with built-in OPeNDAP support. I've used Pydap, but there are no Windows installers that I know of. You can install it using pip or easy_install. And it isn't as clean as the netCDF4 syntax:
from pydap.client import open_url

daymetfile = open_url(r'http://daymet.ornl.gov/thredds/dodsC/allcf/2011/12484_2011/tmax.nc')
var_raw = daymetfile['tmax'].array[100,:,:]  # this slices out data for the 100th day

# then I had to do some reshaping of the array and convert it to a raster using arcpy
# maybe there is a better way to do this?
I would say that netCDF4 is the preferred alternative. The error you're getting doesn't make much sense.

Are you running the condensed code you posted earlier out of a Custom Toolbox or Python Toolbox?

pydap is what I have been using, but recently I have been trying to access data on a thredds server rather than an opendap server, and pydap was giving me issues on the thredds server that I never experienced with the opendap server, so that is when I decided to try netcdf4. The netcdf4 module worked on the thredds server, but only once, like I have explained, and then I keep getting this error each subsequent run. This code is being run out of a normal toolbox. I have never used a python toolbox, do you think that could be the problem?

PhilMorefield · ‎03-27-2013

pydap is what I have been using, but recently I have been trying to access data on a thredds server rather than an opendap server, and pydap was giving me issues on the thredds server that I never experienced with the opendap server, so that is when I decided to try netcdf4. The netcdf4 module worked on the thredds server, but only once, like I have explained, and then I keep getting this error each subsequent run. This code is being run out of a normal toolbox. I have never used a python toolbox, do you think that could be the problem?

Well, the good news is that I can replicate your problem exactly. I'm using ArcGIS 10.0 sp5 on Windows 7 64.

But Rich's tool seems to work fine on ArcGIS 10.1. My guess is that netCDF4 isn't playing nice with the standard toolbox. The only difference I can see between Rich's project and this error is the use of a Python Toolbox.

ClementBouscasse · ‎09-22-2014

Hi All,

I am running into the same issue, using ArcGIS 10.2.2 and the netCDF4 module.

I can confirm that this has nothing to do with the script, which runs fine outside or Arcgis. It also runs fine the first time, before crashing invariably:

File "netCDF4.pyx", line 2986, in netCDF4.Variable.__getitem__ (netCDF4.c:36632)

File "C:\Python27\ArcGIS10.2\ArcGIS10.2\lib\site-packages\netCDF4_utils.py", line 137, in _StartCountStride

if np.iterable(elem):

AttributeError: 'NoneType' object has no attribute 'iterable'

Would a completely statically linked version of netCDF4 fix this problem on 10.2.2 as well?

FYI, for whoever might run into a similar issue, I manage to partially resolve the issue by not running the script "in process". I also had to isolate the netcdf part of the code in its own script, (run outside the main process).

this is very messy because it basically opens a command prompt and starts the script like this:

path_to_script.py "arg1" "arg2" "arg3" .... If users have conflicting python instances, or if py files are not associated with the correct interpreter (for example if associated with a text aditor as many people do) it will fail.

Any help or idea appreciated...

FilipKrál · ‎02-17-2015

We have exactly the same issue, a script from a custom toolbox can be executed only once and then it results in an "AttributeError: 'NoneType' object has no attribute 'iterable'" when trying to subset a variable like:

lt = MyData.variables['lat'][:]

ArcGIS 10.1 SP1 on Win 7 with netCDF4 1.0.4 installed along with the ArcGIS Multidimensional Toolbox.

Any help would be much appreciated!

Filip.

curtvprice · ‎11-05-2015

I think the best solution to this issue is to use the scientific python package available with ArcGIS Pro and soon to be available for ArcGIS 10.4.

Strengthening the Link between GIS and Science | Esri Insider

Esri and the Scientific Community 2015 Roadmap