topic Re: netCDF4 and ArcGIS in Python Questions

netCDF4 and ArcGIS

StephenFricke — Tue, 19 Mar 2013 20:01:13 GMT

I am having issues using the netCDF4 module within my python script for one of my ArcGIS tools. The weird thing is, the tool runs fine the first time, but then when I run the tool a second time I am getting an error. The tool is no longer able to read from the netCDF file, and gives the error when I try to read the latitude values of the file. I then have to restart ArcGIS, and the tool then runs successfully once, but then if I run the tool again I get the same error and need to restart again. Does anyone have any idea what is going on here?

Thank you !

Re: netCDF4 and ArcGIS

PhilMorefield — Tue, 19 Mar 2013 20:28:44 GMT

The conflict between netCDF4 and ArcGIS has a long and illustrious history. Example here.

Your problem sounds different from the one I had a few years ago, though. So small chance it's a problem with your code. Beyond that, there are at least two alternative netCDF readers that do work with ArcGIS:

ScientificPython - I've been submitting Windows installers for this project. Right now the latest one is compatible with AGD 10.0.

scipy.io.netcdf - This will bomb on really, really big files. ScientificPython won't.

There are others, and they all use very similar APIs. Unfortunately, none of them offer the cool bells-and-whistles that netCDF4 offers.

Hope that helps.

Re: netCDF4 and ArcGIS

StephenFricke — Tue, 19 Mar 2013 21:38:40 GMT

The conflict between netCDF4 and ArcGIS has a long and illustrious history. Example here.

Your problem sounds different from the one I had a few years ago, though. So small chance it's a problem with your code. Beyond that, there are at least two alternative netCDF readers that do work with ArcGIS:

ScientificPython - I've been submitting Windows installers for this project. Right now the latest one is compatible with AGD 10.0.

scipy.io.netcdf - This will bomb on really, really big files. ScientificPython won't.

There are others, and they all use very similar APIs. Unfortunately, none of them offer the cool bells-and-whistles that netCDF4 offers.

Hope that helps.

Hey, thank you for the response. I am running AGD 10.1, which uses python 2.7. It looks like the latest windows installer is for python 2.6. I believe I tried this one and was having compatibility issues with ArcGIS 10.1. Do you know any window installers for python 2.7? Thanks again.

Re: netCDF4 and ArcGIS

PhilMorefield — Wed, 20 Mar 2013 12:00:09 GMT

Hey, thank you for the response. I am running AGD 10.1, which uses python 2.7. It looks like the latest windows installer is for python 2.6. I believe I tried this one and was having compatibility issues with ArcGIS 10.1. Do you know any window installers for python 2.7? Thanks again.

I submitted an installer for Python 2.7 to the ScientificPython author. He usually posts those to the webpage in ~24 hours. If you need it before he posts it, send me a message with your email.

Re: netCDF4 and ArcGIS

RichardSignell — Wed, 20 Mar 2013 15:23:51 GMT

Just a week ago Christoph Gohlke built a completely static NetCDF4 module specifically for use with ArcGIS 10.1, and in my testing it has worked flawlessly. Check out the link to netCDF4-1.0.5dev-ArcGIS-10.1.win32-py2.7.exe on Christoph's amazing Python libraries for Windows page: http://www.lfd.uci.edu/~gohlke/pythonlibs/#netcdf4

Re: netCDF4 and ArcGIS

PhilMorefield — Wed, 20 Mar 2013 16:20:33 GMT

Just a week ago Christoph Gohlke built a completely static NetCDF4 module specifically for use with ArcGIS 10.1, and in my testing it has worked flawlessly. Check out the link to netCDF4-1.0.5dev-ArcGIS-10.1.win32-py2.7.exe on Christoph's amazing Python libraries for Windows page: http://www.lfd.uci.edu/~gohlke/pythonlibs/#netcdf4

That's a good point. That netCDF4 installer does work for me on a 64-bit Windows 7 machine, but not on my 32-bit Windows XP computers.

Re: netCDF4 and ArcGIS

StephenFricke — Fri, 22 Mar 2013 18:23:31 GMT

Thank you so much for the responses. I downloaded the python 2.7 version of ScientificPython, and for some reason I am not having success running the module within Arc. I keep getting an error of "IOError: netcdf: Invalid argument", but the same argument seems to be working outside of Arc. I also downloaded the mentioned netCDF4 module and I am getting the same error I was originally getting in the first post. My script runs perfect one time through, but then the next time through I am have problems. I first read in the netcdf file using:

dataset= Dataset('nameofmynetcdf.nc')

And then I try and read the latitudes using the command:

lt = dataset.variables['lat'][:]

This works perfectly the first time through, but then second time through I get an error for this line which states:

AttributeError: 'NoneType' object has no attribute 'iterable'

Does anyone know why this would be? I am reading this data via opendap if that makes any difference at all(?). I would really appreciate any help anyone has to offer. Thank you!

Re: netCDF4 and ArcGIS

PhilMorefield — Fri, 22 Mar 2013 19:45:51 GMT

Thank you so much for the responses. I downloaded the python 2.7 version of ScientificPython, and for some reason I am not having success running the module within Arc. I keep getting an error of "IOError: netcdf: Invalid argument", but the same argument seems to be working outside of Arc. I also downloaded the mentioned netCDF4 module and I am getting the same error I was originally getting in the first post. My script runs perfect one time through, but then the next time through I am have problems. I first read in the netcdf file using:

dataset= Dataset('nameofmynetcdf.nc')

And then I try and read the latitudes using the command:

lt = dataset.variables['lat'][:]

This works perfectly the first time through, but then second time through I get an error for this line which states:

AttributeError: 'NoneType' object has no attribute 'iterable'

Does anyone know why this would be? I am reading this data via opendap if that makes any difference at all(?). I would really appreciate any help anyone has to offer. Thank you!

It might best if you posted some or all of your code, and included the line number of the error. There are instruction at the top of the Python forum on how to post code here.

Re: netCDF4 and ArcGIS

RichardSignell — Fri, 22 Mar 2013 20:22:44 GMT

I also downloaded the mentioned netCDF4 module and I am getting the same error I was originally getting in the first post. I am reading this data via opendap if that makes any difference at all(?). I would really appreciate any help anyone has to offer. Thank you!

See if you can run this python toolbox, which uses NetCDF4 with OPeNDAP in ArcGIS 10.1:
https://github.com/rsignell-usgs/dap2arc

If it works for you more than once, you could then focus on issues with your code.

Re: netCDF4 and ArcGIS

StephenFricke — Sat, 11 Dec 2021 13:33:23 GMT

Here is a condensed version of my code. You probably won't be able to access this data, but if you know of any other opendap data with a latitude variable, you could substitute that in the data line.

from netCDF4 import Dataset
data= 'http://cloud.insideidaho.org:8001/reacch/CMIP5/maca_huss_BNU-ESM_historical_1950_1959_WUSA.nc'
MyData=Dataset(data)
lt = MyData.variables['lat'][:]
arcpy.AddMessage(lt)

When I run as a script in an ArcGIS toolbox, it runs successfully the first time, and prints off all of the latitudes, the second time however, I get this error message:

Traceback (most recent call last):
File "C:\netcdf4 trouble\netcdf4problem.py", line 5, in <module>
lt = MyData.variables['lat'][:]
File "netCDF4.pyx", line 2631, in netCDF4.Variable.__getitem__ (netCDF4.c:32455)
File "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", line 127, in _StartCountStride
if np.iterable(elem):
AttributeError: 'NoneType' object has no attribute 'iterable'

Failed to execute (netcdf4problem).

Re: netCDF4 and ArcGIS

PhilMorefield — Sat, 11 Dec 2021 13:33:25 GMT

Here is a condensed version of my code. You probably won't be able to access this data, but if you know of any other opendap data with a latitude variable, you could substitute that in the data line.
from netCDF4 import Dataset
data= 'http://cloud.insideidaho.org:8001/reacch/CMIP5/maca_huss_BNU-ESM_historical_1950_1959_WUSA.nc'
MyData=Dataset(data)
lt = MyData.variables['lat'][:]
arcpy.AddMessage(lt)
When I run as a script in an ArcGIS toolbox, it runs successfully the first time, and prints off all of the latitudes, the second time however, I get this error message:

Traceback (most recent call last):
File "C:\netcdf4 trouble\netcdf4problem.py", line 5, in <module>
lt = MyData.variables['lat'][:]
File "netCDF4.pyx", line 2631, in netCDF4.Variable.__getitem__ (netCDF4.c:32455)
File "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", line 127, in _StartCountStride
if np.iterable(elem):
AttributeError: 'NoneType' object has no attribute 'iterable'

Failed to execute (netcdf4problem).

Nothing is jumping out at me. I can see that the climate variable has 3,650 time steps. Is your script designed to repeat itself 3,650 times in order to extract the climate variable and lat/lons for each time step?

Re: netCDF4 and ArcGIS

StephenFricke — Tue, 26 Mar 2013 03:38:36 GMT

Nothing is jumping out at me. I can see that the climate variable has 3,650 time steps. Is your script designed to repeat itself 3,650 times in order to extract the climate variable and lat/lons for each time step?

Well the tool is designed so that it can extract a particular climate variable for a particular spatial and temporal extent from the netCDF file. There is 3650 time steps because it is daily data for an entire decade(no leap years). If someone wanted data for every day, I have a tool that can extract data for each time step. I just used this very simple example of the tool because I am having the same issue with this small little snippet of code as I am with my longer more complex code. For some reason I am able to run the tool fine when I first open ArcGIS, but after that when I run it again, for whatever reason, I am getting this error.

Re: netCDF4 and ArcGIS

PhilMorefield — Tue, 26 Mar 2013 12:41:09 GMT

Something isn't making sense. The line that is throwing the error is doing nothing except checking to see if "elem" is an iterable. If you open up this file: "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", it looks like the import of numpy as np (on line 1) is only being executed successfully the first time you run the tool. Subsequent runs fail and only return a NoneType object. I have no idea why.

Hopefully someone else can enlighten us both.

Re: netCDF4 and ArcGIS

StephenFricke — Tue, 26 Mar 2013 15:56:37 GMT

Something isn't making sense. The line that is throwing the error is doing nothing except checking to see if "elem" is an iterable. If you open up this file: "C:\Python27\ArcGIS10.1\lib\site-packages\netCDF4_utils.py", it looks like the import of numpy as np (on line 1) is only being executed successfully the first time you run the tool. Subsequent runs fail and only return a NoneType object. I have no idea why.

Hopefully someone else can enlighten us both.

Yes, very frustrating! I appreciate you taking a look though. So does the ScientificPython module not read opendap? I was thinking that may be a decent alternative if this issue can't be worked out, but I get an invalid argument error when I try and put in a url for the netcdf dataset.

Re: netCDF4 and ArcGIS

PhilMorefield — Sat, 11 Dec 2021 13:33:28 GMT

Yes, very frustrating! I appreciate you taking a look though. So does the ScientificPython module not read opendap? I was thinking that may be a decent alternative if this issue can't be worked out, but I get an invalid argument error when I try and put in a url for the netcdf dataset.

I think netCDF4 is the only package with built-in OPeNDAP support. I've used Pydap, but there are no Windows installers that I know of. You can install it using pip or easy_install. And it isn't as clean as the netCDF4 syntax:

from pydap.client import open_url

daymetfile = open_url(r'http://daymet.ornl.gov/thredds/dodsC/allcf/2011/12484_2011/tmax.nc')
var_raw = daymetfile['tmax'].array[100,:,:]  # this slices out data for the 100th day

# then I had to do some reshaping of the array and convert it to a raster using arcpy
# maybe there is a better way to do this?

I would say that netCDF4 is the preferred alternative. The error you're getting doesn't make much sense.

Are you running the condensed code you posted earlier out of a Custom Toolbox or Python Toolbox?

Re: netCDF4 and ArcGIS

StephenFricke — Sat, 11 Dec 2021 13:33:31 GMT

I think netCDF4 is the only package with built-in OPeNDAP support. I've used Pydap, but there are no Windows installers that I know of. You can install it using pip or easy_install. And it isn't as clean as the netCDF4 syntax:
from pydap.client import open_url

daymetfile = open_url(r'http://daymet.ornl.gov/thredds/dodsC/allcf/2011/12484_2011/tmax.nc')
var_raw = daymetfile['tmax'].array[100,:,:]  # this slices out data for the 100th day

# then I had to do some reshaping of the array and convert it to a raster using arcpy
# maybe there is a better way to do this?
I would say that netCDF4 is the preferred alternative. The error you're getting doesn't make much sense.

Are you running the condensed code you posted earlier out of a Custom Toolbox or Python Toolbox?

pydap is what I have been using, but recently I have been trying to access data on a thredds server rather than an opendap server, and pydap was giving me issues on the thredds server that I never experienced with the opendap server, so that is when I decided to try netcdf4. The netcdf4 module worked on the thredds server, but only once, like I have explained, and then I keep getting this error each subsequent run. This code is being run out of a normal toolbox. I have never used a python toolbox, do you think that could be the problem?

Re: netCDF4 and ArcGIS

PhilMorefield — Wed, 27 Mar 2013 22:55:52 GMT

pydap is what I have been using, but recently I have been trying to access data on a thredds server rather than an opendap server, and pydap was giving me issues on the thredds server that I never experienced with the opendap server, so that is when I decided to try netcdf4. The netcdf4 module worked on the thredds server, but only once, like I have explained, and then I keep getting this error each subsequent run. This code is being run out of a normal toolbox. I have never used a python toolbox, do you think that could be the problem?

Well, the good news is that I can replicate your problem exactly. I'm using ArcGIS 10.0 sp5 on Windows 7 64.

But Rich's tool seems to work fine on ArcGIS 10.1. My guess is that netCDF4 isn't playing nice with the standard toolbox. The only difference I can see between Rich's project and this error is the use of a Python Toolbox.

Re: netCDF4 and ArcGIS

ClementBouscasse — Mon, 22 Sep 2014 14:36:15 GMT

Hi All,

I am running into the same issue, using ArcGIS 10.2.2 and the netCDF4 module.

I can confirm that this has nothing to do with the script, which runs fine outside or Arcgis. It also runs fine the first time, before crashing invariably:

File "netCDF4.pyx", line 2986, in netCDF4.Variable.__getitem__ (netCDF4.c:36632)

File "C:\Python27\ArcGIS10.2\ArcGIS10.2\lib\site-packages\netCDF4_utils.py", line 137, in _StartCountStride

if np.iterable(elem):

AttributeError: 'NoneType' object has no attribute 'iterable'

Would a completely statically linked version of netCDF4 fix this problem on 10.2.2 as well?

FYI, for whoever might run into a similar issue, I manage to partially resolve the issue by not running the script "in process". I also had to isolate the netcdf part of the code in its own script, (run outside the main process).

this is very messy because it basically opens a command prompt and starts the script like this:

path_to_script.py "arg1" "arg2" "arg3" .... If users have conflicting python instances, or if py files are not associated with the correct interpreter (for example if associated with a text aditor as many people do) it will fail.

Any help or idea appreciated...

Re: netCDF4 and ArcGIS

FilipKrál — Tue, 17 Feb 2015 08:22:38 GMT

We have exactly the same issue, a script from a custom toolbox can be executed only once and then it results in an "AttributeError: 'NoneType' object has no attribute 'iterable'" when trying to subset a variable like:

lt = MyData.variables['lat'][:]

ArcGIS 10.1 SP1 on Win 7 with netCDF4 1.0.4 installed along with the ArcGIS Multidimensional Toolbox.

Any help would be much appreciated!

Filip.

Re: netCDF4 and ArcGIS

curtvprice — Thu, 05 Nov 2015 17:35:18 GMT

I think the best solution to this issue is to use the scientific python package available with ArcGIS Pro and soon to be available for ArcGIS 10.4.

Strengthening the Link between GIS and Science | Esri Insider

Esri and the Scientific Community 2015 Roadmap