Reading large NetCDF file

Question

I'm trying to create a large NetCDF file that I can load into ArcMap 10.0
For smaller files (eg 100 MB) it works, but for larger files (eg 5GB) the Make NetCDF Raster Layer tool either takes hours, or hangs indefinitely (possibly because I give up after a few hours).
Are there any limitations on how large a NetCDF file can be?
Are there any particular requirements for ArcMap to load the file in a reasonable time?
The 5GB file I'm trying to load at the moment has this header (from ncdump -h)netcdf largeFillValue { // format variant: 64bit
dimensions:
        x = 39393 ;
        y = 17766 ;
variables:
        double x(x) ;
        double y(y) ;
        double z(x, y) ;
                z:_FillValue = 9.96920996838687e+036 ;
}

Do I need to add any specific parameters to the file?
The values are all eastings/northings, not lon/lat.

JeffreySwain · Answer

If you are creating the NetCDF, I always recommend using a format checker, like the one found here to find any errors with regards to the formatting.  Out or curiosity, how is the performance of a 1 gig NetCDF file on your system?  Most of the NetCDFs that I have used are no where near that size of file and still generate fairly good sized rasters when exported.  Have you taken a look at how large the 5 Gig NetCDF raster output will be?  I still would consider using a smaller file based on your description.

__11 · Answer

I've found the problem. Z must be stored in row-major order, ie z(y, x), not z(x, y).
The latter works, but is unusably slow especially with larger data sets (presumably once the size of z exceeds the cache).

Well that fix works in some cases....
Specifically on Windows 7 64-bit with 32GB RAM, ArcGIS Desktop 10, ArcMap 10.0 build 2414, it works - loading the 5GB NetCDF file in about 2 minutes.
However on Windows XP 32-bit with 4GB RAM, ArcGIS Desktop 10 SP2, ArcMAP 10.0 build 3200, the same file thrashes the harddisk continuously and takes 15 minutes or longer to load (and then is unusably slow).

__11 · Answer

I've found the problem. Z must be stored in row-major order, ie z(y, x), not z(x, y).
The latter works, but is unusably slow especially with larger data sets (presumably once the size of z exceeds the cache).