topic Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder in Spatial Data Science Questions

NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Wed, 07 Jun 2017 22:37:34 GMT

I'm working on a project examining potential environmental changes in the Chesapeake Bay watershed and the resulting potential impacts on local jurisdictions and planning. As part of this, I have a number of NetCDF files containing temperature and precipitation data projections across 93 years. How do I, first, separate out certain years as raster files with attribute tables containing the temperature and precipitation data, and second, automate the process using ModelBuilder so that it is easier to duplicate the process across different jurisdictions and regions? I'm using ArcGIS Advanced 10.4.1. netcdf data and model builder netcdf

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisRingo — Wed, 07 Jun 2017 23:14:51 GMT

I haven't used netCDF files in quite some time, but the tools to pull out the individual layers are in the "Multidimension Tools" toolbox; you'd want "Make NetCDF Raster Layer". See the article http://desktop.arcgis.com/en/arcmap/10.4/manage-data/netcdf/reading-netcdf-data-as-a-raster-layer.htm.

My memory is that when you enter an input netCDF file, you'll see a drop-down list of variables contained in the netCDF file - you'll have to see exactly how the data are organized in your files. Then it's easy to pull that tool into model builder (if you know how to use it, that is).

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Thu, 08 Jun 2017 21:56:06 GMT

Thanks for your reply. I'm familiar with the "Make NetCDF Raster" Tool, but when I try and run it, even after defining the projection, ModelBuilder says that there is no spatial reference. I tried both Define Projection and Project (although, by convention, the original NetCDF file should be in WGS 84). And it's mostly after applying that tool that the process becomes a bit more murky for me, in the separating out the individual years and months of interest (namely, 4 specific years and 4 months of each year, one representing each season), and individual jurisdictions of interest. We're looking at 4 particular years and 4 particular months of each year, but the NetCDF files contain data for each month of each year, not annual data. So, in theory, I suppose one must generate an individual raster for each month of each year of interest, then use the Raster Calculator to combine said months into one raster representing the year as a whole. The data also has to be clipped, considering the original raster generated is national in scope and we are looking more regional and local. But I'm very uncertain about the procedure after applying the aforementioned tool. The spatial reference error generally kicks in around where the Clip tool is applied (current flowpath for each month in question is Make NetCDF Raster (month of interest selected under Dimension Values)-->Define Projection (WGS 84)-->Project (MD State Plane NAD 83)-->Clip (Somerset County, MD, MD SP NAD83)-->Raster Calculator (combined with other months, divided by 12)-->Representative Annual Raster for Year. There are preconditions in place to ensure the sequence for each month completes before the subsequent month runs and to ensure all monthly rasters generate before the Raster Calculator runs.)

I've also tried removing the Define Projection, as well as shuffling the flowpath order, flipping the order of the Clip and Project, shifting the clip features into WGS 84 first to run the Clip, then projecting the clipped jurisdiction into the State Plane system afterward. It always results in the same error regarding the spatial reference.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Thu, 08 Jun 2017 22:58:28 GMT

Update: I also tried using the NetCDF Time Series Extract script as detailed at How To: Export each time slice from a NetCDF layer as a single raster (*.tif), but I still got a spatial reference error. I checked the properties for the multidimensional raster generated by the Make NetCDF Raster tool as the preliminary step to using the script, and WGS 84 is set as the datum in the properties, so I'm still unsure where the error is coming from. Could it have something to do with trying to run a transformation to NAD 83 as part of the script environment? The precise error returned from the script was the following. I believe the latter portion is the same as the error message received from ModelBuilder when I attempted the procedure described previously:

ExecuteError: ERROR 999999: Error executing function.
The table name is invalid.
No spatial reference exists.
The table was not found. [NetCDF_Raster.tif]
Failed to execute (CopyRaster).

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisRingo — Fri, 09 Jun 2017 15:35:48 GMT

Are you able to run the Make NetCDF Raster Tool outside of Model Builder? Might be time to attach a sample of your data and your code. The error message seems to indicate a naming problem.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Fri, 09 Jun 2017 16:38:52 GMT

Yes; the tool itself doesn't seem to throw the error. It's any processes run on the output generated by the tool that generate the error.

I will confess that I'm not entirely sure how to attach the code itself. I can attach one of the files I am using relatively easily shortly.

Sent from my iPhone

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisRingo — Fri, 09 Jun 2017 16:46:13 GMT

Have you tried saving the raster layer to an actual raster before running the other processes?

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Fri, 09 Jun 2017 17:34:02 GMT

I'll check that. Unfortunately the NetCDF file I'm using is too large to attach by itself.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Fri, 09 Jun 2017 17:52:59 GMT

I ran Copy Raster on the output of the Make NetCDF Raster tool, but the same error appeared - no spatial reference exists, the table was not found.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisRingo — Fri, 09 Jun 2017 18:03:04 GMT

To get further on this, I think we're going to have to see your exact job stream. Did the same error appear when you ran the Copy Raster tool, or the next step in your job stream? It's difficult to tell from your posts.

You said previously that the error you are getting was:

ExecuteError: ERROR 999999: Error executing function.
The table name is invalid.
No spatial reference exists.
The table was not found. [NetCDF_Raster.tif]
Failed to execute (CopyRaster).

I'm wondering about the "The table name is invalid" part - the problem may not have anything to do with the spatial reference.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Fri, 09 Jun 2017 18:27:14 GMT

I tried running Project Raster on the output of the Make NetCDF Raster Tool, projecting it into the already-applied coordinate system, and then running the time series extract script from there. That seems to resolve the spatial error, as I did not get that. However, it also seems to remove the multidimensional component of the NetCDF raster in the process of running the project at that point, so only one raster is generated by the script, as opposed to one per NetCDF time series entry. So that appears to solve one problem and create another.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Fri, 09 Jun 2017 18:42:07 GMT

That error appeared both when I ran the Copy Raster tool directly on the output of Make NetCDF Raster and also when I attempted to run a Clip after attempting to project it into NAD 83/MD State Plane. I've wondered about the table name & table not found portions of the error myself; there's no obvious cause for it, with no attribute table involved. I've also wondered about the table not found error.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Sat, 10 Jun 2017 17:16:41 GMT

I'll try and post a screenshot of the model as well, presuming that's what you mean by "job stream," as I'm not entirely sure if that's the case, keeping in mind that I have tried several different orders of the tools as well as several different tools, in some cases, and continued to get the same result (aside from the situation described below, which removed this problem but seems to have created another in the process).

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisRingo — Mon, 12 Jun 2017 16:36:22 GMT

You could export the model to python and attach the code. Even more helpful, you could run that code in the python window and provide the results.

Also, did you download the data from a site to which you could provide a link?

At this point unless I can see exactly what you're doing and what data you're doing it to, I really can't speculate on what might be going on. Maybe someone else has an idea from what you've provided so far, but I would need to see exactly what you're working with.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Mon, 12 Jun 2017 19:46:50 GMT

I'll look into doing the code export as soon as I get my external drive with my data on it hooked up again, probably within an hour or so.

The FTP link to the download for the data I am currently using is ftp://gdo-dcp.ucllnl.org/pub/dcp/archive/cmip5/bcsd/BCSD/bcc-csm1-1/rcp26/mon/r1i1p1/pr/.

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

ChrisMcCloud — Wed, 21 Jun 2017 18:54:48 GMT

The output from the ModelBuilder process is as follows:

Executing (Make NetCDF Raster Layer (25)): MakeNetCDFRasterLayer "D:\Todd Research\GIS Datasets\Climate_Projections\CMIP_5_Data2\Monthly\RCP_4.5\Precipitation\BCSD_0.125deg_pr_Amon_bcc-csm1-1_rcp45_r1i1p1_200601-209912.nc" pr longitude latitude Precip_bcc-csm1-1_RCP45_Dec2025 # "time '12/16/2025 12:00:00 PM'" BY_VALUE
Start Time: Wed Jun 21 14:52:43 2017
Succeeded at Wed Jun 21 14:52:43 2017 (Elapsed Time: 0.12 seconds)
Executing (Project): Project "D:\Todd Research\Processing\Chesapeake_Bay.gdb\Somerset_County_MD_2016" "D:\Todd Research\Processing\Chesapeake_Bay.gdb\Somerset_County_MD_2016_WGS84" PROJCS['WGS_1984_World_Mercator',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Mercator'],PARAMETER['False_Easting',0.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',0.0],PARAMETER['Standard_Parallel_1',0.0],UNIT['Meter',1.0]] WGS_1984_(ITRF00)_To_NAD_1983 GEOGCS['GCS_North_American_1983',DATUM['D_North_American_1983',SPHEROID['GRS_1980',6378137.0,298.257222101]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]] NO_PRESERVE_SHAPE # NO_VERTICAL
Start Time: Wed Jun 21 14:52:43 2017
Succeeded at Wed Jun 21 14:52:44 2017 (Elapsed Time: 1.66 seconds)
Executing (Clip (159)): Clip Precip_bcc-csm1-1_RCP45_Dec2025 "283.763542 37.886605 284.460367 38.298944" "D:\Todd Research\Processing\Tangier_Smith_Islands.gdb\Precip_bcc-csm1-1_RCP45_Dec2025_Somerset_MD_WGS84" "D:\Todd Research\Processing\Chesapeake_Bay.gdb\Somerset_County_MD_2016_WGS84" 1.000000e+020 NONE NO_MAINTAIN_EXTENT
Start Time: Wed Jun 21 14:52:45 2017
ERROR 999999: Error executing function.
The table name is invalid.
No spatial reference exists.
FDO error: -2147220948
The table was not found. [Precip_bcc-csm1-1_RCP45_Dec2025_Somerset_MD_WGS84]
Failed to execute (Clip (159)).
Failed at Wed Jun 21 14:52:45 2017 (Elapsed Time: 0.14 seconds)

Re: NetCDF Analysis of Temperature & Precipitation Data + ModelBuilder

IanMurray — Wed, 21 Jun 2017 19:21:31 GMT

As for your most recent error, your output file(Precip_bcc-csm1-1_RCP45_Dec2025_Somerset_MD_WGS84) is an ESRI Grid which can only be 13 characters or less, you need to add a file extension to allow for a longer name(I'd keep going with .tif).

I guess that also begs the question why your initial raster is even created since it is also apparently a Grid ( I don't see a file extension) that is too long of a file name. (Precip_bcc-csm1-1_RCP45_Dec2025)