using dap2arc and multidimensional tools via python to visualize netdcf results

[ashleytarver1]

HI all

Im in a bit of a pickle.  I'm graduating late, and I'm at a roadblock for my thesis.  I need to pull netcdf files into Arc to visualize some results. I've downloaded both the zipped open2arc folder an the multidimensional tool service package, but now I'm stuck. 

Although I am somewhat familiar with Arc, I have no experience with python or programming.  Which text file from the folders I downloaded need to be copy and pasted into the python window under the "geoprocessing" tab in order to make the tools available for use?

Any suggestions are appreciated!

Thanks in advance!

Ashley

[IanMurray · Accepted solution]

Hi Ashley,

As everyone might not be familiar with the tools you are using, next time please post links to the downloads you are using, without being able to look at them it would be difficult to help you out.

I'm assuming the MultiDimensional Tool Service Package is the one found here, provided by ESRI.  This file contains the netcdf-python module that came with the download, which the scripts from githib needs to work correctly.  You will need to install the module before any of the scripts work.  As for which scripts, that depends on what you are trying to do.  The Github page for dap2arc has a youtube example video which you may want to look at though the scripts are now in the src folder, not in the layers folder it seems.

Disclaimer: I have never used these tools and my opinion is based on what I have discovered from rsignell-usgs/dap2arc · GitHub  and the video provided.

[FilipKrál]

Hi Ashley,

Are you working from local netCDF files or over the web through OpenDAP protocol? Is is a gridded dataset? Are you sure you cannot do what you need using the standard Multidimension Toolbox​?

By open2arc, do you mean dap2arc? I haven't got experience with either but the video at the bottom of the REAMDE.md​ explains quite a bit.

If the zip folders you download have actual toolboxes (.pyt or .tbx) in them, unzip the downloaded folders and you should be able to browse to the unzipped folders and see a toolbox with tools you can use as normal ArcGIS tools.

However, If you need to call these extra tools from Python, you'll need to call arcpy.AddToolbox first.

Let us know how you get on.

Filip.

[ashleytarver1]

Hi guys

Thanks for your response. I'm trying to access ADCIRC netcdf files via the

Thredds server at the OpenDap link here

http://opendap.renci.org:1935/thredds/catalog/tc/isaac/39/GM_LA_TX_v3_chk/blueridge.renci.org/hfip/nhcConsensus/catalog.html

I need to import an unstructured grid (fortran 14 file) into Arc to

visualize the water elevation from Hurricane Isaac along the coast, so I

need this dap2arc extension in order to create the dap2tin python tool that

will convert it to a tin in arc.

I downloaded and extracted the zipped folder, but when I try to pull it

into Arc like the guy in the video, the dap2tinapyt file shows a red "x";

Does this mean I need to first import the toolbox via python? I attached

the downloaded file for reference in case you need it.

Thanks!

[IanMurray]

This is a truncated part of the script, and the part that I think is the issue

class Dap2tin(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Dap2tin"
        self.description = "Access data from the FVCOM Ocean Model via DAP and return a TIN"
        self.canRunInBackground = False
        self.url = None
        self.dataset = None
        self.cols = {
            'url': 0,
            'dataset_var': 1,
            'iyear': 2,
            'imonth': 3,
            'iday': 4,
            'ihour': 5,
            'klev': 6,
            'sr' : 7,
            'out_tin': 8
        }
        self.default_urls = ['http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/hindcasts/30yr_gom3/mean',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/hindcasts/wave_gom3',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/archives/necofs_gom3',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/archives/necofs_mb',
        'http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/necofs/forecasts/necofs_fvcom_ocean_massbay_for...']


    def getParameterInfo(self):
        """Define parameter definitions"""


        # Parameter: URL
        url = arcpy.Parameter(
            displayName="OPeNDAP URL",
            name="url",
            datatype="String",
            parameterType="Required",
            direction="Input")
        # set default value to 30 year hindcast monthly mean dataset
        url.value = 'http://www.smast.umassd.edu:8080/thredds/dodsC/fvcom/hindcasts/30yr_gom3/mean'
        url.filter.type = "ValueList"
        url.filter.list = self.default_urls

I dug into that python toolbox and the issue seems to be with the url parameter, it only accepts web addresses that are defaulted in that list.  If you could add your url to the list, then it could try to process.  I don't have time to try and fix it and upload a new toolbox with script tool.

Filip Král​, you able to help on this?

Also, Curtis Price​, you name is on the credits for a few of those other tools, for helping with them.  You got any suggestions or comments that can help out?

[curtvprice]

Rich Signell is a really nice fellow - I would suggest contacting him - he will help you if he can.

[ashleytarver1]

Ian

I thought at first this was the issue, but after copying and pasting the opendap url into the first line of the dap2tin tool, the url is saved, automatically populates, and becomes an option to select from the drop down menu the next time.

So I assume this means the corresponding script is automatically updated as well.

[IanMurray]

To be honest, I barely tested the tool myself, was in a bit of a hurry after an exam and didn't want to get yelled out for getting home late.  I found something that about matched your description and went with it. 

I'm assuming that you have it working now and can get what you need for your thesis?

[ashleytarver1]

No worries, I appreciate your help!  I'm still getting an error, which I've described in my most recent reply to the thread.    It reads :

KeyError:  'lon'

Not really sure why-I wasnt sure what to choose for my spatial reference, since it is an unstructured grid measured in lat/long units.  So I just chose the State Plane for Louisiana (my area of interest). 

And I'm not really sure what value to enter for the vertical layer-I realize that represents my "z" value, but I'm mapping vind velocity, so I dont think it would be either "-1" or "0".  I tried both ways, but to no avail.

I hope your exam went well! 

[IanMurray]

It doesn't have anything to do with the spatial reference.

A key error occurs when a dictionary key is used to pull a value from a dictionary, and no value returns.

Digging a bit more into the code lines 243-263

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        url = parameters[0]
        variable = parameters[1]


        if url.value is not None and self.url != url.value:
            self.url = url.value
            try:
                self.dataset = netCDF4.Dataset(self.url, filter_out_nd_coordinates=True)
            except:
                arcpy.AddError("Unable to open NetCDF source: {}".format(self.url))
                return
               
            var_names = list(self.dataset.variables.keys())
            parameters[1].filter.list = var_names
            # add back the new url to to filter list
            parameters[0].filter.list = [self.url] + self.default_urls
            if variable.value not in var_names:
                parameters[1].value = var_names[0]
        return

Apparently, I was putting in an invalid URL, so it wasn't updating parameters. Yours was so it didn't x out and added the windx variable to the dropdown.

Lines 295-297

        nc = netCDF4.Dataset(url)
        x = nc.variables['lon'][:]
        y = nc.variables['lat'][:]

This I believe is where we are crashing.  I think in the url it is looking for some sort of lat/long parameters.  Does your error happen to give a line number or any other information so we can make sure what the issue is?

My lack of familiarity with this data is killing me.....