topic Re: using dap2arc and multidimensional tools via python to visualize netdcf results in Geoprocessing Questions

using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Tue, 05 May 2015 20:15:18 GMT

HI all

Im in a bit of a pickle. I'm graduating late, and I'm at a roadblock for my thesis. I need to pull netcdf files into Arc to visualize some results. I've downloaded both the zipped open2arc folder an the multidimensional tool service package, but now I'm stuck.

Although I am somewhat familiar with Arc, I have no experience with python or programming. Which text file from the folders I downloaded need to be copy and pasted into the python window under the "geoprocessing" tab in order to make the tools available for use?

Any suggestions are appreciated!

Thanks in advance!

Ashley

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

IanMurray — Tue, 05 May 2015 21:41:18 GMT

Hi Ashley,

As everyone might not be familiar with the tools you are using, next time please post links to the downloads you are using, without being able to look at them it would be difficult to help you out.

I'm assuming the MultiDimensional Tool Service Package is the one found here, provided by ESRI. This file contains the netcdf-python module that came with the download, which the scripts from githib needs to work correctly. You will need to install the module before any of the scripts work. As for which scripts, that depends on what you are trying to do. The Github page for dap2arc has a youtube example video which you may want to look at though the scripts are now in the src folder, not in the layers folder it seems.

Disclaimer: I have never used these tools and my opinion is based on what I have discovered from rsignell-usgs/dap2arc · GitHub and the video provided.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

FilipKrál — Tue, 05 May 2015 21:52:26 GMT

Hi Ashley,

Are you working from local netCDF files or over the web through OpenDAP protocol? Is is a gridded dataset? Are you sure you cannot do what you need using the standard Multidimension Toolbox?

By open2arc, do you mean dap2arc? I haven't got experience with either but the video at the bottom of the REAMDE.md explains quite a bit.

If the zip folders you download have actual toolboxes (.pyt or .tbx) in them, unzip the downloaded folders and you should be able to browse to the unzipped folders and see a toolbox with tools you can use as normal ArcGIS tools.

However, If you need to call these extra tools from Python, you'll need to call arcpy.AddToolbox first.

Let us know how you get on.

Filip.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Tue, 05 May 2015 23:44:57 GMT

Hi guys

Thanks for your response. I'm trying to access ADCIRC netcdf files via the

Thredds server at the OpenDap link here

http://opendap.renci.org:1935/thredds/catalog/tc/isaac/39/GM_LA_TX_v3_chk/blueridge.renci.org/hfip/nhcConsensus/catalog.html

I need to import an unstructured grid (fortran 14 file) into Arc to

visualize the water elevation from Hurricane Isaac along the coast, so I

need this dap2arc extension in order to create the dap2tin python tool that

will convert it to a tin in arc.

I downloaded and extracted the zipped folder, but when I try to pull it

into Arc like the guy in the video, the dap2tinapyt file shows a red "x";

Does this mean I need to first import the toolbox via python? I attached

the downloaded file for reference in case you need it.

Thanks!

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

IanMurray — Sat, 11 Dec 2021 09:33:26 GMT

This is a truncated part of the script, and the part that I think is the issue

class Dap2tin(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Dap2tin"
        self.description = "Access data from the FVCOM Ocean Model via DAP and return a TIN"
        self.canRunInBackground = False
        self.url = None
        self.dataset = None
        self.cols = {
            'url': 0,
            'dataset_var': 1,
            'iyear': 2,
            'imonth': 3,
            'iday': 4,
            'ihour': 5,
            'klev': 6,
            'sr' : 7,
            'out_tin': 8
        }
        self.default_urls = ['http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/hindcasts/30yr_gom3/mean',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/hindcasts/wave_gom3',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/archives/necofs_gom3',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/archives/necofs_mb',
        'http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc',
        'http://www.smast.umassd.edu:8080/thredds/dodsc/fvcom/necofs/forecasts/necofs_fvcom_ocean_massbay_forecast.nc']


    def getParameterInfo(self):
        """Define parameter definitions"""


        # Parameter: URL
        url = arcpy.Parameter(
            displayName="OPeNDAP URL",
            name="url",
            datatype="String",
            parameterType="Required",
            direction="Input")
        # set default value to 30 year hindcast monthly mean dataset
        url.value = 'http://www.smast.umassd.edu:8080/thredds/dodsC/fvcom/hindcasts/30yr_gom3/mean'
        url.filter.type = "ValueList"
        url.filter.list = self.default_urls

I dug into that python toolbox and the issue seems to be with the url parameter, it only accepts web addresses that are defaulted in that list. If you could add your url to the list, then it could try to process. I don't have time to try and fix it and upload a new toolbox with script tool.

Filip Král, you able to help on this?

Also, Curtis Price, you name is on the credits for a few of those other tools, for helping with them. You got any suggestions or comments that can help out?

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

curtvprice — Wed, 06 May 2015 05:05:42 GMT

The dap2tinapyt file shows a red "x"; does this mean I need to first import the toolbox via python?

It means there is a syntax error in the pyt file. If you right click the toolbox there is a "Check Syntax" menu option you can click.

There is no need to import toolboxes of any kind in 10x; they can be accessed directly from the catalog location.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

curtvprice — Wed, 06 May 2015 05:07:59 GMT

Rich Signell is a really nice fellow - I would suggest contacting him - he will help you if he can.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

IanMurray — Wed, 06 May 2015 14:20:53 GMT

I took her response to mean she opened the toolbox in the catalog, but once she tried to put in the URL parameter, it said it was invalid, so the red x showed up next to the parameter. The tool had no syntax errors when I tried it, I suppose she needs to be sure that the netcdf python module is installed.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

curtvprice — Wed, 06 May 2015 15:52:02 GMT

Ian Murray wrote:

I suppose she needs to be sure that the netcdf python module is installed.

Which can be a challenge with 10.1, as Rich has discovered. (He was involved in some of the Python discussions between the oceanography/meteorology community and Esri, fortunately.)

Fortunately the netcdf module will be included in the Python stack that will ship with 10.3.1 so tools like Richard Signell's will have netcdf pre-loaded and ready to roll.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Wed, 06 May 2015 16:07:48 GMT

I have ArcMap 10.2. Is that the problem?

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

curtvprice — Wed, 06 May 2015 16:46:40 GMT

I don't think so. 10.1 SP 1 through 10.2.x are pretty much compatible.

Does the toolbox fail to load (which I thought was the issue) or is the URL parameter not validating (which Ian suspects)? if the latter, all you need to do is add your URLs to the filter code as he suggests.

The netcdf issue is a sticky wicket because it involves finding a version that runs compatible with arcpy. To see if that's an issue, try "import netcdf" in your arcmap python window.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Wed, 06 May 2015 22:51:00 GMT

Forgive me, its like you're all speaking a foreign language to me. Yuk yuk

(couldn't help myself)

I think the problem is that I need to install the netcdf python module. I

have the .pyt (dap2tin) saved to my desktop, and when it shows up in my

catalog it has a red "x" next to it and when I check the syntax it reads :

'Parsing error SyntaxError: invalid syntax line'.

Thank you, when I get to my desktop tonight I will mark the helpful

comments in the hopes this may help another beginner to python like me!

-Ashley

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

IanMurray — Thu, 07 May 2015 13:48:58 GMT

Okay, once you downloaded the multidimeonsional tool zip and unzip it, open the folder and there is an netCDF4-1.0.4.win32-py2.7.exe. That is the installer for the netcdf python module. Let it run and it will install the netcdf python module on your computer.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Thu, 07 May 2015 23:01:57 GMT

Ian

I thought at first this was the issue, but after copying and pasting the opendap url into the first line of the dap2tin tool, the url is saved, automatically populates, and becomes an option to select from the drop down menu the next time.

So I assume this means the corresponding script is automatically updated as well.

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Thu, 07 May 2015 23:12:24 GMT

OK! Progress has been made, this pleases me! I installed the module, which enabled me to access the dap2tin toolbox in the catalogue window.

I accessed the fortran 74 file (fort.74.nc) from the link here http://opendap.renci.org:1935/thredds/catalog/tc/isaac/catalog.html

It shows the wind velocity at all the nodes on my unstructured grid.

Then I typed in the corresponding date for Hurricane Isaac (the storm simulated within the file), and chose the "windx" variable from the drop-down window, which will show the east/west wind velocity of the storm.

Unlike the video, Arc10.2 has an additional line in the toolbox that requires the spatial reference. I am unsure what to put here, since the output was created on a lat/long grid, so I chose the corresponding State Plane Coordinate system for Louisiana (the area of interest).

I chose a location (my C drive) for the output TIN, then clicked "OK", but a box pops up with red letters that read:

KeyError 'lon'

Failed to Execute dap2tin

Do you know what I am doing wrong? The file i'm attempting to pull in is a netcdf format (fort.74.nc), which should already have the mesh built in.

Since I was unsure what input value I should set as the vertical layer, I tried both "0" and "-1", but to no avail. The same error I described above pops up.

Thanks!

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

IanMurray — Thu, 07 May 2015 23:14:39 GMT

To be honest, I barely tested the tool myself, was in a bit of a hurry after an exam and didn't want to get yelled out for getting home late. I found something that about matched your description and went with it.

I'm assuming that you have it working now and can get what you need for your thesis?

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

IanMurray — Thu, 07 May 2015 23:22:22 GMT

can you put in a screenshot of your input parameters you are using for the tool?

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Thu, 07 May 2015 23:26:46 GMT

No worries, I appreciate your help! I'm still getting an error, which I've described in my most recent reply to the thread. It reads :

KeyError: 'lon'

Not really sure why-I wasnt sure what to choose for my spatial reference, since it is an unstructured grid measured in lat/long units. So I just chose the State Plane for Louisiana (my area of interest).

And I'm not really sure what value to enter for the vertical layer-I realize that represents my "z" value, but I'm mapping vind velocity, so I dont think it would be either "-1" or "0". I tried both ways, but to no avail.

I hope your exam went well!

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Thu, 07 May 2015 23:45:14 GMT

Sure. This is the most recent attempt and result

Re: using dap2arc and multidimensional tools via python to visualize netdcf results

ashleytarver1 — Thu, 07 May 2015 23:47:02 GMT

When I try to save the output to my C drive another way:

On Thu, May 7, 2015 at 7:44 PM, Ashley Tarver <tarverashley190@gmail.com>