topic Re: Garbled grid path during geoprocessing in Geoprocessing Questions

Garbled grid path during geoprocessing

StacieWolny — Fri, 25 May 2012 21:08:10 GMT

Hi all -

I developed a python script and toolbox on Arc 10, where it worked fine. Exported the toolbox to 9.3 and now I'm trying to run it on 9.3.1 and something is happening that I've never seen before (after writing and running a lot of geoprocessing scripts in 9.2, 9.3 and 10.) In the script I successfully create a flow accumulation grid, then use it in a SetNull statement. When SetNull runs, I get this error - the path is totally garbled:

ERROR 010316: Unable to open the input grid: G:GISSCRATCHTMP_RI~2OUTPUTHYDRO_~1LOW_ACC
ERROR 010067: Error in executing grid expression.
Failed to execute (SetNull).

Yet, if I print out the grid path right before running SetNull, it looks like this, which is correct:
G:\GIS\scratch\tmp_rios_preproc2\Output\Hydro_layers\flow_acc

There's no problem adding the grid to the map, its folder appears correctly in the Properties. Then if I try to, say, do a math computation on it, that works fine, as does Extract by Mask (just trying things randomly.) However, if I try the same SetNull command from ArcToolbox, or Con, I get the same error as above. The same grid is used in SingleOutuptMapAlgebra within the script with no problems.

It doesn't matter if I write everything to a different directory or disk. I tried restarting the .mxd, tried making a new .mxd, tried making a whole new toolbox using 9.3.1, even tried rebooting the whole darn machine, nothing helps. Has anyone else run into this?

Thanks!

~ Stacie

Re: Garbled grid path during geoprocessing

curtvprice — Fri, 25 May 2012 21:13:20 GMT

When SetNull runs, I get this error - the path is totally garbled:

Stacie, please post your Python script. The map algebra tools are so different in 9.3 so I can see where there may be issues in how you are setting them up.

Re: Garbled grid path during geoprocessing

StacieWolny — Sat, 11 Dec 2021 06:56:39 GMT

Hi Curtis -

The whole script is rather long. Here are the relevant bits, but if you actually want the whole script, I can post it.

# Early in script, define hydrology layers, global variables - this code has been used and re-used for years and worked fine on all Arc versions
        outputws = gp.workspace + os.sep +  "Output" + os.sep
        Hydrows = outputws + "Hydro_layers" + os.sep
        if not gp.exists(Hydrows + "flow_dir"):
            # Create flow direction raster
            gp.FlowDirection_sa(DEM, Hydrows + "flow_dir", "NORMAL")
        flow_dir = Hydrows + "flow_dir"

        if not gp.exists(Hydrows + "flow_acc"):
            # Create flow accumulation raster
            gp.FlowAccumulation_sa(Hydrows + "flow_dir", Hydrows + "flow_acc", "", "FLOAT")
        flow_acc = Hydrows + "flow_acc"

# Later, set flow direction raster to null where there are streams
    def define_channels():
        try:
            gp.AddMessage("\nDefining flow direction channels...")
            exp = "\"VALUE\" > " + str(int(threshold_flowacc))
            # THIS IS WHERE THE ERROR OCCURS
            gp.SetNull_sa(flow_acc, flow_dir, flowdir_channels, exp)
            # ALTO TRIED THIS
            gp.SingleOutputMapAlgebra_sa("CON(" + flow_acc + " <= " + threshold_flowacc + ", " + flow_dir, ")", flowdir_channels)
        except:
            gp.AddError ("\nError defining flow direction channels:  " + gp.GetMessages(2))
            raise Exception

I tried taking this out of a function and putting it into the main body of code, hard-coding the path as a string, using one of the script inputs in place of the generated flow_acc (for the heck of it), all produced the similar errors. Also tried fussing with inputting filename backslashes as either "\" or "\\", didn't help.

When trying the CON code, the error is a bit different:

ERROR 010394: Map Algebra expression: Syntax error at location 3 (around (G:\G) of expression CON(G:\GIS\scratch\tmp_rios_preproc2\Output\Hydro_layers\flow_acc <= 1100, G:\GIS\scratch\tmp_rios_preproc2\Output\Hydro_layers\flow_dir.

Thanks for taking a look at it...

~ Stacie

Re: Garbled grid path during geoprocessing

curtvprice — Tue, 05 Jun 2012 17:56:04 GMT

tried fussing with inputting filename backslashes as either "\" or "\\", didn't help.

You must either use escaped back slashes ("\\") or use raw strings r"G:\mypath".

Something to try:

Because of issues with long paths and the old map algebra intepreter, using raster layers in single-output map algebra expressions is generally a lot safer than passing grid paths directly. In 10.0 this being handled for you because you pass tools like Con raster objects, not rasters. I think the "~" characters in the 8.3-ized path are causing problems for you.

Using raster layers protects the map algebra interpreter from pathnames that can break the syntax, and has other benefits. For example, selected rows (if any) in raster tables are honored, and even better, the raster data are "pre-validated" -- ie pre-checked that they exist etc -- so the tool can start doing its work faster.

lyrFAC = "lyrFAC" lyrFDIR = "lyrFDIR" gp.MakeRasterLayer(flow_acc,lyrFAC) gp.MakeRasterLayer(flow_dir,lyrFDIR) expr = "con(%s <= %s, %s)" % (lyrFAC,1100,lyrFDIR) gp.SingleOutputMapAlgebra_sa(expr,flowdir_channels)

Re: Garbled grid path during geoprocessing

StacieWolny — Tue, 05 Jun 2012 20:19:31 GMT

That worked! Thanks a lot, Curtis. I still don't understand how I've gotten away with using this same code for 4 years without this happening, but such are the vagaries of ArcGIS I guess. Of note, there is no "~" or any other odd character in the path that I enter - I learned that lesson a long time ago. Something in the interpreter was munging my input. And I've learned something about using raster layers. In general I've found them inconvenient because they don't exist anywhere that I can look at them while debugging or just tracing my algorithm's results, but it appears that they have other benefits. Thanks again...

~ Stacie

Re: Garbled grid path during geoprocessing

curtvprice — Sat, 11 Dec 2021 06:56:42 GMT

In general I've found them inconvenient because they don't exist anywhere that I can look at them while debugging or just tracing my algorithm's results

Glad that helped.

Au contraire, raster layer data do 'exist' - Raster layers are like feature layers - they aren't datasets, they are instead objects that point to a dataset source. For this reason the output of tools that create data are datasets, not layers.

When I set up models and tools I almost use layer input so you the tool can see a picklist of layers in the map. Saves a lot of browsing!

The troublesome "~" characters were part of the 8.3-version of your long path -- any path with long names in it is likely to be converted to an "8.3" format behind the scenes by Windows OS routines - and SOMA does not play well with those raw 8.3 pathnames.

It looks like you've just been very prudent so you haven't invoked 8.3 names for the last four years where you were using raster tools (no mixed case, no spaces, short names, etc). The only way I know to check is using "dir /x" at the command line. But not today:

E:\work\test1>dir /x
 Directory of E:\work\test1

06/05/2012  03:42 PM    <DIR>                       .
06/05/2012  03:42 PM    <DIR>                       ..
06/05/2012  03:39 PM    <DIR>                       testa1
06/05/2012  03:40 PM    <DIR>          TESTBV~1     testbverylongname
06/05/2012  03:42 PM    <DIR>          TMP_RI~1     tmp_rios_preproc2

               0 File(s)              0 bytes
               4 Dir(s)  92,880,891,904 bytes free

Isn't Windows great?