Problem running Region Group tool

[ClintCabanero]

Hello all,

I'm simply trying to run the RegionGroup Spatial Analyst extension tool in ArcToolbox.  For some reason, when running this against a large raster (northern California), the tools runs 'successfully' - meaning it completes without error - except that the output has a data anomaly - whereby there are striped bands across the output.  The cells in these striped bands also have large integer values (e.g. 1991 while all other values are 1, 2, 3, 4, etc. for each unique group).  This tool problem has been repeated by different people on my team.

I'm seeing this same RegionGroup tool problem in ArcGIS Desktop 10 AND ArcGIS Desktop 10 Service Pack 1.

Can anyone confirm this same problem - or recommend a work around?

Thank you

[curtvprice]

generally, reading writing grid format in SA tools is way faster than FGDB raster format. If you use FGDB format stuff, behind the scenes it is formats to grid, runs the SA tool algorithm, and then, writes the grid output back to FGDB raster fromat. There is a lot of time that is potentially lost in all that reformating...

This is true at 9.3 but not at 10.0. Most raster formats are now supported natively by the tools. ESRI Team Raster people have told me that they have committed themselves GRID and FGDB will probably be the fastest (and probably will have less bit-depth weirdness problems, IMHO). I have heard good things from users, some of which who have moved to FGDB for their raster processing at 10.

What's" rel="nofollow" target="_blank">http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#... new in ArcGIS Spatial Analyst 10
- look for the  heading: Native Read/Write

[PavanYadav]

Greetings,
This issue has been logged as a bug in ArcGIS 10. We will do our best to get a resolution to this issue in an upcoming service pack. Please feel free to let us know about any additional information you feel is relevant to this bug.

Here is the bug ID and synopsis:

NIM070461: The Region Group tool may produce unexpected output - the data may have an anomaly whereby there are striped bands across the output. This happens when the input is fairly large.


Thanks
Pavan

[ChrisSnyder]

Off topic, but...

This is true at 9.3 but not at 10.0

ESRI never said all their SA tools support native read/wriite, and of course never indicated specifically which tools still rely on native grid. In my experience, many of the v10 hydrologic tools (fill, flowdir, etc.) are still much faster in GRID format. Unless someday ESRI says - "now in ArcGIS v23.1 all the SA tools are native FGDB read/write and are now all faster when used in FGDB format than GRID format" I will keep using good ole' GRID format as my default raster format of choice.

[EricRice]

Hi Chris,

All Spatial Analyst tools support native I/O. We enhanced the underlying Spatial Analyst Engine to support native read/write, which every SA tool relies on.

Our development teams did substantial work to make FGDB comparable and sometimes even faster.  Our efforts continue into 10.1 and beyond.  The real benefit is for people who wanted to use FGDB or other formats all along.  i.e. Using FGDB in 10.0 is way faster than using FGDB in 9.3.1.

Performance is gained simply by us not creating scratch GRID's behind the scenes.  Also, performance is not the only benefit to native I/O. See below.

• Any field name and path length limitations imposed by the ESRI GRID format are overcome. (Unless you still choose to use GRID.)


• The 2.1 GB shapefile size limit is avoided by writing outputs to file GDB or SDE.


• If supported by the specified input and output formats, time values in the date fields are preserved, and nulls are treated as such and no longer being converted to zeros.

Our intention with the native I/O project was not to drive people away from using GRID.  If the field and file name limits never bothered you or hindered you in accomplishing something, please continue to use the GRID format.  It is still definitely one of the top two fastest formats to work with.

[ChrisSnyder]

I will plead ignorance here, but why is it that for me a standard 90m USGS DEM for WA State consistently takes 30 seconds to run a Fill command in GRID format while the same raster in FGDB format takes 90 seconds? This users conclusion: FGDB is still, for whatever reason, sometimes significantly slower than GRID. I am left to theorize that for some tools there is still some extra processing overhead happening when the input/output is in a FGDB (or other non-GRID) format.

However, granted, noted, and appreciated that most of the SA tools perform at par or sometimes faster when used with FGDB format (when compared with GRID format). Just curious here, but why aren't all SA the tools faster with FGDB format? Is the algorithm code somehow tailored to specific data formats?

[SamanthaSifleet1]

Greetings,
This issue has been logged as a bug in ArcGIS 10. We will do our best to get a resolution to this issue in an upcoming service pack. Please feel free to let us know about any additional information you feel is relevant to this bug.

Here is the bug ID and synopsis:

NIM070461: The Region Group tool may produce unexpected output - the data may have an anomaly whereby there are striped bands across the output. This happens when the input is fairly large.

I have spent the day dealing with this Bug NIM070461.  I am working on grids of the MW US 30mx30m, one state at a time. Today I play with Minnesota. The tool ran fine this morning out of FGDB (plain 'ole grid). I had no success running the tool from/to FGDB.  I got the stripes.  Now I am running the tool again outside of the FGDB.  Same thing but different.  The tool completes to 100% but never finishes.  In the results window I see an empty output raster.  I have opened that raster and seen the regions as complete, however the attribute table won't display.  So no stripes but no attribute table either.  I am on my second replication of this, tool started running at 15:12:00 said it completed at 15:14:00 it is now 15:28:00 and I see no signs of completion.  Just killed it same thing...no stripes but no attribute table either. - Maybe this will help in fixing the bug.

Side note: Anyone able to get Patch Grids from Patch Analyst 4 up and running in 10?

[EricRice]

Hi Samantha,

We don't always automatically build raster attribute tables.  Try running Build Raster Attribute Table on the result to get the table.

By default, the size of a raster attribute table is limited to 65,535 unique values. You can increase this number on the Options dialog box by clicking the Raster Attribute Table tab on the Raster tab.

Regards,
Eric

[FelixStracke]

After trying to fix this region group issue with workarounds using scripting blablabla - my suggestion is straight from my heart: Use the good old ESRI GRID format. It is indeed a lot of faster (e.g. 1 minute using ESRI Grid as compared to ~ 15min using tif or FGDB raster) and produces a correct output!!!!!

Regards

Felix

[ChrisSnyder]

To add to this issue, in v10.0 SP3, I found that Eulcidean Allocation also suffers from the same sort of buggy behavior as RegionGroup (weird linear artifacts) when the euclidean allocation output grid is placed in FGDB format. When run entirely using grid format, the output is correct.

In my Python code I was able to rectify the problem very simply: By changing the arcpy.env.Workspace variable to a folder (thus forcing the Grid format) instead of using my FGDB as the workspace.

Somewhat related, I find it a bit frustrating in the new v10.0 map algebra that you cannot seem to force the "temp" output rasters to be a particular name, and have to rely on the .save method to, in effect, rename it.

I wish you could do this:

r"C:\Temp\euc_aloc" = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")

But it seems you have to do this:

arcpy.env.workspace = r"C:\Temp"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"C:\Temp\euc_aloc"
junk.save(eucAlocGrd)

If I do this, the euclidean allocation output (in FGDB format) gets wonkey:

arcpy.env.workspace = r"C:\Temp\test.gdb"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"C:\Temp\test.gdb\euc_aloc"
junk.save(eucAlocGrd)

#roadGrd is in Grid format BTW...

[curtvprice]

Chris, did you try:

arcpy.env.scratchWorkspace = r"C:\Temp"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"C:\Temp\test.gdb\euc_aloc"
junk.save(eucAlocGrd) # copies "C:\Temp\raster3" to "C:\Temp\test.gdb\euc_aloc"

This would (in theory) do the processing using grids and then copy the output over to file gdb format (that is, how it works in 9.3 when you specify fgdb output) .

I have been told that (in 9.3, probably same as 10.0) if you set the scratch and current workspace to the same location, the temp grid (say, "raster3") will be created and then renamed (rather than copied) when the tool completes. Not quite what you were looking for, but just as fast:

arcpy.env.scratchWorkspace = r"C:\Temp"
arcpy.env.workspace = r"C:\Temp"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"outgrid"
junk.save(eucAlocGrd) # renames "raster3" to "outgrid"

If you can, *please* submit test data that reproduces these problems with file gdb rasters to ESRI support! Sometimes it is very difficult for them to fix things unless they have a dataset that reproduces the issue.