Problem running Region Group tool

Hopefully the other tools mentioned in this thread that suffer from the same/simlar issue were also addressed.

This is still an issue with 10.0.3.....

[post=224254]As mentioned above by Pavan,[/post] this was fixed in 10.0 SP4 and later. You really should update to 10.0 SP5 or 10.1 SP1 if you can.

NIM 070461 The Region Group tool may produce unexpected output - the data may have an anomaly whereby there are striped bands across the output. This happens when the input is fairly large.

als21

This is still an issue with 10.0.3.....

there are bands of lost data that are filled with bands of "copied" data from above.

Using a GRID file as input & output has solved this issue for me.

thanks for all the posts...

Yes, I could submit some repro data... but more importantly I just want to raise the issue that there is likely a more of a systemic issue going on here though that doesn't involve just a single tool.... I belive the the 'NoData' horizontal banding issue is likely a symptom of some change that was made to the SA tools back in v10.0. I have seen the same basic issue with RegionGroup, Eulcidean Allocation, and now ExtractMultiValuesToPoints. Seems that it's only a problem when running 'largerish' datasets. My fix idea for the day (tiling the ExtractMultiValuesToPoints job into tiles) worked - and it's faster that running the whole thing at once doe to large areas of NoData pixels in my input rasters... So I guess it's a win-win sort of. My worry is that there may be a bunch of other SA tools that may be exhibiting the 'NoData' horizontal banding issue - I just have yet to notice them.

Have you got a NIM, or a reproducible test case someone can send in?

In v10.1 SP1 I found that the 'ExtractMultiValuesToPoints' tool, when exucuted on "large" datasets sproatically assigns NULL values in weird horizontal stripped patterns - very reminicent of the issues that had been reported here with RegionGroup and other SA tools. Funny, because I had started using the ExtractMultiValuesToPoints tool instead of the Sample tool due to a bug with Sample (Sample tool bombs on large datasets - which looks to be fixed in v10.1 SP2).

My immediate work around is to.... break the analysis up into samller pieces (tiles).

Greetings,
This issue has been logged as a bug in ArcGIS 10. We will do our best to get a resolution to this issue in an upcoming service pack. Please feel free to let us know about any additional information you feel is relevant to this bug.

Here is the bug ID and synopsis:

NIM070461: The Region Group tool may produce unexpected output - the data may have an anomaly whereby there are striped bands across the output. This happens when the input is fairly large.

Thanks
Pavan

NIM070461 has been resolved at 10.0 SP 4 and 10.1.

Thanks
Pavan

BrettUWM

I have a script that I did not write but has worked fine until version 10. I do a con -> Thin - > focal max -> Region group. The tools work fine when done individually but in the model I get a 000863, 000864 ( ERROR 000864 Input raster: The input is not within the defined domain. ERROR 000863: Invalid GP data type) when I it goes to do the region group. I re-wrote just the beginning of the script and got the same error. I cannot figure why it is throwing this error. They are all grids and not overly large (18 mb). Like I mentioned, this worked fine last year on 9.3.1. Any help would be greatly appreciated. I need for my course this semester. Thanks.

Greetings,

If you can provide us with you test data, please send me an email at pyadav DOT esri DOT com so that I can reply to your email with FTP location where you can put your data.

Thank you.
Pavan

Having submitted quite a few bugs to ESRI over the years, I am frankly a bit... jaded... and am now reluctant to submit every little issue I find (of which there are many). However, if I can't find a good work around to an issue, I will still take the time to submit a formal bug fix and repro case... In this case though, I have a good work around (keep scratch raster in GRID format).

I did try to post my dataset to this thread, but it is too large (1.9 MB zipped)... A clue that helps: I can repoduce the error with my original dataset that is 8287 cols x 12513 (~30,000 unique values)rows, but when I try to repro with a smaller dataset (4913 cols x 7165 rows and 18,000 unique values - which I created in order to make some repro data small enough to post to the forum), the output result is correct (i.e. I can't reproduce the issue with the smaller dataset). Like Pavan from ESRI said, it seem to only affect "fairly large" datasets.

Hi Chris,
We have a fix in place that seems to work for the datasets we have received already, but it would be good to have more datasets to check against to be sure. I was wondering if we can get your dataset for our test. If so, you could please send me an email at pyadav DOT esri DOT com so that I can email your and provide with our FTP location where you can put your data.

Regards,
Pavan

Having submitted quite a few bugs to ESRI over the years, I am frankly a bit... jaded... and am now reluctant to submit every little issue I find (of which there are many). However, if I can't find a good work around to an issue, I will still take the time to submit a formal bug fix and repro case... In this case though, I have a good work around (keep scratch raster in GRID format).

I did try to post my dataset to this thread, but it is too large (1.9 MB zipped)... A clue that helps: I can repoduce the error with my original dataset that is 8287 cols x 12513 (~30,000 unique values)rows, but when I try to repro with a smaller dataset (4913 cols x 7165 rows and 18,000 unique values - which I created in order to make some repro data small enough to post to the forum), the output result is correct (i.e. I can't reproduce the issue with the smaller dataset). Like Pavan from ESRI said, it seem to only affect "fairly large" datasets.

Chris, did you try:

arcpy.env.scratchWorkspace = r"C:\Temp"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"C:\Temp\test.gdb\euc_aloc"
junk.save(eucAlocGrd) # copies "C:\Temp\raster3" to "C:\Temp\test.gdb\euc_aloc"

This would (in theory) do the processing using grids and then copy the output over to file gdb format (that is, how it works in 9.3 when you specify fgdb output) .

I have been told that (in 9.3, probably same as 10.0) if you set the scratch and current workspace to the same location, the temp grid (say, "raster3") will be created and then renamed (rather than copied) when the tool completes. Not quite what you were looking for, but just as fast:

arcpy.env.scratchWorkspace = r"C:\Temp"
arcpy.env.workspace = r"C:\Temp"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"outgrid"
junk.save(eucAlocGrd) # renames "raster3" to "outgrid"

If you can, *please* submit test data that reproduces these problems with file gdb rasters to ESRI support! Sometimes it is very difficult for them to fix things unless they have a dataset that reproduces the issue.

To add to this issue, in v10.0 SP3, I found that Eulcidean Allocation also suffers from the same sort of buggy behavior as RegionGroup (weird linear artifacts) when the euclidean allocation output grid is placed in FGDB format. When run entirely using grid format, the output is correct.

In my Python code I was able to rectify the problem very simply: By changing the arcpy.env.Workspace variable to a folder (thus forcing the Grid format) instead of using my FGDB as the workspace.

Somewhat related, I find it a bit frustrating in the new v10.0 map algebra that you cannot seem to force the "temp" output rasters to be a particular name, and have to rely on the .save method to, in effect, rename it.

I wish you could do this:

r"C:\Temp\euc_aloc" = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")

But it seems you have to do this:

arcpy.env.workspace = r"C:\Temp"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"C:\Temp\euc_aloc"
junk.save(eucAlocGrd)

If I do this, the euclidean allocation output (in FGDB format) gets wonkey:

arcpy.env.workspace = r"C:\Temp\test.gdb"
junk = arcpy.sa.EucAllocation(roadGrd, "", "", "", "VALUE", "", "")
eucAlocGrd = r"C:\Temp\test.gdb\euc_aloc"
junk.save(eucAlocGrd)

#roadGrd is in Grid format BTW...

FelixStracke

After trying to fix this region group issue with workarounds using scripting blablabla - my suggestion is straight from my heart: Use the good old ESRI GRID format. It is indeed a lot of faster (e.g. 1 minute using ESRI Grid as compared to ~ 15min using tif or FGDB raster) and produces a correct output!!!!!

Regards

Felix

EricRice

Hi Samantha,

We don't always automatically build raster attribute tables. Try running Build Raster Attribute Table on the result to get the table.

By default, the size of a raster attribute table is limited to 65,535 unique values. You can increase this number on the Options dialog box by clicking the Raster Attribute Table tab on the Raster tab.

Regards,
Eric

SamanthaSifleet1

Greetings,
This issue has been logged as a bug in ArcGIS 10. We will do our best to get a resolution to this issue in an upcoming service pack. Please feel free to let us know about any additional information you feel is relevant to this bug.

Here is the bug ID and synopsis:

NIM070461: The Region Group tool may produce unexpected output - the data may have an anomaly whereby there are striped bands across the output. This happens when the input is fairly large.

I have spent the day dealing with this Bug NIM070461. I am working on grids of the MW US 30mx30m, one state at a time. Today I play with Minnesota. The tool ran fine this morning out of FGDB (plain 'ole grid). I had no success running the tool from/to FGDB. I got the stripes. Now I am running the tool again outside of the FGDB. Same thing but different. The tool completes to 100% but never finishes. In the results window I see an empty output raster. I have opened that raster and seen the regions as complete, however the attribute table won't display. So no stripes but no attribute table either. I am on my second replication of this, tool started running at 15:12:00 said it completed at 15:14:00 it is now 15:28:00 and I see no signs of completion. Just killed it same thing...no stripes but no attribute table either. - Maybe this will help in fixing the bug.

Side note: Anyone able to get Patch Grids from Patch Analyst 4 up and running in 10?

I will plead ignorance here, but why is it that for me a standard 90m USGS DEM for WA State consistently takes 30 seconds to run a Fill command in GRID format while the same raster in FGDB format takes 90 seconds? This users conclusion: FGDB is still, for whatever reason, sometimes significantly slower than GRID. I am left to theorize that for some tools there is still some extra processing overhead happening when the input/output is in a FGDB (or other non-GRID) format.

However, granted, noted, and appreciated that most of the SA tools perform at par or sometimes faster when used with FGDB format (when compared with GRID format). Just curious here, but why aren't all SA the tools faster with FGDB format? Is the algorithm code somehow tailored to specific data formats?

EricRice

Hi Chris,

All Spatial Analyst tools support native I/O. We enhanced the underlying Spatial Analyst Engine to support native read/write, which every SA tool relies on.

Our development teams did substantial work to make FGDB comparable and sometimes even faster. Our efforts continue into 10.1 and beyond. The real benefit is for people who wanted to use FGDB or other formats all along. i.e. Using FGDB in 10.0 is way faster than using FGDB in 9.3.1.

Performance is gained simply by us not creating scratch GRID's behind the scenes. Also, performance is not the only benefit to native I/O. See below.

Any field name and path length limitations imposed by the ESRI GRID format are overcome. (Unless you still choose to use GRID.)
The 2.1 GB shapefile size limit is avoided by writing outputs to file GDB or SDE.
If supported by the specified input and output formats, time values in the date fields are preserved, and nulls are treated as such and no longer being converted to zeros.

Our intention with the native I/O project was not to drive people away from using GRID. If the field and file name limits never bothered you or hindered you in accomplishing something, please continue to use the GRID format. It is still definitely one of the top two fastest formats to work with.

Off topic, but...

This is true at 9.3 but not at 10.0

ESRI never said all their SA tools support native read/wriite, and of course never indicated specifically which tools still rely on native grid. In my experience, many of the v10 hydrologic tools (fill, flowdir, etc.) are still much faster in GRID format. Unless someday ESRI says - "now in ArcGIS v23.1 all the SA tools are native FGDB read/write and are now all faster when used in FGDB format than GRID format" I will keep using good ole' GRID format as my default raster format of choice.

Greetings,
This issue has been logged as a bug in ArcGIS 10. We will do our best to get a resolution to this issue in an upcoming service pack. Please feel free to let us know about any additional information you feel is relevant to this bug.

Here is the bug ID and synopsis:

NIM070461: The Region Group tool may produce unexpected output - the data may have an anomaly whereby there are striped bands across the output. This happens when the input is fairly large.

Thanks
Pavan

generally, reading writing grid format in SA tools is way faster than FGDB raster format. If you use FGDB format stuff, behind the scenes it is formats to grid, runs the SA tool algorithm, and then, writes the grid output back to FGDB raster fromat. There is a lot of time that is potentially lost in all that reformating...

This is true at 9.3 but not at 10.0. Most raster formats are now supported natively by the tools. ESRI Team Raster people have told me that they have committed themselves GRID and FGDB will probably be the fastest (and probably will have less bit-depth weirdness problems, IMHO). I have heard good things from users, some of which who have moved to FGDB for their raster processing at 10.

What's" rel="nofollow" target="_blank">http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/What_s_new_in_ArcGIS_Spatial_Analyst/00qp0000001n000000/]What's new in ArcGIS Spatial Analyst 10
- look for the heading: Native Read/Write

Hi All,

We are currently investigating this potential bug. Once, our testing is done and we have a bug ID, I will contact you back. This issue might be related to two aspects:
1. Large input raster in terms of number of rows and columns; and/or
2. Unique value range and complexity of unique pixel values distribution.

Comments on this forum indicate that if the spatial extents (coverage) of the input data is fairly large, the output will have horizontal lines. So maybe the second condition is not related to this issue.

I will greatly appreciate if you could contact Esri Support and share your test data with us. Testing with different datasets will help us narrow down the problem.