Features disappear

[DaliborKusic]

We have features stored in an Oracle 10gR2 database (SDO_GEOMETRY) with valid spatial indexes (r-tree, 2-dimensional). ArcSDE 9.3.1 with SP2 is installed. All features can be seen fine through 3rd party viewers, but when viewing through ArcMap connecting through SDE (either direct connect or through a service) some features disappear and appear depending on zoom level and panning (i.e. you pan a little to the right and features appear). What can cause such behaviour?

The same data in a different database (but still 10gR2) viewed through SDE 9.2 instance shows up fine through ArcMap.....

I've tried rebuilding indexes, even creating quad tree instead of r-tree...compared configuration options between these two SDE instances, but to no effect.... 😞

[VinceAngelo]

Does your registered rowid column map to an SE_INT32_TYPE integer (as per 'sdetable -o describe'),
and is the value returned in that column:
+ a non-zero positive value,
+ distinct (non-repeating across the result set), and
+ unique (any given value is returned by exactly one row)?

The behavior you're desribing is common when the rowid is not distinct.

- V

[DaliborKusic]

Does your registered rowid column map to an SE_INT32_TYPE integer (as per 'sdetable -o describe'),
and is the value returned in that column:
+ a non-zero positive value,
+ distinct (non-repeating across the result set), and
+ unique (any given value is returned by exactly one row)?

The behavior you're desribing is common when the rowid is not distinct.

- V

Yes, sdetable reports rowid column as SE_INT32 type and it's a primary key column in Oracle (which pretty much ensures uniqueness).
And as I said before, the same dataset in our development environment shows up fine through SDE. The database is the same (10gR2), but SDE in our dev environment is of lower version (9.2 vs 9.3.1). Although I don't think that's the problem since direct connect (bypassing the service) suffers from the same problem.

[VinceAngelo]

In the past, the existance of a primary key has prevented successful registration of a layer.
I haven't tried to register a table with a primary key more recently than 9.2.

At this point you'll probably need to work with Tech Support.

- V

[DaliborKusic]

I've pinpointed the problem. Since all of our data is 3D (has a Z value as well), there are also vertical polygons. These polygons fail the validate geometry functions of both Oracle and SDE. Oracle returns ORA-13349 (polygon intersects itself) and SDE returns -151 (SE_INVALID_OUTER_SHELL - The polygon outer shell does not completely enclose all of the donuts in a part of the shape) or -155 (SE_SELF_INTERSECTING A simple line or polygon boundary intersects itself.). This happens through both SDE 9.2 and 9.3.1. However it seems 9.2 still returns the goemetries in question.

The log file on the server sde_<servicename>.log for 9.3 contains the following lines (which appear after querying the area which contains the "invalid" polygons:

[Fri Nov 04 13:01:58 2011] [3005] [bcp] in set_invalidshape_mode
[Fri Nov 04 13:01:58 2011] [3005] [bcp] load_buffer error -151, current rowid=505072
[Fri Nov 04 13:08:12 2011] [3001] [xpmin] load_buffer error -155 on TEMP_JARK, rowid -1
[Fri Nov 04 13:08:17 2011] [3001] [xpmin] load_buffer error -155 on TEMP_JARK, rowid 504973

So apparently it does not want to read these polygons.

However the 9.2 log contains only:
[Fri Nov 04 13:23:04 2011] [4296] [zg-hac] in set_invalidshape_mode

and returns polygons as normal.

sdelayer -o feature_info on both 9.2 and 9.3.1 flags these geometries as invalid (-155 or -151).

Is there a change in behaviour from 9.2 to 9.3.1 in a way that it does not want to display invalid geometries any more? Is there a configuration keyword that would allow to loosen-up these validation algorithms? I've checked the config files of both 9.2 and 9.3.1 and they seem the same.

[DaliborKusic]

No ideas? 😞

[VinceAngelo]

There is no way around enforcement of Clementini topology rules. You can talk with
Tech Support, but vertical polygons are not supported by ArcSDE (they can only be
stored as Lines ["-e s+"]). The fact that it worked at 9.2 must have been a bug.

- V

[DaliborKusic]

There is no way around enforcement of Clementini topology rules. You can talk with
Tech Support, but vertical polygons are not supported by ArcSDE (they can only be
stored as Lines ["-e s+"]). The fact that it worked at 9.2 must have been a bug.

- V

Do you perhaps have any idea if ArcSDE 10 supports them (so I wouldn't upgrade in vain)?

[DaliborKusic]

Actually I figured something out. This link states that

At ArcSDE 9.2, a new flag was added to layers to indicate whether the layer was created using ArcSDE or registered to ArcSDE using the sdelayer â??o register command. This flag is added automatically by ArcSDE.  

If the layer was created using ArcSDE, the features are assumed to be valid since ArcSDE does not allow the creation of invalid geometries. Therefore, to help performance, layers that are flagged as having been created using ArcSDE are not verified and corrected before they are converted.  

If the layer was created outside of ArcSDE then registered to ArcSDE, the features are verified and corrected before they are converted since there is the possibility that layers created outside of ArcSDE can contain invalid geometries.

So, I've tryed creating a layer on SDE9.3.1 database using ArcCatalog (9.3). I've set the same resolution and tolerance values as I did with the original layer, the same coordinate system, etc... Afterwards, I imported data from the old layer into this new layer and - everything works!!!

So there is this flag that tells SDE not to verify geometries and simply send them to the client. But, apparently this cannot be set from the user side. I've checked the definition of the layer by using sdelayer -o describe_long and the descriptions of this new ESRI layer and my old one are the same except for the entity mask which is nac3+ for the esri one and a3+ for mine (I've tried changing mine to nac3+ but it didn't do the trick).

The EFLAG attribute in SDE.LAYERS table is of course different. I could try manually editing the EFLAG attribute based on the ESRI layer, but do not like doing things blindly. Is this binary flag documented somewhere perhaps?

[VinceAngelo]

Your quoted link just says that tables with potentially corrupt geometries will be scanned for
validity with each query (with a performance cost). It does not say that corrupt geometries
will be processed or rendered, only that an attempt at correction will be made. If both Oracle
tools and 'sdelayer -o feature_info' report a geometry as invalid, it's invalid.

The ArcSDE *API* has a function to allow clients to (reluctantly) accept invalid shapes in a stream
(as opposed to failing with a database error at the start of the first transfer buffer that would have
transferred the first invalid shape), but it then needs to test for that case and do something about
it (using this also has a performance cost). No layer flag is going to change a client to incorporate
SE_stream_invalid_shape_mode code if it is not already present.

There is a third issue here: If ArcGIS sees a CAD object, it's going to use that over the invalid
geometry. It's very likely that the "it works" state is not due to ArcSDE permitting operations on
invalid geometries (it won't), but to ArcGIS ignoring the invalid shapes in preference to CAD
entities. Unfortunately, there is no way you can generate those CAD objects externally (much
less via SQL).

Altering the undocumented (and continually evolving) EFLAGS mask column is not recommended;
you probably shouldn't do this outside of close coordination with Tech Support.

- V