topic Re: Features disappear in Data Management Questions

Features disappear

DaliborKusic — Wed, 02 Nov 2011 11:48:24 GMT

We have features stored in an Oracle 10gR2 database (SDO_GEOMETRY) with valid spatial indexes (r-tree, 2-dimensional). ArcSDE 9.3.1 with SP2 is installed. All features can be seen fine through 3rd party viewers, but when viewing through ArcMap connecting through SDE (either direct connect or through a service) some features disappear and appear depending on zoom level and panning (i.e. you pan a little to the right and features appear). What can cause such behaviour?

The same data in a different database (but still 10gR2) viewed through SDE 9.2 instance shows up fine through ArcMap.....

I've tried rebuilding indexes, even creating quad tree instead of r-tree...compared configuration options between these two SDE instances, but to no effect.... 😞

Re: Features disappear

VinceAngelo — Wed, 02 Nov 2011 13:11:14 GMT

Does your registered rowid column map to an SE_INT32_TYPE integer (as per 'sdetable -o describe'),
and is the value returned in that column:
+ a non-zero positive value,
+ distinct (non-repeating across the result set), and
+ unique (any given value is returned by exactly one row)?

The behavior you're desribing is common when the rowid is not distinct.

- V

Re: Features disappear

DaliborKusic — Thu, 03 Nov 2011 05:59:40 GMT

Does your registered rowid column map to an SE_INT32_TYPE integer (as per 'sdetable -o describe'),
and is the value returned in that column:
+ a non-zero positive value,
+ distinct (non-repeating across the result set), and
+ unique (any given value is returned by exactly one row)?

The behavior you're desribing is common when the rowid is not distinct.

- V

Yes, sdetable reports rowid column as SE_INT32 type and it's a primary key column in Oracle (which pretty much ensures uniqueness).
And as I said before, the same dataset in our development environment shows up fine through SDE. The database is the same (10gR2), but SDE in our dev environment is of lower version (9.2 vs 9.3.1). Although I don't think that's the problem since direct connect (bypassing the service) suffers from the same problem.

Re: Features disappear

VinceAngelo — Thu, 03 Nov 2011 10:13:08 GMT

In the past, the existance of a primary key has prevented successful registration of a layer.
I haven't tried to register a table with a primary key more recently than 9.2.

At this point you'll probably need to work with Tech Support.

- V

Re: Features disappear

DaliborKusic — Fri, 04 Nov 2011 11:47:25 GMT

I've pinpointed the problem. Since all of our data is 3D (has a Z value as well), there are also vertical polygons. These polygons fail the validate geometry functions of both Oracle and SDE. Oracle returns ORA-13349 (polygon intersects itself) and SDE returns -151 (SE_INVALID_OUTER_SHELL - The polygon outer shell does not completely enclose all of the donuts in a part of the shape) or -155 (SE_SELF_INTERSECTING A simple line or polygon boundary intersects itself.). This happens through both SDE 9.2 and 9.3.1. However it seems 9.2 still returns the goemetries in question.

The log file on the server sde_<servicename>.log for 9.3 contains the following lines (which appear after querying the area which contains the "invalid" polygons:

[Fri Nov 04 13:01:58 2011] [3005] [bcp] in set_invalidshape_mode
[Fri Nov 04 13:01:58 2011] [3005] [bcp] load_buffer error -151, current rowid=505072
[Fri Nov 04 13:08:12 2011] [3001] [xpmin] load_buffer error -155 on TEMP_JARK, rowid -1
[Fri Nov 04 13:08:17 2011] [3001] [xpmin] load_buffer error -155 on TEMP_JARK, rowid 504973

So apparently it does not want to read these polygons.

However the 9.2 log contains only:
[Fri Nov 04 13:23:04 2011] [4296] [zg-hac] in set_invalidshape_mode

and returns polygons as normal.

sdelayer -o feature_info on both 9.2 and 9.3.1 flags these geometries as invalid (-155 or -151).

Is there a change in behaviour from 9.2 to 9.3.1 in a way that it does not want to display invalid geometries any more? Is there a configuration keyword that would allow to loosen-up these validation algorithms? I've checked the config files of both 9.2 and 9.3.1 and they seem the same.

Re: Features disappear

DaliborKusic — Fri, 11 Nov 2011 05:30:28 GMT

No ideas? 😞

Re: Features disappear

VinceAngelo — Fri, 11 Nov 2011 09:37:50 GMT

There is no way around enforcement of Clementini topology rules. You can talk with
Tech Support, but vertical polygons are not supported by ArcSDE (they can only be
stored as Lines ["-e s+"]). The fact that it worked at 9.2 must have been a bug.

- V

Re: Features disappear

DaliborKusic — Sat, 12 Nov 2011 12:39:34 GMT

There is no way around enforcement of Clementini topology rules. You can talk with
Tech Support, but vertical polygons are not supported by ArcSDE (they can only be
stored as Lines ["-e s+"]). The fact that it worked at 9.2 must have been a bug.

- V

Do you perhaps have any idea if ArcSDE 10 supports them (so I wouldn't upgrade in vain)?

Re: Features disappear

DaliborKusic — Sun, 13 Nov 2011 12:39:52 GMT

Actually I figured something out. This link states that

At ArcSDE 9.2, a new flag was added to layers to indicate whether the layer was created using ArcSDE or registered to ArcSDE using the sdelayer â??o register command. This flag is added automatically by ArcSDE.

If the layer was created using ArcSDE, the features are assumed to be valid since ArcSDE does not allow the creation of invalid geometries. Therefore, to help performance, layers that are flagged as having been created using ArcSDE are not verified and corrected before they are converted.

If the layer was created outside of ArcSDE then registered to ArcSDE, the features are verified and corrected before they are converted since there is the possibility that layers created outside of ArcSDE can contain invalid geometries.

So, I've tryed creating a layer on SDE9.3.1 database using ArcCatalog (9.3). I've set the same resolution and tolerance values as I did with the original layer, the same coordinate system, etc... Afterwards, I imported data from the old layer into this new layer and - everything works!!!

So there is this flag that tells SDE not to verify geometries and simply send them to the client. But, apparently this cannot be set from the user side. I've checked the definition of the layer by using sdelayer -o describe_long and the descriptions of this new ESRI layer and my old one are the same except for the entity mask which is nac3+ for the esri one and a3+ for mine (I've tried changing mine to nac3+ but it didn't do the trick).

The EFLAG attribute in SDE.LAYERS table is of course different. I could try manually editing the EFLAG attribute based on the ESRI layer, but do not like doing things blindly. Is this binary flag documented somewhere perhaps?

Re: Features disappear

VinceAngelo — Sun, 13 Nov 2011 13:58:08 GMT

Your quoted link just says that tables with potentially corrupt geometries will be scanned for
validity with each query (with a performance cost). It does not say that corrupt geometries
will be processed or rendered, only that an attempt at correction will be made. If both Oracle
tools and 'sdelayer -o feature_info' report a geometry as invalid, it's invalid.

The ArcSDE *API* has a function to allow clients to (reluctantly) accept invalid shapes in a stream
(as opposed to failing with a database error at the start of the first transfer buffer that would have
transferred the first invalid shape), but it then needs to test for that case and do something about
it (using this also has a performance cost). No layer flag is going to change a client to incorporate
SE_stream_invalid_shape_mode code if it is not already present.

There is a third issue here: If ArcGIS sees a CAD object, it's going to use that over the invalid
geometry. It's very likely that the "it works" state is not due to ArcSDE permitting operations on
invalid geometries (it won't), but to ArcGIS ignoring the invalid shapes in preference to CAD
entities. Unfortunately, there is no way you can generate those CAD objects externally (much
less via SQL).

Altering the undocumented (and continually evolving) EFLAGS mask column is not recommended;
you probably shouldn't do this outside of close coordination with Tech Support.

- V

Re: Features disappear

DaliborKusic — Sun, 13 Nov 2011 20:14:40 GMT

Your quoted link just says that tables with potentially corrupt geometries will be scanned for
validity with each query (with a performance cost). It does not say that corrupt geometries
will be processed or rendered, only that an attempt at correction will be made. If both Oracle
tools and 'sdelayer -o feature_info' report a geometry as invalid, it's invalid.

I understood that it says if the flag in question is not set (i.e. layer is created using sdelayer -o register), ArcSDE will verify the geometries prior to converting them and sending them to client. If they are "invalid" (i.e. vertical polygons) they will not even be converted and nothing will be displayed. However if the flag is set, the conversion proceeds without verification. This conversion and subsequent rendering may or may not succeed, but as it turns out it does succeed since the "invalidity" of the polygons is obviously not severe.

There is a third issue here: If ArcGIS sees a CAD object, it's going to use that over the invalid
geometry. It's very likely that the "it works" state is not due to ArcSDE permitting operations on
invalid geometries (it won't), but to ArcGIS ignoring the invalid shapes in preference to CAD
entities. Unfortunately, there is no way you can generate those CAD objects externally (much
less via SQL).

Didn't quite understand this part with CAD entities. Are you refering to the c in entity mask? I don't think this makes much difference since there are no CAD objects in the table and adding or substracting this entity type from the layer definition makes no difference (registered layers always fail).

Altering the undocumented (and continually evolving) EFLAGS mask column is not recommended;
you probably shouldn't do this outside of close coordination with Tech Support.

Well yes I'm hopeful this will be done with ESRI's assistance, but since I'm not a customer myself, but rather my client is, it will be difficult to coordinate this. However it is frustrating that layers created through ArcCatalog and by using sdelayer command behave differently without any possibility of altering this behaviour (i.e. switch with sdelayer -o register command).
Just because both Oracle and ESRI (even though Oracle in a far lesser degree than ArcGIS at v9.3.1) can't handle the 3rd dimension correctly it does not mean the geometry is invalid.

Re: Features disappear

DaliborKusic — Sun, 13 Nov 2011 21:57:04 GMT

Another update.... It seems the "problem" wasn't with the EFLAGS, but with the OBJECT_FLAGS attribute in the SDE.TABLE_REGISTRY table. If I take my old table (the one that originally did not work), do not change anything, but only copy this attribute from the table that was created using ArcCatalog and overwriting with it the OBJECT_FLAGS attribute of my old table, the geometries display in ArcMap regardless of their "validity" (also no warnings or errors in SDE log files). Unfortunately I have no idea what this OBJECT_FLAGS is besides it is another binary attribute. I simply copied a value of 147463 over my old value of 133. This is not a perfect solution, but it works for now. Will try to reach ESRI's tech support on this. Anybody perhaps knows the structure of this attribute?

Re: Features disappear

DaliborKusic — Mon, 14 Nov 2011 07:58:04 GMT

Hopefully the last update:
It appears the bit 17 of the OBJECT_FLAGS attribute has something to do with SDE verifications. After setting this bit, layers show "normally" (no anomalies so far) in ArcMap and no errors are logged in the SDE log files.
So..lets hope this little bit of reverse engineering holds up through stress tests 🙂