topic Re: ArcSDE Java Client - Different geometries with different Service Pack versions in Data Management Questions

ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Wed, 13 Jul 2011 11:38:32 GMT

Hi,

I am new to the Spatial world, and even newer to the ArcGIS/ESRI world.

I am having an issue with reading geometries back from an Arc database (Oracle, using the LONG RAW column for geometries).

Basically, the software I have written very coarsley, goes through a layer and for each row, gets an identifier and the geometry. This seems to work fine in most cases. This is using service pack 3 of the jsde92_sdk and jpe92_sdk jars.

However a small amount (approx 10%) of the geometry come back incorrectly. In addition, the extent from the shape is similarily incorrect. I can see in the "f" tables, what the extent should be.

Upgrading to service pack 6 (anything later then service pack 3 seems to show this behaviour), the results are the opposite. ie the correct results are incorrect, and the incorrect ones are correct.

For example.

Using Service Pack 3 :

GEOMETRY 1        -5147258.777231589, -1.5492365164206002E7
GEOMETRY 2         527566.067680127, 195430.87323492393

Using Service Pack 6 :

GEOMETRY 1       385129.7801152915, 256278.68402232602
GEOMETRY 2        4.578372853940902E7, 1.2575864698354974E8

All stored as British National Grid. It seems to be that the way the LONG_RAW column is read has changed between these service packs? I am of the understanding the ArcMAP can read the data in a consistent , correct fashion.

Can someone shed some light on how I could get consistent results using 1 service pack?

Any help greatly appreciated.

Thanks

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Wed, 13 Jul 2011 12:28:16 GMT

ArcSDE clients don't read geometries from databases -- they read geometries from ArcSDE servers.

What version of ArcSDE is installed in the database?

What version of ArcSDE is installed in the server?

Your client API should always be equal to or greater than the server.

Your server install should always be equal to the database.

If you use Direct Connect, the client is the server, and should be equal to the database.

Given that the last update to 9.2 occurred two years ago (SP3 goes back nearly four years),
if you're still using 9.2, you really should be running the terminal release (SP6 + Connection
Performance + Memory Allocation + Compress/Concurrent Editors) across the board.

Taking a step backwards, what version of Oracle are you using? [A.B.C.D notation, e.g., "10.2.0.3"]
If using Direct Connect, what version of Oracle is on your client?

The fact that 'C' API applications like ArcGIS read the data successfully indicates a possible
problem with your application (and/or the underlying Java SDK). What are the coordinate
reference parameters of the layer being queried? Does your code change the coordinate
reference in any way? Was the layer upgraded to HIGH precision from a previous BASIC
precision layer at 9.1 or earlier (is there an entry in the secondary coordref property)?

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Wed, 13 Jul 2011 12:56:28 GMT

Vince,

thanks for your reply.

To answer some of your questions :

The version of Oracle we are using 10.2.0.3.0.

I am happy to use which ever service pack version of the SDK which works.

In respect to a problem with the application, SESHape.getAllCoords returns the double values representing the coordinates. From an application level, there is nothing I can do to influence this?

The code doesn't change the coordinate reference parameters in anyway.
The coordinate reference parameters (I think you mean this) are

Coord ref system PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]

There is a secondary srid set in the sde.layers table for the layer in question. This is "26". Not sure what that represents.

I do also believe that the layer was upgraded to HIGH precision at some point in the past.

I do feel there is an issue with the underlying SDK, as the same set of code (using different service pack versions of the JAR files) produce different results. Service pack 3 gets SET A correct and SET B incorrect, and service pack 4 and above gets SET B correct and SET A incorrect.

I am waiting the versions of the database version of ARCSDE and the server version of ARCSDE. I will update when I have that, but if you could provide any more help in the meantime, that would be great.

Thanks again.
EDIT: Just to add I am NOT using direct connection.

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Wed, 13 Jul 2011 13:42:22 GMT

There's actually a ton of things you can do to influence how getAllCoords parses the
geometry, few of them good (and all of them have to do with coordinate reference
override).

I seem to remember a bug where HYBRID coordref (BASIC upgraded to HIGH) layers
were not interpreted correctly by the Java API. This would produce the behavior
you're seeing (where the the HIGH precision shapes are parsed with the BASIC
coordref at SP3, and vice versa at SP6).

The easiest solution would be to use a single coordref for all shapes, but the route to
get there depends on whether the layer is versioned, participates in complex topologies,
or has CAD objects associated with any rows. If the layer is a simple feature class, you
can backup the table, then use 'sdeexport' and then 'sdeimport' with the 'init' option
(all new shapes are always created in the HIGH coordref, and the 'init' option truncates
the table and reloads all features); Tech Support would be able to help steer you if this
is other than a simple feature class.

Client side correction is likely to be clumsy, since you'd have to:
+ Copy the shape to a temporary object
+ Get the envelope
+ Override the coordref on the temp object with the HIGH coordref (via setCoordref)
+ Get the envelope again
+ Compare the envelopes
+ If changed, use the changeCoordref request to transform the temp object into the
HIGH coordref in the original shape for use in the application.
The only good news is that the changeCoordref will just be a transform, not a true
"project", so it won't be as expensive, but it will require you to upgrade the 90%
of BASIC shapes on the fly for use (you could go the other direction, but then the
shapes are losing precision [which isn't always a bad thing]).

You might also see if the 9.3 or 9.3.1 API fixed this issue.

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Wed, 13 Jul 2011 14:07:30 GMT

Hi VInce,

Thanks for that.
I think the client side fix is the only realistic option for me right now.

Is there anyway of easily identifying using metadata, high precision/low precistion shapes?
Are you implying that the coordinate reference system will be different?
It doesn't seem to be. Is that stored in the database somehow against each row of data?

Thankss again.
EDIT - I did try 9.3 versions and they show the same behaviour as the later 9.2 service packs/

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Wed, 13 Jul 2011 14:27:08 GMT

HYBRID coordinate references were supposed to be transparent (geometries always
interpreted as HIGH precision, even if they were originally stored as BASIC). It is for
this reason that the layer upgrade had constraints on what parameters were used
(the HIGH precision coordinate grid had to align with the BASIC one [evenly divisible]).
This obviated the need to mark shapes as being BASIC or HIGH, and eliminated the
need for such functions in the API. It was a clever design, though it appears the Java
implementation didn't take it into account. 😞

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Wed, 13 Jul 2011 15:38:56 GMT

I took another look at the Java API documentation and SeLayer does NOT include a
getSecondaryCoordref request the way the 'C' API does. It's possible that the request
exists and the doc is just broken, but given the nature of Javadocs, this seems unlikely
(worse, it's not in the 10.0sp1 documentation either). This will make it more difficult to
obtain the BASIC coordref without hard-coding it into your application.

Note that the coordinate reference is *not* the coordinate system which you referred to
earlier. The coordSYS is part of the coordREF, but the coordref also incorporates:
+ The PRECISION
+ X/Y offsets and scale
+ Z offset and scale
+ M offset and scale
+ An authenticated SRID
+ A description
At 9.2 and higher, X/Y, Z, and M also have optional cluster tolerances, and 9.3 introduced
vertical coordinate systems (vertcs). There's an entire White Paper on coordinate management
which should be mandatory reading for anyone who wants to use the 'C' or Java API; I strongly
recommend it to you.

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Wed, 13 Jul 2011 15:43:22 GMT

So the client side workaround would require us to determine if EVERY geometry was "correct"?
I guess I could check that the numbers returned were sensible in terms of a bounding box.

Out of interest, how do the C APIs detemine in what wat they should be read?

I think SeShape.getCoordRef returns a SeCoordinateReference. An SeCoordinateReference has a gePrecision method.
Would this be any use in determing HIGH/BASIC geometries? Or, do I really need the secondary coordinate reference system to determine this?

Will have a read of that white paper tomorrow, but just to summarise it seems to be that there isn't a work around?
The bug in the java api (HYBRID bug) seems never to have been fixed, even in 9.3?

I am happy to use different versions of the service packs (yes, I am that desperate) but really need a way to determine which rows should be read with which version.

Thanks again.

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Wed, 13 Jul 2011 15:49:11 GMT

It's not enough to check if the bounding box is sensible -- you must check if it
is the *same*.

This is not an issue with the 'C' API because all shapes from HYBRID layers
appear to be HIGH precision.

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Wed, 13 Jul 2011 16:17:20 GMT

I meant, if the geomerty for a given row, is within the bounding box it should be.

From the example above
Using Service Pack 3 :

GEOMETRY 1 -5147258.777231589, -1.5492365164206002E7
GEOMETRY 2 527566.067680127, 195430.87323492393

Using Service Pack 6 :

GEOMETRY 1 385129.7801152915, 256278.68402232602
GEOMETRY 2 4.578372853940902E7, 1.2575864698354974E8

It's pretty clear that GEOMETRY 1 in service pack 3 isn't valid in terms of a bounding box, and GEOMETRY 2 in service pack 6 isn't valid in terms of the bounding box.

Couldn't I just discard the ones which don't work with SP3 and read them with SP6? I know this is "dirty".

But what are the options (I don't want to/can't change the data in the database)

Thanks again!

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Wed, 13 Jul 2011 17:33:17 GMT

If my assumption that native BASIC shapes are returned as BASIC geometry and native HIGH
shapes are returned as HIGH is correct, then it depends on how the coordref origin shifted
during upgrade as to how the incorrectly coded shapes would be interpreted. The white
paper explains the details, but the setCoordref request doesn't change the binary encoding
of the geometry, just how is is interpreted, thus doing a setCoordref(hcr) will only alter
the envelope if the new coordref is different.

It's entirely possible that a shape toward the lower-left boundary could be shifted to a valid-
seeming point in the upper-right, and escape notice.

If you scan the full table, you should be able to tell which objectid represents the first
shape added after the upgrade, since this will be the point where changeCoordref is
no longer necessary (SP6) or becomes necessary (SP3).

I don't recommend doing BASIC computation with SP3 and HIGH operations with SP6 --
Stick with SP6 and tweak the BASIC shapes into HIGH precision.

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Thu, 14 Jul 2011 10:23:27 GMT

Hi Vince,

Thanks again for your help.

I think I am grasping the issue a little more, but I just don't think there is support in the API to achieve what needs doing.

As you point out the SeLayer doesn't contain a secondaryCordReference. Bizarelly though, it does have a setSecondaryCordRef method. It's not ideal that I will have to hard code it, but there doesn't seem to be much choice.

A did skim the white paper which makes a little sense now, when you point out the coordinate reference is made up of different things. I can see using different service packs (at least one) element of the coordinate reference attribute is different.

In SP3

XY Cluster TOL 0.0

In SP6

XY Cluster TOL 0.0013969838638747901

Which, without understanding exactly what they mean, does point your earlier comment that each SP is using the SAME way of transforming the coordinates for every geometry.

So, to follow your workaround, I need to change the coordinate reference system on the SeShape object to see if the envelope changes.

However, changing the coordinate reference xy cluster value and setting it back on the shape (i.e.)

SeShape s = oldShape.clone();
//new Coord Ref has modified xy cluster value
s.setCoordRef(newCoordRef);
.....
s.getCordRef() ;

The last line returns the *original* xy cluster value and not the new one.

It seems lke it's being lost, or not being used properly by the API.

Is this what you expect?

I have noticed a method which may help me transform the coordinates which is.

SeShape.changeCoordRef (SeCoordinateReference , PeGeogTransformations);

Is this a suggested route (it returns a new shape)?

I am not sure what to populate PeGeogTransformations with?

If that isn't an option, it seems like the bug in 9.2 has no workaround?

Thanks again for your help.

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Thu, 14 Jul 2011 11:00:52 GMT

Cluster tolerances don't really mean anything to ArcSDE (it just holds the value for
ArcGIS to use), so this isn't particularly useful for comparison purposes.

The real questions are:
1) What are the primary and secondary coordref parameters of the layer (most importantly,
the x/y offsets and scale)?
2) Do the coordinate values returned by getAllPoints at SP3 & SP6 correspond to the same
bytestream, but interpreted by different coordrefs?
3) Does using setCoordref with the appropriate coordref repair "broken" shapes?

The changeCoordref request accepts a null transformation when NADCON is not necessary
(in this case, there isn't any coordsys change at all, much less a datum change).

The goal is to use setCoordref to replace the primary coordref with the appropriate secondary
coordref, then use changeCoordref to recode the bytestream so it conforms to the primary
coordref. If the answers to question 2 and 3 above are "Yes", then generateLabelPoint or
getExtent should be usable as a way to determine if the bytestream is inappropriate for the
primary coordref.

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Thu, 14 Jul 2011 11:34:58 GMT

Vince,

Once again thanks.

1) As you have suggested, there is no way to get the secondary coordref parameters of the layer, but I will look into it again.
2) I have no visibility of how getAllPoints works under the hood, I suspect the bytesteam is hidden deep within the API.
3) I tried to explain in the previous post, that setCoordRef on SeShape DOES NOT change the coordref (when a subsequenct SeShape.getCoordRef() is executed, let alone repair the shapes.

All said, are you saying that I need to (using, probably a hard coded secondary coord ref)

SeShape s = new SeShape().
s.setCoordRef(secondaryCoordRef);
SeShape s1 = s.changeCordRef(secondaryCoordRef, null);

Then I would expect to see the points possibluy repaired?

I will try that and let you know....

Thanks

EDIT

When you refer to x/y offsets and scale what fields do those relate to on either

http://edndoc.esri.com/arcsde/9.3/api/japi/docs/com/esri/sde/sdk/client/SeCoordinateReference.html

or

http://edndoc.esri.com/arcsde/9.3/api/japi/docs/com/esri/sde/sdk/geom/SeCoordRef.html

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Sat, 11 Dec 2021 22:42:34 GMT

Every layer has an entry in the SDE.LAYERS table. Every LAYERS row has a mandatory SRID and
an optional SECONDARY_SRID. These are are foreign keys into the SDE.SPATIAL_REFERENCES
table, where the FALSEX, FALSEY, and XYUNITS columns store the offsets and scale. What
are the values stored in your instance for the layer you are accessing?

The bytestream of the shape is extracted into an integer array which is easily accessible to
the SeShape object. setCoordref changes how that array is interpreted without changing the
array (at least that's how it worked when I last used it). changeCoordref modifies the array.

In a worst case, you can do the math yourself:

 
newX = (floor((oldX - primFalseX) / primXYUnits + 0.5) * secXYUnits) + secFalseX;
newY = (floor((oldY - primFalseY) / primXYUnits + 0.5) * secXYUnits) + secFalseY;

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Thu, 14 Jul 2011 14:10:10 GMT

I can see those values in the database, but not in the API....

I have made some progress using setCordRef, and changeCordRef.
It does seem to bring out the coordinates as sensible values.

However, I still am unable to determine which ones I need to change.

You mentioned

If the answers to question 2 and 3 above are "Yes", then generateLabelPoint or
getExtent should be usable as a way to determine if the bytestream is inappropriate for the
primary coordref.
However, given a Shape s1 (which all the shapes in a given layer will have the SAME coord ref by default, as the API doesn't seem to take into account the secondary one - ever).
Setting the SECONDARY coordref on s1 (irrespective if it's "correct" or not) will ALWAYS change the extent?

This is the behaviour I am seeing.

I need a way of checking something in the SeShape to see if I need to do changeCordRef.

Right now, I am having to do it on ALL SeShapes. The ones which don't need changing seems to result in an out of range exception. I don't think , as you have already indicated that this is reliable?

Can you explain a bit more about how I an use getExtent to determine if the bytestream is inappropriate? Do I do this before I set the secondary coord ref? If so, what am I looking for? As I tried to explain above, setting the coordref to the SECONDARY one, will ALWAYS change the extent.

Thanks again!

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

VinceAngelo — Thu, 14 Jul 2011 16:09:40 GMT

Without knowing the SRID constants and layer envelope, I can't tell you how to detect
incorrectly decoded shapes. Also of use would be the output of 'sdelayer -o stats', to
report the sizes involved.

You should also start an incident with Tech Support, since the problem may exist in
versions which are still under active maintenance.

- V

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Fri, 15 Jul 2011 09:46:39 GMT

Please see below detailed descriptions of primary and secondary srids, and envelope for the layer.

<table border='1'>
<tr>

<th>SRID</th>
<th>DESCRIPTION</th>
<th>AUTH_NAME</th>
<th>AUTH_SRID</th>
<th>FALSEX</th>
<th>FALSEY</th>
<th>ZUNITS</th>
<th>FALSEM</th>
<th>MUNITS</th>
<th>SRTEXT</th>
<th>OBJECT_FLAGS</th>
<th>XYCLUSTER_TOL</th>
<th>ZCLUSTER_TOL</th>
<th>MCLUSTER_TOL</th>

</tr>

<tr>
<td>26</td>
<td></td>
<td></td>
<td>0</td>
<td>-200000.000698492</td>
<td>-.000698491931937391</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>0</td>
<td></td>
<td></td>
<td></td>
</tr>



<tr>
<td>41</td>
<td></td>
<td></td>
<td></td>
<td>-5220399.99983331</td>
<td>-15524399.9997961</td>
<td>4</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>1</td>
<td>.00139698386387479</td>
<td>2</td>
<td>2</td>
</tr>


</table>

Is that sufficient?

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Fri, 15 Jul 2011 09:48:47 GMT

<HTML>
<table border='1'>
<tr>

<th>SRID</th>
<th>DESCRIPTION</th>
<th>AUTH_NAME</th>
<th>AUTH_SRID</th>
<th>FALSEX</th>
<th>FALSEY</th>
<th>ZUNITS</th>
<th>FALSEM</th>
<th>MUNITS</th>
<th>SRTEXT</th>
<th>OBJECT_FLAGS</th>
<th>XYCLUSTER_TOL</th>
<th>ZCLUSTER_TOL</th>
<th>MCLUSTER_TOL</th>

</tr>

<tr>
<td>26</td>
<td></td>
<td></td>
<td>0</td>
<td>-200000.000698492</td>
<td>-.000698491931937391</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>0</td>
<td></td>
<td></td>
<td></td>
</tr>



<tr>
<td>41</td>
<td></td>
<td></td>
<td></td>
<td>-5220399.99983331</td>
<td>-15524399.9997961</td>
<td>4</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>1</td>
<td>.00139698386387479</td>
<td>2</td>
<td>2</td>
</tr>


</table>
</HTML>

Re: ArcSDE Java Client - Different geometries with different Service Pack versions

NormanMann — Fri, 15 Jul 2011 09:56:32 GMT

Please find attached a JPG screenshot of the srid values, hope this is adequate?
Not sure of a suitable way of displaying this data on a forum.

Unfortunately I don't have console access to the server to run the commands.

Also , see below the minx, miny, maxx and maxy for the layer.

-485237.71 -7315167.4 5069344.18 1330454.86

Thanks again for your help. I have been waiting for a while for my companys account number so I can raise a case with technical support. Although, as you suggest a fix probably isn't going to happen in the version we use?