ArcSDE Java Client - Different geometries with different Service Pack versions

[NormanMann]

Hi,

I am new to the Spatial world, and even newer to the ArcGIS/ESRI world.

I am having an issue with reading geometries back from an Arc database (Oracle, using the LONG RAW column for geometries).

Basically, the software I have written very coarsley, goes through a layer and for each row, gets an identifier and the geometry. This seems to work fine in most cases. This is using service pack 3 of the jsde92_sdk and jpe92_sdk jars.

However a small amount (approx 10%) of the geometry come back incorrectly. In addition, the extent from the shape is similarily incorrect. I can see in the "f" tables, what the extent should be.

Upgrading to service pack 6 (anything later then service pack 3 seems to show this behaviour), the results are the opposite. ie the correct results are incorrect, and the incorrect ones are correct.

For example.


Using Service Pack 3  :

GEOMETRY 1        -5147258.777231589, -1.5492365164206002E7
GEOMETRY 2         527566.067680127, 195430.87323492393

Using Service Pack 6 :

GEOMETRY 1       385129.7801152915, 256278.68402232602
GEOMETRY 2        4.578372853940902E7, 1.2575864698354974E8

All stored as British National Grid. It seems to be that the way the LONG_RAW column is read has changed between these service packs? I am of the understanding the ArcMAP can read the data in a consistent , correct fashion.

Can someone shed some light on how I could get consistent results using 1 service pack?

Any help greatly appreciated.

Thanks

[VinceAngelo]

If my assumption that native BASIC shapes are returned as BASIC geometry and native HIGH
shapes are returned as HIGH is correct, then it depends on how the coordref origin shifted
during upgrade as to how the incorrectly coded shapes would be interpreted. The white
paper explains the details, but the setCoordref request doesn't change the binary encoding
of the geometry, just how is is interpreted, thus doing a setCoordref(hcr) will only alter
the envelope if the new coordref is different.

It's entirely possible that a shape toward the lower-left boundary could be shifted to a valid-
seeming point in the upper-right, and escape notice.

If you scan the full table, you should be able to tell which objectid represents the first
shape added after the upgrade, since this will be the point where changeCoordref is
no longer necessary (SP6) or becomes necessary (SP3).

I don't recommend doing BASIC computation with SP3 and HIGH operations with SP6 --
Stick with SP6 and tweak the BASIC shapes into HIGH precision.

- V

[NormanMann]

Hi Vince,

Thanks again for your help.

I think I am grasping the issue a little more, but I just don't think there is support in the API to achieve what needs doing.

As you point out the SeLayer doesn't contain a secondaryCordReference. Bizarelly though, it does have a setSecondaryCordRef method. It's not ideal that I will have to hard code it, but there doesn't seem to be much choice.

A did skim the white paper which makes a little sense now, when you point out the coordinate reference is made up of different things. I can see using different service packs (at least one) element of the coordinate reference attribute is different.

In SP3

XY Cluster TOL  0.0

In SP6

XY Cluster TOL  0.0013969838638747901

Which, without understanding exactly what they mean, does point your earlier comment that each SP is using the SAME way of transforming the coordinates for every geometry.


So, to follow your workaround, I need to change the coordinate reference system on the SeShape object to see if the envelope changes.

However, changing the coordinate reference xy cluster value and setting it back on the shape (i.e.)

SeShape s = oldShape.clone();
//new Coord Ref has modified xy cluster value
s.setCoordRef(newCoordRef);
.....
s.getCordRef() ;

The last line returns the *original* xy cluster value and not the new one.

It seems lke it's being lost, or not being used properly by the API.

Is this what you expect?

I have noticed a method which may help me transform the coordinates which is.

SeShape.changeCoordRef (SeCoordinateReference , PeGeogTransformations);

Is this a suggested route (it returns a new shape)?

I am not sure what to populate PeGeogTransformations with?

If that isn't an option, it seems like the bug in 9.2 has no workaround?

Thanks again for your help.

[VinceAngelo]

Cluster tolerances don't really mean anything to ArcSDE (it just holds the value for
ArcGIS to use), so this isn't particularly useful for comparison purposes.

The real questions are:
1) What are the primary and secondary coordref parameters of the layer (most importantly,
the x/y offsets and scale)?
2) Do the coordinate values returned by getAllPoints at SP3 & SP6 correspond to the same
bytestream, but interpreted by different coordrefs?
3) Does using setCoordref with the appropriate coordref repair "broken" shapes?

The changeCoordref request accepts a null transformation when NADCON is not necessary
(in this case, there isn't any coordsys change at all, much less a datum change).

The goal is to use setCoordref to replace the primary coordref with the appropriate secondary
coordref, then use changeCoordref to recode the bytestream so it conforms to the primary
coordref. If the answers to question 2 and 3 above are "Yes", then generateLabelPoint or
getExtent should be usable as a way to determine if the bytestream is inappropriate for the
primary coordref.

- V

[NormanMann]

Vince,

Once again thanks.

1) As you have suggested, there is no way to get the secondary coordref parameters of the layer, but I will look into it again.
2)  I have no visibility of how getAllPoints works under the hood, I suspect the bytesteam is hidden deep within the API.
3) I tried to explain in the previous post, that setCoordRef on SeShape DOES NOT change the coordref  (when a subsequenct SeShape.getCoordRef() is executed, let alone repair the shapes.

All said, are you saying that I need to (using, probably a hard coded secondary coord ref)

SeShape s = new SeShape().
s.setCoordRef(secondaryCoordRef);
SeShape s1 = s.changeCordRef(secondaryCoordRef, null);

Then I would expect to see the points possibluy repaired?

I will try that and let you know....

Thanks

EDIT

When you refer to x/y offsets and scale what fields do those relate to on either

http://edndoc.esri.com/arcsde/9.3/api/japi/docs/com/esri/sde/sdk/client/SeCoordinateReference.html

or

http://edndoc.esri.com/arcsde/9.3/api/japi/docs/com/esri/sde/sdk/geom/SeCoordRef.html

[VinceAngelo]

Every layer has an entry in the SDE.LAYERS table. Every LAYERS row has a mandatory SRID and
an optional SECONDARY_SRID. These are are foreign keys into the SDE.SPATIAL_REFERENCES
table, where the FALSEX, FALSEY, and XYUNITS columns store the offsets and scale. What
are the values stored in your instance for the layer you are accessing?

The bytestream of the shape is extracted into an integer array which is easily accessible to
the SeShape object. setCoordref changes how that array is interpreted without changing the
array (at least that's how it worked when I last used it). changeCoordref modifies the array.

In a worst case, you can do the math yourself:

 
newX = (floor((oldX - primFalseX) / primXYUnits + 0.5) * secXYUnits) + secFalseX;
newY = (floor((oldY - primFalseY) / primXYUnits + 0.5) * secXYUnits) + secFalseY;

- V

[NormanMann]

I can see those values in the database, but not in the API....

I have made some progress using setCordRef, and changeCordRef.
It does seem to bring out the coordinates as sensible values.

However, I still am unable to determine which ones I need to change.

You mentioned

If the answers to question 2 and 3 above are "Yes", then generateLabelPoint or
getExtent should be usable as a way to determine if the bytestream is inappropriate for the
primary coordref.
However, given a Shape s1 (which all the shapes in a given layer will have the SAME coord ref by default, as the API doesn't seem to take into account the secondary one - ever).
Setting the SECONDARY coordref on s1 (irrespective if it's "correct" or not) will ALWAYS change the extent?

This is the behaviour I am seeing.

I need a way of checking something in the SeShape to see if I need to do changeCordRef.

Right now, I am having to do it on ALL SeShapes. The ones which don't need changing seems to result in an out of range exception. I don't think , as you have already indicated that this is reliable?

Can you explain a bit more about how I an use getExtent to determine if the bytestream is inappropriate? Do I do this before I set the secondary coord ref? If so, what am I looking for? As I tried to explain above, setting the coordref to the SECONDARY one, will ALWAYS change the extent.

Thanks again!

[VinceAngelo]

Without knowing the SRID constants and layer envelope, I can't tell you how to detect
incorrectly decoded shapes. Also of use would be the output of 'sdelayer -o stats', to
report the sizes involved.

You should also start an incident with Tech Support, since the problem may exist in
versions which are still under active maintenance.

- V

[NormanMann]

Please see below detailed descriptions of primary and secondary srids, and envelope for the layer.

<table border='1'>
<tr>

<th>SRID</th>
<th>DESCRIPTION</th>
<th>AUTH_NAME</th>
<th>AUTH_SRID</th>
<th>FALSEX</th>
<th>FALSEY</th>
<th>ZUNITS</th>
<th>FALSEM</th>
<th>MUNITS</th>
<th>SRTEXT</th>
<th>OBJECT_FLAGS</th>
<th>XYCLUSTER_TOL</th>
<th>ZCLUSTER_TOL</th>
<th>MCLUSTER_TOL</th>

</tr>

<tr>
<td>26</td>
<td></td>
<td></td>
<td>0</td>
<td>-200000.000698492</td>
<td>-.000698491931937391</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>0</td>
<td></td>
<td></td>
<td></td>
</tr>                                                 
                                                                        

      
<tr>
<td>41</td>
<td></td>
<td></td>
<td></td>
<td>-5220399.99983331</td>
<td>-15524399.9997961</td>
<td>4</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>1</td>
<td>.00139698386387479</td>
<td>2</td>
<td>2</td>
</tr>                                 
                                                                       

</table>


Is that sufficient?

[NormanMann]

<HTML>
<table border='1'>
<tr>

<th>SRID</th>
<th>DESCRIPTION</th>
<th>AUTH_NAME</th>
<th>AUTH_SRID</th>
<th>FALSEX</th>
<th>FALSEY</th>
<th>ZUNITS</th>
<th>FALSEM</th>
<th>MUNITS</th>
<th>SRTEXT</th>
<th>OBJECT_FLAGS</th>
<th>XYCLUSTER_TOL</th>
<th>ZCLUSTER_TOL</th>
<th>MCLUSTER_TOL</th>

</tr>

<tr>
<td>26</td>
<td></td>
<td></td>
<td>0</td>
<td>-200000.000698492</td>
<td>-.000698491931937391</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>0</td>
<td></td>
<td></td>
<td></td>
</tr>                                                 
                                                                        

      
<tr>
<td>41</td>
<td></td>
<td></td>
<td></td>
<td>-5220399.99983331</td>
<td>-15524399.9997961</td>
<td>4</td>
<td>0</td>
<td>1</td>
<td>PROJCS["British_National_Grid",GEOGCS["GCS_OSGB_1936",DATUM["D_OSGB_

1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",

0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"

],PARAMETER["False_Easting",400000.0],PARAMETER["False_Northing",-100000

.0],PARAMETER["Central_Meridian",-2.0],PARAMETER["Scale_Factor",0.999601

272],PARAMETER["Latitude_Of_Origin",49.0],UNIT["Meter",1.0]]</td>
<td>1</td>
<td>.00139698386387479</td>
<td>2</td>
<td>2</td>
</tr>                                 
                                                                       

</table>
</HTML>

[NormanMann]

Please find attached a JPG screenshot of the srid values, hope this is adequate?
Not sure of a suitable way of displaying this data on a forum.

Unfortunately I don't have console access to the server to run the commands.

Also , see below the minx, miny, maxx and maxy for the layer.

-485237.71 -7315167.4 5069344.18 1330454.86

Thanks again for your help. I have been waiting for a while for my companys account number so I can raise a case with technical support. Although, as you suggest a fix probably isn't going to happen in the version we use?