Converting from SDE Views to Regular SQL Views - Any Gotchas?

7927
19
02-21-2014 12:59 PM
RandyKreuziger
Occasional Contributor III
We are upgrading from SQL 2008 / ArcSDE 10.0 to SQL 2012 / ArcSDE 10.1.  Since we are moving to a new server at the same time I'd like to recreate the SDE Views as SQL Views.

Are there any gotchas I need to look out for?

For MXDs can we simply change the SDE database through the ArcCatalog Set Data Sources function?
19 Replies
VinceAngelo
Esri Esteemed Contributor
The 'sdelayer -o register' command is used to register tables or views in native storage
(or ST_GEOMETRY, where available) with ArcSDE (note that views cannot be registered with
the geodatabase, just with ArcSDE, which is what that message means).

C:\>sdelayer -o register

         Error: Table name and spatial column must be specified.
         Error: Entity Type Mask must be specified.
        Error: A geometry storage type must be specified (-t).
        Valid types are ST_GEOMETRY, SDO_GEOMETRY.

ArcSDE 10.2
Layer    Administration Utility
-----------------------------------------------------
sdelayer -o register -l <table,column> -e <entity_mask> -t <storage_type>
                  [Spatial_Index] [{-R <SRID> | [Spatial_Ref_Opts]}]
                  [-C <row_id_column>[,{SDE|USER}[,<min_ID>]]]
                  [-E {empty | xmin,ymin,xmax,ymax}][-P {BASIC | HIGH}]
                  [-S <layer_description_str>] [-q]
                  [-k <config_keyword>] [-i <service>] [-s <server_name]
                  [-u <DB_User_name>] [-p <DB_User_password>] [-D <database>]
Where
      [Spatial_Ref_Opts] := [-x <xoffset,yoffset,xyscale[,xyClusterTol]>]
                            [-z <zoffset,zscale[,zClusterTol]>]
                            [-m <moffset,mscale[,mClusterTol]>]
                            [-G {<projection_ID> | file=<proj_file_name>}]

      [Spatial_Index]    := [-g {<Grid_Options> |
                                  GRID,<Grid_Options> |
                                  AUTOMATIC |
                                  NONE |
                                  RTREE}]

      [Grid_Options]     := [<grid_sz0>[,<grid_sz1>[,<grid_sz2>]]
                                                   [,{FULL | SPARSE}]]
sdelayer -h
sdelayer -?


You must specify an appropriate coordinate reference (including a HIGH precision), include
a USER-set rowid column (from the existing table/view, that conforms with NOT NULL unique
integer), and specify a single topology class in the entity mask (ArcGIS requirement).  The
registration must be performed by the table/view owner.  If the object is a view, then you
should make sure its characteristics match that of the source table ('sdelayer -o describe_long').

- V
0 Kudos
VinceAngelo
Esri Esteemed Contributor
Marco, I would not classify a Query Layer as a "view" at all.  It is QUERY. 

A query is an expression which retrieves data from tables.  A view is a
database object that stores an SQL query in such a way to reduce the
overhead of parsing the query.

The Query Layer stores the query and rendering rules against either a
table or a view, and reapplies that query each time it needs to
(including at maximum extent, at which point it might not be the most
efficient way to query).

One solution to slow spatial-first queries would be to create two scale-
dependent layers, one whose query forces the database to use a full
table scan, and one that lets the spatial index operate.

- V
0 Kudos
MarcoBoeringa
MVP Regular Contributor
Marco, I would not classify a Query Layer as a "view" at all.  It is QUERY. 

A query is an expression which retrieves data from tables.  A view is a
database object that stores an SQL query in such a way to reduce the
overhead of parsing the query.


The Query Layer stores the query and rendering rules against either a
table or a view, and reapplies that query each time it needs to


Thanks for that addition that clarifies it further. I already put the "view" notion of the Query Layer in quotes, and tried to emphasize the rendering aspect of it.

(note that views cannot be registered with
the geodatabase, just with ArcSDE, which is what that message means).


Made some changes to accommodate that. Going to log out now...
0 Kudos
RandyKreuziger
Occasional Contributor III


It's not just spatial views, it's all tables which involve GEOMETRY/GEOGRAPHY objects,
but these are an aspect of Microsoft's database implementation (and the difficulty of
tuning it), not anything Esri has control over.

- V


I've noticed the horrible responce for Geometry verses SDEBindary.  Has ESRI said how long SDEBindary will continue to be a storage option? 

There is a known bug NIM089293 from a year ago dealing with the draw time for the GEOMETRY spatial type verses the draw time for SDEBINARY.  Unfortunately, it's severity is "low."  In my experience that usually means it's not going to be fixed.
0 Kudos
VinceAngelo
Esri Esteemed Contributor
What could Esri possibly do about this issue?  Stage a coup? -V
0 Kudos
MarcoBoeringa
MVP Regular Contributor
I've noticed the horrible response for Geometry verses SDEBinary.  Has ESRI said how long SDEBinary will continue to be a storage option?


A more interesting question is if there are technical reasons why SDEBINARY could no longer be supported in the future in SQL Server. In case of Oracle, the LONGRAW field type that stored SDEBINARY was deprecated by Oracle, so that set a clear endpoint. The more interesting (naive) question is therefore if the field type used by SDEBINARY in SQL Server, is set to go away.

Part of that answer lies probably in the below quote from this Help page and additionally, see this one for more explanation:

"In ArcGIS 9.3 and lower releases, ArcSDE Compressed Binary storage in SQL Server is stored as an image data type. New data created using ArcSDE Compressed Binary storage in SQL Server at ArcGIS 10 and later releases is stored as a varbinary(max) data type."

varbinary is explained here in Microsoft Technet pages:
binary and varbinary (Transact-SQL)

This datatype was one of the data types that is replacing ntext, text and image data types set for deprecation, see this page:
ntext, text, and image (Transact-SQL)

As can been seen from the quote above, image was the field type ESRI used for storing SDEBINARY at ArcGIS 9.3.x and below. So ESRI already anticipated and accommodated the deprecation of this datatype by Microsoft by switching to varbinary(max) at 10.0, so technically the option for storing SDEBINARY doesn't seem to be going anywhere soon...

Of course, the main reason ESRI has been pushing for a shift to "native" Geometry storage or ST_Geometry, is the better accessibility by third party (CAD) software to the data, and the abilities to easily define views etc. on the RDBMS side including spatial data (no separation of geometries and attributes in different tables - feature and business - for a single Feature Class, no "inaccessible" Compressed Binary storage). Another more technical reason may lie in this latter remark, as maintaining the referential integrity between feature and business table is more challenging (see this Help page and the remarks about the triggers set to manage this)

But of course, if your business / organizational workflow won't involve anything other than ESRI software accessing geodatabases, than these advantages are mute...

There is a known bug NIM089293 from a year ago dealing with the draw time for the GEOMETRY spatial type verses the draw time for SDEBINARY. Unfortunately, it's severity is "low."  In my experience that usually means it's not going to be fixed.


This issue should probably not have been logged as a "known bug" at ESRI, as, as Vince explained already, this is not an issue ESRI is responsible for, but Microsoft. If it should be logged as a bug (which is already disputable since true bugs are generally code errors, not performance issues which would more realistically classify as "enhancement requests"), it should be logged at Microsoft.
0 Kudos
RandyKreuziger
Occasional Contributor III
What could Esri possibly do about this issue?  Stage a coup? -V


Sorry, but I'm not the one who classified it as a bug it was ESRI.

But why has ESRI made Geometry the default storage format at ArcSDE 10.1 pushing users towards the slower performing Microsoft spatial datatype in the first place? 

I'd still like to know what ESRI's plan is with SDEBinary.  If it's going to be around for the next 3 years I'll stick with it.
0 Kudos
VinceAngelo
Esri Esteemed Contributor
But why has ESRI made Geometry the default storage format at ArcSDE 10.1

Interoperability.

It's not always slower -- the hellish B/S/F table join should be much slower than
native search; it's just that Microsoft's spatial index algorithm needs work.

I have no knowledge of Esri plans, yet I seriously doubt SDEBINARY will last another
three years. 

- V
0 Kudos
NateArnold
Occasional Contributor
The 'sdelayer -o register' command is used to register tables or views in native storage
(or ST_GEOMETRY, where available) with ArcSDE


Thanks for that.  Bummer more command line.

Nate
0 Kudos
ThomasColson
MVP Frequent Contributor

I've come to be pretty dependent on both Geometry and Geography storage types, given that I can do significantly more automated data management and processing with them both. Knowing that I'm taking a performance hit by using them, there are a lot of tuning options available, which I've managed to get Geography performing as well, if not faster in some cases, than binary. The biggest gain comes from tweaking the spatial index​, but there are other gains from adding other non-clustered indexes and or statistics based on specific use.

As for view performance, one can force indexes on a view at creation with schemabinding:

CREATE VIEW [dbo].[ELKOFFTAKE_VIEW_LOCATIONS]
WITH SCHEMABINDING
AS
SELECT 
OBJECTID, VERROR, MAPSOURCE, SOURCEDATE, EDITDATE, MAPMETHOD, 
QUADNAME, UNITNAME, HERROR, COUNTY, STATE, X_COORD, Y_COORD, LAT, LON, 
WATERSHED, STREAMNAME, ELEVATION, YR, NOTES, COORD_UNITS, COORD_SYSTEM, 
DATUM, UTM_ZONE, MANAGEMENTZONE, PARKDISTRICT, EDITBY, CREATEBY, CREATEDATE, 
UNITCODE, RESTRICTION, VEG_CLASS, LANDFORM, GlobalID AS 'LOC_ID', LOC_NAME,  TRAIL, ROAD, VALID_RESULT
FROM [dbo].[WILD_ELK_OFFTAKE_LOC_PT]
GO

Then

CREATE UNIQUE CLUSTERED INDEX IDX_WILD_ELK_OFFTAKE_LOC_PT 
  ON ELKOFFTAKE_VIEW_LOCATIONS (LOC_ID);
0 Kudos