Best way to move versioned ST_GEOMETRY FCs between Oracle 11gR2 instances?

8754
23
03-08-2012 07:15 AM
danan
by
Occasional Contributor III
What is the current recommended way to move versioned feature classses having ST_GEOMETRY columns from one 11gR2 Oracle instance to another? Is this supported at all with datapump? The primary use case: developers want occassional data refreshes from production spatial data on another instance at the same Oracle/SDE version. Another use case might be for recovery.

We have tried using datapump and had errors like the following for any FC with an ST_GEOMETRY column:

ORA-31693: Table data object "<schema here>"."<feature class here>" failed to load/unload and is being skipped due to error:
ORA-02354: error in exporting/importing data
ORA-00600: internal error code, arguments: [kpudpxcs_ctxConvertStream_ref_1], [SYS_TYPEID("SHAPE")], [], [], [], [], [], [], [], [], [], []

We've used expdp and impdp in various ways and always get the same type of error above.
We've tried:

* Exported SDE and all data owner schemas in one expdp statement; then used impdp to import all schemas simultaneously with that dump file
* Exported SDE and data owner schemas with separate expdp statements; then imported the SDE first and data owner schemas afterwards

In a related thread from a year ago (where I'd asked about using datapump to go from a 10g to 11g instance), some suggested that datapump wasn't up to this sort of task; that there were too many dependencies like PUBLIC synonyms and various grants to PUBLIC, etc. Does that hold true for the 11.2.0.2 incarnation of datapump working with ArcSDE 9.3.1 SP2?

We are using:
Oracle version: 11.2.0.2 on RHEL 5.4
SDE version: 9.3.1 SP2

What options do we have considering ST_GEOMETRY is a "user defined" data type? It makes sense to me that the SDE schema should be imported first to get ST_GEOMETRY created; presuming the type lives in the SDE schema as it seems to. Here is an object count from the Oracle source instance:

select owner, object_type, count(*) from dba_objects where owner = 'SDE' and object_type like '%TYPE%' group by owner, object_type order by object_type;

OWNER                          OBJECT_TYPE           COUNT(*)
------------------------------ ------------------- ----------
SDE                            INDEXTYPE                    1
SDE                            TYPE                        39
SDE                            TYPE BODY                   24


After importing, the Oracle target instance has only 31 TYPEs under the SDE schema. It is missing the following 8. However, all %TYPE% objects that *did* import have a status of VALID. So again, here are objects that are valid on the source instance but missing on the target instance. Not sure datapump even complained about these. I noticed they all have subobject names:

OWNER                          OBJECT_TYPE         OBJECT_NAME                    SUBOBJECT_NAME         STATUS
------------------------------ ------------------- ------------------------------ -----------------------------
SDE                            TYPE                ST_DOMAIN_METHODS              $VSN_4                 VALID
SDE                            TYPE                ST_DOMAIN_METHODS              $VSN_3                 VALID
SDE                            TYPE                ST_DOMAIN_METHODS              $VSN_2                 VALID
SDE                            TYPE                ST_DOMAIN_METHODS              $VSN_5                 VALID
SDE                            TYPE                ST_DOMAIN_METHODS              $VSN_1                 VALID
SDE                            TYPE                ST_DOMAIN_STATS                $VSN_3                 VALID
SDE                            TYPE                ST_DOMAIN_STATS                $VSN_2                 VALID
SDE                            TYPE                ST_DOMAIN_STATS                $VSN_1                 VALID


UPDATE:
Would doing a datapump import of the SDE schema first (which has been error-free for us), creating public synonyms as recommended by the KB article below, *then* importing the data owner schemas be a good workaround? Or are we looking at having to use the full option in datapump to make things work?

Exporting and importing an Oracle SDE schema does not include public synonyms
http://support.esri.com/en/knowledgebase/techarticles/detail/34328
0 Kudos
23 Replies
JaneLamont
New Contributor
Hi Dana,
I have encountered the exact problem and symptoms as you have described, with my Oracle 11.2.0.3 and SDE 9.3.  How to move the data between instances? 
We have also encountered in 3 databases that Data Pump broke.  Rebuilding the DBMS_DATAPUMP PL/SQL package has not been successful. Oracle Support has designated it a bug, 13829814, with their development.
Through testing or rebuilding instances and bringing in the SDE data that the SDE data breaks Data Pump.   Then yesterday I encountered the same errors as yours.
The previous thread of using export/import with the grants to PUBLIC is being tested as I write.  Please keep this thread updated with any new information as I will also.
Thanks,
Jane
0 Kudos
danan
by
Occasional Contributor III
Hi Dana,
I have encountered the exact problem and symptoms as you have described, with my Oracle 11.2.0.3 and SDE 9.3.  How to move the data between instances? 
We have also encountered in 3 databases that Data Pump broke.  Rebuilding the DBMS_DATAPUMP PL/SQL package has not been successful. Oracle Support has designated it a bug, 13829814, with their development.
Through testing or rebuilding instances and bringing in the SDE data that the SDE data breaks Data Pump.   Then yesterday I encountered the same errors as yours.
The previous thread of using export/import with the grants to PUBLIC is being tested as I write.  Please keep this thread updated with any new information as I will also.
Thanks,
Jane


Thanks Jane. Sorry for your troubles and hope you succeed. I'll add to the thread if we have any success. Hoping others out there can comment since data migration is a fundamental function. But not something I've seen definitively addressed where ST_GEOMETRY is concerned.

I suppose one solution (compromise) might be to jettison ST_GEOMETRY completely and convert to SDE_LOB. Another might be to use RMAN to clone an instance. I wonder if datapump failing is specific to Esri's ST_GEOMETRY UDT (User Defined Type). Or if the datapump problem exists for migrating UDTs from a variety of vendors. Based on some other issues we've had with ST_GEOMETRY, I wonder if it's best to switch to SDE_LOB unless ST_GEOMETRY spatial *functions* are an absolute business requirement.
0 Kudos
VinceAngelo
Esri Esteemed Contributor
SDE.ST_GEOMETRY has consistently better performance than SDE_LOB.  You should
evaluate the cost of moving to LOBs before deciding to do so.

- V
0 Kudos
danan
by
Occasional Contributor III
SDE.ST_GEOMETRY has consistently better performance than SDE_LOB.  You should
evaluate the cost of moving to LOBs before deciding to do so.

- V


Thanks Vince. I have also heard the opposite. In principle ST_GEOMETRY should be consistently faster. But I don't know if that's the result for every customer on every sub-platform. Seems reasonable to test in one's own environment to see what is actually the case. That said, what to do about migrating feature classes with the ST_GEOMETRY User Defined Type? Is there a current Best Practice? Is datapump no longer recommended (if it ever was), etc? It's also been recommended to me to use RMAN to clone a DB instance as a means of moving ST_GEOMETRY-based data from instance to instance.

Dana
0 Kudos
JaneLamont
New Contributor
I can replicate your missing SDE types.
There is 1 missing subobject:
OBJECT_NAME                    SUBOBJECT_NAME                 STATUS
------------------------------ ------------------------------ ----------
ST_DOMAIN_METHODS              $VSN_1                         VALID
0 Kudos
danan
by
Occasional Contributor III
I can replicate your missing SDE types.
There is 1 missing subobject:
OBJECT_NAME                    SUBOBJECT_NAME                 STATUS
------------------------------ ------------------------------ ----------
ST_DOMAIN_METHODS              $VSN_1                         VALID


No progress here. Jane, are you using ST_GEOMETRY because it's the default spatial data type for SDE on Oracle, because you have a business requirement for spatial SQL functions, or another reason? For us, SQL functions were a "nice to have." Don't know that anyone is currently using them. And what's very much not nice to have: limited options for moving data around. I believe we also chose it because of a belief that ST_GEOMETRY being the default spatial data type means it would be the focus of future development.
0 Kudos
JaneLamont
New Contributor
We have come to the conclusion that our problems of Data PUmp not working are related to Oracle Note, Ora-21700 Object Does Not Exist Or Is Marked For Delete When Dropping a User With ST_GEOMETRY Dependencies [ID 1385929.1].  Dictionary gets corrupted when tables using a specific object type (SDE.ST_GEOMETRY) are tried to be dropped after the type was tried to get dropped.    This corruption in the dictionary may have affected the DBMS_DATAPUMP package.  I have tried re-installing the package with no success. 

ESRI has a note on the subject of dropping the user SDE and/or other users who use the ST_GEOMETRY datatype, http://support.esri.com/en/knowledgebase/techarticles/detail/34483. Drop the users using the ST_GEOMETRY, before dropping the SDE user, not the other way around.

Yes, we use the SDE for business.  We are in the midst of moving the instance to a new server and have been pulling our hair out trying to move SDE.  On our source database, Data Pump is broken with no solution from Oracle Support to fix it.  Now the new destination database has a broken Data Pump as well with some schemas up and running in production.  We are working towards re-building the instance, re-build SDE from scratch and bring in the other schemas again.   And be very clear which user schemas use the ST_GEOMETRY datatype and never drop them!
0 Kudos
danan
by
Occasional Contributor III
We have come to the conclusion that our problems of Data PUmp not working are related to Oracle Note, Ora-21700 Object Does Not Exist Or Is Marked For Delete When Dropping a User With ST_GEOMETRY Dependencies [ID 1385929.1].  Dictionary gets corrupted when tables using a specific object type (SDE.ST_GEOMETRY) are tried to be dropped after the type was tried to get dropped.    This corruption in the dictionary may have affected the DBMS_DATAPUMP package.  I have tried re-installing the package with no success. 

ESRI has a note on the subject of dropping the user SDE and/or other users who use the ST_GEOMETRY datatype, http://support.esri.com/en/knowledgebase/techarticles/detail/34483. Drop the users using the ST_GEOMETRY, before dropping the SDE user, not the other way around.

Yes, we use the SDE for business.  We are in the midst of moving the instance to a new server and have been pulling our hair out trying to move SDE.  On our source database, Data Pump is broken with no solution from Oracle Support to fix it.  Now the new destination database has a broken Data Pump as well with some schemas up and running in production.  We are working towards re-building the instance, re-build SDE from scratch and bring in the other schemas again.   And be very clear which user schemas use the ST_GEOMETRY datatype and never drop them!


Hi Jane,

Thanks for posting your findings. Sorry you're having so much trouble. We may ask our customer to open a ticket with Esri. Although the support forums are terrific, they aren't intended to be a replacement for Esri Support. Many problems can be solved here but this issue is particularly stubborn / tricky. I cannot open a ticket myself so I can only suggest it as an option.

Incidentally, at a class this week on Oracle Enterprise Manager (Grid Control) I mentioned the datapump issue to an instructor. His take was that many 3rd party UDTs have trouble with datapump. My (over-?)interpretation: Esri may be the victim here, not the culprit. Still, it would be nice to have official guidance. I'll post an update if we open a ticket or have success in some other way.

What about using a file geodatabase as an intermediate medium for versioned feature classes? There's no ST_GEOMETRY for Oracle spatial data type in a file GDB (naturally). But I wonder if the customer could run a Python geoprocessing task to recreate things as ST_GEOMETRY from a FGDB source to the desired Oracle target. Anyone done this? It's not entirely clear to me if the customer needs versioned edits and state trees preserved during the migration process. Will ask for clarification. Likely overthinking this?

Happy Friday. 🙂

Best,

Dana
0 Kudos
JaneLamont
New Contributor
Our service request with Oracle is not providing workarounds, tips or solutions for working with SDE to avoid the data dictionary corruption or avoiding the breakage of Data Pump. We have just rebuilt SDE manually in another database so have a good export to try importing into another working instance, hopefully not breaking Data Pump and keeping the d.d. intact and no missing ST_GEOMETRY data types.  I have just completed a support call to ESRI for their help.  I will keep you updated.

Jane
0 Kudos