topic Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle in Data Management Questions

ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Wed, 16 Apr 2014 14:38:46 GMT

I'm maintaing an ArcSDE client application which apparently was working with ArcSDE 9.1

Current configuration: ArcSDE 10.1 with Oracle 10.1, Win 7 x32/x64

Symptoms when calling SE_stream_calculate_table_statistics() with SE_DISTINCT_STATS as mask on a STRING property:

   1. crashes with SE_NET_FAILURE (-10)
   2. memory violation when using direct connect in sdeora11gsrvr101.dll
   3. Works fine with numerial properties (integer/double)
   4. Works fine with SqlServer with any property type (integer/double/string)

Note: 2) and 3) prove the code is correct.

Any information greatly appreciated. Thanks in advance.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Wed, 16 Apr 2014 15:38:19 GMT

Oracle 10gR1 is not supported with ArcGIS 10.1. The library you mention only
works with Oracle 11g. Can you please prove the exact Oracle release in
A.B.C.D.E notation?

Have you applied SP1 to 10.1, and applied the QIP on top of that?

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Wed, 16 Apr 2014 17:15:54 GMT

Thank you Vince for your fast response.

I double checked and the Oracle server is in fact 'Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production'.

This means there is no compatibility issue. As a matter of fact the application works very well except for this particular function. Is there an known bug? I have been looking around in vain for other users experiencing the same.

Thanks.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Wed, 16 Apr 2014 18:41:10 GMT

Have you applied SP1 to 10.1, and applied the QIP on top of that?

Not many folks use the 'C' API, and with Direct Connect, there is no
performance advantage to using SE_stream_calculate_table_statistics()
over a local query that does the same (though you'd have to write your
own distinct value generator from a stream with order_by set).

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Thu, 17 Apr 2014 17:49:01 GMT

I have applied SDE101sp1-CApiSDK.msp. No difference. What is a QIP?

Reimplementing this query is not an easy decision to make, unless I have an official answer this is an ArcSDE bug and there is no chance to be fixed soon.

Thanks.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

MarcoBoeringa — Thu, 17 Apr 2014 18:47:12 GMT

I have applied SDE101sp1-CApiSDK.msp. No difference. What is a QIP?

Look here, it is a larger patch up on top of SP1, kind of like a mini "service pack":
ArcGIS 10.1 SP1 for (Desktop, Engine, Server) Quality Improvement Patch

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Thu, 17 Apr 2014 19:44:13 GMT

Are you using wide strings (SE_WCHAR) or ANSI strings (CHAR)? I'm trying
to reproduce your issue.

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Thu, 17 Apr 2014 20:10:40 GMT

Are you using wide strings (SE_WCHAR) or ANSI strings (CHAR)? I'm trying
to reproduce your issue.

- V

The SE column type doesn't matter here since SE_stream_calculate_table_statistics() doesn't use it. Please see:

http://edndoc.esri.com/arcsde/9.1/capi_functions/stream_query/se_strm_calculate_table_stats.htm

Based on the connection and the name of the property it allocates and populates the stats structure. Internally it gets the column type in order to allocate the appropriate amount of memory.

Thanks.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Thu, 17 Apr 2014 20:24:28 GMT

There are two ways to compile the 'C' API, one of which redefines all functions which take
string parameters to use ACHAR (char) and one which redefines all functions to use SE_WCHAR
(unsigned short) parameters. There are different SE_stream_calculate_table_statistics
functions based on how you compile, and they return different structures (SE_DISTINCT vs
SE_DISTINCTW) with different member types (SE_VALUE vs SE_VALUEW).

Do you compile with SDE_UNICODE enabled (SE_WCHAR parameters) or not?

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Thu, 17 Apr 2014 21:33:28 GMT

At least the server code is consistent -- SE_DISTINCT_STATS computation on an
SE_NSTRING_TYPE column crashes both SE_stream_calculate_table_statistics
and SE_stream_calculate_table_statisticsW with 10.1 SP1 (no QIP).

It doesn't crash on 10.2.0, but it doesn't produce correct output either.

That leaves you with two options -- Using SE_STRING_TYPE columns (with
AL32UTF8 character set for international characters) or coding your own
distinct function for SE_NSTRING_TYPE cols.

You should certainly contact Tech Support (and have them contact me for
sample code to demonstrate the problem).

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

MelitaKennedy — Thu, 17 Apr 2014 22:01:38 GMT

This may be a red herring, but when you contact tech support, you should also tell them that there was a very, very similar bug that was fixed in version 10.0 but for the SDE Java API. I looked at the fix and the same issue is not apparent to me in the C API code--which is why it might not be the same bug.

Reference NIM040332: Running SeQuery.createTableStatistics with the 'SeTable.SeTableStats.SE_DISTINCT_STATS' option hangs when processing an nvarchar field from an ArcSDE 9.2 SQL Server 2005 database and gives an error when processing an nvarchar2 field from an ArcSDE 9.2 Oracle database.

Melita

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Fri, 18 Apr 2014 20:02:55 GMT

This may be a red herring, but when you contact tech support, you should also tell them that there was a very, very similar bug that was fixed in version 10.0 but for the SDE Java API. I looked at the fix and the same issue is not apparent to me in the C API code--which is why it might not be the same bug.

Reference NIM040332: Running SeQuery.createTableStatistics with the 'SeTable.SeTableStats.SE_DISTINCT_STATS' option hangs when processing an nvarchar field from an ArcSDE 9.2 SQL Server 2005 database and gives an error when processing an nvarchar2 field from an ArcSDE 9.2 Oracle database.

Melita

Do you compile with SDE_UNICODE enabled (SE_WCHAR parameters) or not?

Yes.

That leaves you with two options -- Using SE_STRING_TYPE columns

Such change will break for sure cases which currenly work fine. Not an option. My point is the code used to work and we deal with an ArcSDE regression.

You should certainly contact Tech Support (and have them contact me for sample code to demonstrate the problem).

Not sure I understand. You reproduced the problem, located the issue and I would expect ESRI to fix it. That is, I expect you provide me with a case # and a tentative SP so I can inform my customer. Is this not the case?

Regards.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

MelitaKennedy — Fri, 18 Apr 2014 21:45:14 GMT

Hi,

Both Vince and I are on the development-side, not the technical support side. It is possible for me to enter a bug into the development bug-tracking software and export it to the public/technical support software, but it's frowned upon. Partially, because I don't have access to the customer information, and don't know the right questions to ask (like DBMS/OS/release/etc) that's useful to have when putting together a bug report. Technical Support will also be able to identify if the issue has been reported before; I can't always do that reliably. It also gets the bug report into the correct queue; neither Vince nor I will be directly involved in triaging, fixing, or testing it and an analyst familiar with the area will have a better idea where to put it.

Vince was able to repro it, I identified a similar bug but in a different API and stated that I couldn't identify the same issue in the C API code, so someone will still have to figure out what the real problem is. I pointed out the other bug because it might help an analyst/developer to compare working code (even in a different language).

Melita

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Fri, 18 Apr 2014 21:52:50 GMT

No, this is not the case. I don't work in Tech Support, and I don't have access to the
tools they have for incident management. Even if I called them to create an incident,
it would be under my user number, not yours, so you wouldn't receive notifications.
These User Forums are not a replacement for Tech Support.

I would point out that your 9.1 code did not work with SE_NSTRING_TYPE columns,
since SE_NSTRING_TYPE columns weren't supported before ArcGIS 9.2. The default
SE_NSTRING_TYPE creation of Desktop can be overridden through the use of
the "UNICODE_STRING=FALSE" flag in the DBTUNE file (see the documentation
here and here for more info).

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Wed, 23 Apr 2014 19:25:39 GMT

Hi. Our ESRI representative indicated there is no support for ArcSDE 10.0 and 10.1

Could you please confirm?

Thanks.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

MelitaKennedy — Wed, 23 Apr 2014 19:48:41 GMT

Okay, please do not consider this a binding/legal Esri answer, but going by the ArcGIS for Server Product life cycle document, 10.0 is in mature support while 10.1 is still in extended support. Further information on what that means is available here.

Melita

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

MarcoBoeringa — Wed, 23 Apr 2014 19:56:39 GMT

Okay, please do not consider this a binding/legal Esri answer, but going by the ArcGIS for Server Product life cycle document, 10.0 is in mature support while 10.1 is still in extended support. Further information on what that means is available here.

Melita

Ergo (looking at the document Melita linked): they should help you out in Technical Support and answer your questions, as long as you have a support contract. Even for the "Mature" 10.0 that doesn't get any new service packs or patches.

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

VinceAngelo — Wed, 23 Apr 2014 21:21:41 GMT

The 'C' API is deprecated at 10.2, and won't be released with the next rev. This adds
some quirks to the base support policy (I found documentation bugs that are no longer
being corrected), but 10.1 SP1 should be supported (as an Extended product). 10.0
is not yet Retired, but it won't be getting QFEs the way 10.1 would. Try pointing out
the support policy doc to the person with whom you spoke.

The workaround is to write a helper function to do a GROUP BY, ORDER BY query
when the column type is SE_NSTRING_TYPE -- It's probably no more than a
dozen API calls:

SE_queryinfo_get_tables to get the table name

SE_table_describe to get all column types

SE_stream_calculate_table_statistics on non-SE_NSTRING_TYPE columns

SE_stream_prepare_sql to execute the query with the stream , in the format

SELECT count(colname) cnt,colname FROM tablename GROUP BY colname ORDER BY colname

SE_stream_execute to start the stream

se_stream_bind_output_column to a CHAR array long enough to hold the

SE_stream_fetch for up to the maximum number of distinct values

SE_stream_close to end the query

SE_table_free_descriptions to clean up the describe

Also in there is a malloc of enough space for the maximum number of pointers
and a buffer large enough to store all the returned strings, plus the SE_DISTINCT
and SE_VALUE elements. You'd also need your own cleanup function (to call
SE_table_free_stats or to free your own datatype [which needs a flag to indicate
how the data is stacked]). In this situation, I'd name the wrapper
se_stream_calculate_table_statistics and the cleanup se_table_free_stats.
If I weren't on deadline for a delivery I'd code the thing myself right now.

- V

Re: ArcSDE C-API SE_stream_calculate_table_statistics() crash with Oracle

sbrockwell — Thu, 24 Apr 2014 14:44:21 GMT

Excellent information about 10.0 and 10.1 life cycles.

Vince, I know how to code such query. My problem was not how but why 🙂

Thanks everyone here for the outstanding level of support.

Best regards.