Arcsde Service "hangs" intermittently

[User35489]

Dear All,

Arcsde Service "hangs" intermittently

Below are contents of giomgr.log

Sun Feb 24 14:03:07 2013 - Error (-9):Couldn't Start Server Task.
Sun Feb 24 14:03:07 2013 - Process 6529, no shared information block established
Sun Feb 24 14:03:07 2013 - SDE Server 6529 exit'd with status 3
Sun Feb 24 14:03:17 2013 - SDE Server Pid 6552 Registered, User: abu Schema: sde.
Sun Feb 24 14:03:23 2013 - Error (-51):Couldn't Start Server Task.
Sun Feb 24 14:03:23 2013 - Process 6557, no shared information block established
Sun Feb 24 14:03:23 2013 - SDE Server 6557 exit'd with status 3
Sun Feb 24 14:03:23 2013 - Error (-51):Couldn't Start Server Task.


Managed by killing the service and restarting it.

Please Share your experience

Thanks

[MarcoBoeringa]

Googling for "Error (-9):Couldn't Start Server Task." yielded the following old ESRI links that may be of some use. Especially see Danny Krouk's remarks in the discussion thread:

http://forums.esri.com/Thread.asp?c=2&f=59&t=81786

http://support.esri.com/es/knowledgebase/techarticles/detail/19035

Also, these two links might be of some use:

http://oracledbatips.com/arcsde/ArcSDE-error-codes-0-20.html

http://edndoc.esri.com/arcsde/9.1/capi_functions/returncodes_incl.htm

From the second link:
SE_INVALID_USER     (-9)
The user attempting to access the ArcSDE server does not have a valid login account on the ArcSDE host machine. Returned by the SE_connection_create call only.

SE_DB_IO_ERROR     (-51)
A database-level I/O error occurred. You can call the extended error code immediately after this code is returned to inquire about the specific database error code.

[VinceAngelo]

Hanging application servers used to be a problem in the 8.x/9.0 era, and happened occasionally
in the 9.1 era.  Haven't heard of that issue in 9.2, 9.3, 9.3.1, 10.0, or 10.1.  Of course, the
current direction is to use Direct Connect, so you'd hardly notice if it did happen.

- V

[User35489]

Is there any way to trace the source that causes HANG ? Log Files etc ?

-AS

[MarcoBoeringa]

Is there any way to trace the source that causes HANG ? Log Files etc ?

-AS

Since the -9 error indicates possible issues with database login, maybe taking a look at RDBMS error logs may give some idea of what is going on:

View the SQL Server Error Log (SQL Server Management Studio)

Using Trace and Error Logs (Oracle)

Vince can probably give you better instruction or insight if this is useful in this specific case...

EDIT: I now see -9 actually just says "login", so maybe having a look at the server OS (Windows?) error logs could be of some use too, primarilly for SQL Server, where database authentication is usually derived from Windows. Additional possibly useful link in this SQL Server contect: SQL SERVER �?? A Quick Look at Logging and Ideas around Logging.

[User35489]

Since the -9 error indicates possible issues with database login, maybe taking a look at RDBMS error logs may give some idea of what is going on:

View the SQL Server Error Log (SQL Server Management Studio)

Using Trace and Error Logs (Oracle)

Vince can probably give you better instruction or insight if this is useful in this specific case...

EDIT: I now see -9 actually just says "login", so maybe having a look at the server OS (Windows?) error logs could be of some use too, primarilly for SQL Server, where database authentication is usually derived from Windows. Additional possibly useful link in this SQL Server contect: SQL SERVER �?? A Quick Look at Logging and Ideas around Logging.

Dear,

Thanks for your comment.
I forgot to mention above, We are on
RHEL 5 - Oracle 10gR2 - ArcSDE 9.3

I was trying to figure out the error through sde logs(giomgr, sde.log etc). I will escalate it to the Database / Operating System support teams as needed.

NOTE: One more important thing which i would like to share with you all is One of our user is running Caching Process(may be huge) nowadays. Do you think tracing this activity will give us some good clue towards troubleshooting ?
If yes please suggest a method

Thanks all for valuable comments

-AS

[VinceAngelo]

The best possible source for Oracle hangs is using Oracle 10.2.0.1 (which is unsupported)
(and very buggy -- it should not be used).

All 9.3 instances should have been upgraded to 9.3.1 SP2 years ago, after it was released.

There is no way to trace a hang in the ArcSDE logs.

- V

[User35489]

The best possible source for Oracle hangs is using Oracle 10.2.0.1 (which is unsupported)
(and very buggy -- it should not be used).

All 9.3 instances should have been upgraded to 9.3.1 SP2 years ago, after it was released.

There is no way to trace a hang in the ArcSDE logs.

- V

Vince I agree with you to move our system to the recommended versions. And we are planned to migrate to 10.1 within few days.
Meanwhile i need to support the users, and struggling for that. I hope you understand the situation.
Dear Let me ask you a question
I feel that some limitations which are set in GIOMGR is been exceeded by a user who is running cache process.
Is there a way to find the entries in GIOMGR has been exceeded by any process that leads to HANGING ?


Thanks for your support
-AS

[VinceAngelo]

I repeat: There is no way to trace a hang in the ArcSDE logs.

Keep in mind that a service hang is 'giomgr' failure, while an application crash is 'gsrvr'
failure.  There should be no correlation at all between the two (except an unsupported
database release).

- V

PS: Migration to 10.1 won't help with an old Oracle instance.  ArcSDE 9.3 and 10.1 were
both certified with Oracle 10.2.0.3.  If you're not running at least 10.2.0.3, then that is
the cause of your hangs.

[User35489]

I repeat: There is no way to trace a hang in the ArcSDE logs.

Keep in mind that a service hang is 'giomgr' failure, while an application crash is 'gsrvr'
failure.  There should be no correlation at all between the two (except an unsupported
database release).

- V

PS: Migration to 10.1 won't help with an old Oracle instance.  ArcSDE 9.3 and 10.1 were
both certified with Oracle 10.2.0.3.  If you're not running at least 10.2.0.3, then that is
the cause of your hangs.

It will be
RHEL 6.2, Oracle Database 11g Enterprise Edition 11.2.0.3.0 64bit Production, ArcSDE 10.1


-AS