Map service periodically crashes (ArcGIS for Server 10.1 SP1)

[DahaeHwang]

We are experiencing periodic crashing of map services on request - it occurs randomly about 5 -10 times during a day... We have map services connecting to both Oracle and Postgresql databases thru SDE direct connections, but it seems not a matter of which database it is accessing to.
We are using clustered GIS Servers under AGS Web adaptor in a virtual environment, and refresh(Stop/start) AGS Services every morning on the servers.
When it crashes, the used memory on the server is less than 70%... (Max 8GB is allocated to each GIS machine)

I've attached the dump files from error report.
Any hep would be appreciated.
Thanks in advance,

[RichardWatson]

http://resources.arcgis.com/en/help/arcobjects-net/componenthelp/index.html#//003z00000150000000

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000074801609 (ServerContainer!CServerObjectFactoryX::AddAddInFolder+0x0000000000000019)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000000
Attempt to read from address 0000000000000000

This says that the function AddAddInFolder is being called with a NULL pointer for the argument addInFolderPath.

[DahaeHwang]

Thanks for your help, Richard.
So the exception indicates that 'AddAddInFolder' method was called without 'addInFolderPath' parameter...
I am still trying to understand what 'IServerObjectFactoryX.AddAddInFolder' does and where it is called.
Is this the 'AddInFolderList.xml' in C:\Users\username\AppData\Roaming\ESRI\Server10.1? Is this method supposed to be called by default when creating a new instance on the server..?

Thanks again,
Dahae

[RichardWatson]

This is what C:\Users\arcgis\AppData\Roaming\ESRI\Server10.1\AddInFoldersList.xml looks like on my machine:

<?xml version="1.0"?>
<ESRI.ArcGIS.AddInFolders ver="1"><ARCSOC/></ESRI.ArcGIS.AddInFolders>

What does it look like on yours?

Note that mine is under user arcgis because that is the default account that ArcGIS Server uses.

I do not know what ESRI is, or is not, supposed to do.  My >guess< is that you have some type of configuration error and we are trying to debug it by reverse engineering from the failure.  Sometimes that works well, sometimes it does not.

[DahaeHwang]

Thanks Richard.
Yes. Mine looks same as your xml file... Thanks for the info and I will keep reserching on this crashing issue.

[RichardWatson]

Maybe you upgraded a site from version 10 (or earlier) and have registry settings which need to be cleared out?

http://gis.stackexchange.com/questions/2406/why-are-my-additional-add-in-folders-disappearing

This is pure guess on my part.  I have a clean 10.1 SP1 machine (not upgraded from 10) and cannot find an AddInFolders registry key on my machine.

[DahaeHwang]

We had a clean 10.1 installation too... I checked the registry and it didn't have a key for AddInFolder.
Thanks for the suggestion.

[AnthonyGiles]

Dahae,

Not sure if it is related but there was a patch released last week which fixes services becoming locked:

http://support.esri.com/en/downloads/patches-servicepacks/view/productid/66/metaid/1991

Regards

Anthony

[DahaeHwang]

Thanks Anthony.
I will give a try.

[DahaeHwang]

Okay, I installed the new patch but am still getting the crashing errors.
The debugged logs were as below... it was crashed @13:26:25 when tried to create instance of service.
I am not sure if this is related to a known bug with clusted GIS machines.
I've attached the dump file from this error.. Any help/suggestion is appreciated.
Thanks,

<Msg time="2013-07-26T13:26:22,153" type="DEBUG" code="9999" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">Creating server context. Isolation level = LOW.</Msg>
<Msg time="2013-07-26T13:26:22,379" type="INFO" code="8270" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">Creating instance of service 'Maps/ServiceName.MapServer'.</Msg>
<Msg time="2013-07-26T13:26:22,470" type="DEBUG" code="9999" source="Server" process="14516" thread="17" methodName="" machine="ServerName" user="" elapsed="">Starting worker thread to host server object.</Msg>
<Msg time="2013-07-26T13:26:22,472" type="DEBUG" code="9999" source="Server" process="14516" thread="17" methodName="" machine="ServerName" user="" elapsed="">Start - initializing engine</Msg>
<Msg time="2013-07-26T13:26:22,474" type="DEBUG" code="9999" source="Server" process="14516" thread="17" methodName="" machine="ServerName" user="" elapsed="">End - initializing engine</Msg>
<Msg time="2013-07-26T13:26:22,977" type="DEBUG" code="9999" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">Creating server context. Isolation level = LOW.</Msg>
<Msg time="2013-07-26T13:26:25,376" type="SEVERE" code="8273" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">Services containing process crashed. Maps/ServiceName.MapServer instance has crashed.</Msg>
<Msg time="2013-07-26T13:26:25,377" type="DEBUG" code="9999" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">java.lang.Exception: Maps/ServiceName.MapServer instance has crashed.
at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:365)
at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:219)
at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:154)
at com.esri.arcgis.discovery.ejb.impl.GenericServerBean.init(GenericServerBean.java:51)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164)
at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122)
at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237)
at org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:167)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:165)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:136)
at org.apache.openejb.server.ejbd.EjbRequestHandler.doEjbObject_BUSINESS_METHOD(EjbRequestHandler.java:238)
at org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:129)
at org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
at org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
at org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)
at org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
at org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
at org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.net.SocketException: Connection reset
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:227)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:142)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at $Proxy43.createService(Unknown Source)
at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:342)
... 34 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:213)
... 40 more
</Msg>
<Msg time="2013-07-26T13:26:25,606" type="DEBUG" code="9999" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">Cleaning up previous service instance and server context.</Msg>
<Msg time="2013-07-26T13:26:26,487" type="INFO" code="8270" source="Server" process="5608" thread="1" methodName="" machine="ServerName" user="" elapsed="">Creating instance of service 'Maps/EarlyAssistance.MapServer'.</Msg>
<Msg time="2013-07-26T13:26:26,807" type="DEBUG" code="9999" source="Server" process="16252" thread="17" methodName="" machine="ServerName" user="" elapsed="">Starting worker thread to host server object.</Msg>
<Msg time="2013-07-26T13:26:26,823" type="DEBUG" code="9999" source="Server" process="16252" thread="17" methodName="" machine="ServerName" user="" elapsed="">Start - initializing engine</Msg>
<Msg time="2013-07-26T13:26:26,927" type="DEBUG" code="9999" source="Server" process="16252" thread="17" methodName="" machine="ServerName" user="" elapsed="">End - initializing engine</Msg>