AnsweredAssumed Answered

[SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

Question asked by csparpaglione on Jan 18, 2013
Latest reply on Mar 9, 2018 by sandeep_pas
Hi,

I've been running ArcGIS Server 10.1 with SP1 on a Windows Server 2012 64-bit machine for months without any problems in publishing map services and geoprocessing services. I have an ArcMap 32-bit installation on the same machine of the server, which I use to view maps and publish services on the server.

Yesterday, this crazy situation happened to me: I tried to publish a new map service through ArcMap, but it was stuck and said "Not responding" for minutes, so I killed it. Tried again and again: no fortune. Then I examined the server log file, which in my machine is at the path "C:\arcgisserver\logs\<machine_name>\server" and I found lines like the following ones:

< ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. Utilities/PrintingTools.GPServer instance has crashed.</Msg>  < ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg>



The log file contained one crash-report line for each of the geoprocessing services running on the server.

I therefore tried to investigate: I opened the ArcGIS Manager web application and tried to restart the PublishingTools service (which comes bundled with the server): it took a lot of time, then the service  correctly stopped but in the end this message popped up:

System.PublishingTools.GPServer: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset


I tried to restart all of the other geoprocessing (either System's and my custom ones), and for each of them I got similar messages printed on screen.
I then tried to restart all of the other services (map services, geometry services) and everything went fine!
So it seems to be the just an error on the geoprocessing services.

I further investigated by setting the server log's verbosity to DEBUG (that is: print everything!) and this is the full stack-trace of the exception:

<Msg time="..." type="SEVERE" code="8273" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg> <Msg time="..." type="DEBUG" code="9999" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">java.lang.Exception: System/PublishingTools.GPServer instance has crashed.     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:363)     at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:218)     at com.esri.arcgis.discovery.ejb.impl.GPServerSyncBean.init(GPServerSyncBean.java:47)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164)     at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122)     at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237)     at org.apache.openejb.core.stateless.StatelessInstanceManager.createInstance(StatelessInstanceManager.java:448)     at org.apache.openejb.core.stateless.StatelessInstanceManager.access$100(StatelessInstanceManager.java:72)     at org.apache.openejb.core.stateless.StatelessInstanceManager$BackgroundThread.run(StatelessInstanceManager.java:507) Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:      java.net.SocketException: Connection reset     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:209)     at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)     at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:124)     at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)     at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)     at $Proxy43.createService(Unknown Source)     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:340)     ... 22 more Caused by: java.net.SocketException: Connection reset     at java.net.SocketInputStream.read(SocketInputStream.java:168)     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)     at java.io.DataInputStream.readByte(DataInputStream.java:248)     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)     ... 28 more </Msg>



As additional information:

  • All firewall systems on my machine are fully deactivated

  • No recent modification has been made to the networking in my LAN

  • I installed a few Windows updates last week

Can anyone help me in solving the issue?


Another question: could this problem be tied to a "dirty" Python installation? I'm not sure that GP services rely on Python (I'm getting a Java exception!)

Outcomes