[SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

27382
14
Jump to solution
01-18-2013 02:09 AM
ClaudioSparpaglione
New Contributor II
Hi,

I've been running ArcGIS Server 10.1 with SP1 on a Windows Server 2012 64-bit machine for months without any problems in publishing map services and geoprocessing services. I have an ArcMap 32-bit installation on the same machine of the server, which I use to view maps and publish services on the server.

Yesterday, this crazy situation happened to me: I tried to publish a new map service through ArcMap, but it was stuck and said "Not responding" for minutes, so I killed it. Tried again and again: no fortune. Then I examined the server log file, which in my machine is at the path "C:\arcgisserver\logs\<machine_name>\server" and I found lines like the following ones:

< ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. Utilities/PrintingTools.GPServer instance has crashed.</Msg>  < ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg>



The log file contained one crash-report line for each of the geoprocessing services running on the server.

I therefore tried to investigate: I opened the ArcGIS Manager web application and tried to restart the PublishingTools service (which comes bundled with the server): it took a lot of time, then the service  correctly stopped but in the end this message popped up:

System.PublishingTools.GPServer: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset


I tried to restart all of the other geoprocessing (either System's and my custom ones), and for each of them I got similar messages printed on screen.
I then tried to restart all of the other services (map services, geometry services) and everything went fine!
So it seems to be the just an error on the geoprocessing services.

I further investigated by setting the server log's verbosity to DEBUG (that is: print everything!) and this is the full stack-trace of the exception:

<Msg time="..." type="SEVERE" code="8273" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg> <Msg time="..." type="DEBUG" code="9999" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">java.lang.Exception: System/PublishingTools.GPServer instance has crashed.     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:363)     at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:218)     at com.esri.arcgis.discovery.ejb.impl.GPServerSyncBean.init(GPServerSyncBean.java:47)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164)     at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122)     at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237)     at org.apache.openejb.core.stateless.StatelessInstanceManager.createInstance(StatelessInstanceManager.java:448)     at org.apache.openejb.core.stateless.StatelessInstanceManager.access$100(StatelessInstanceManager.java:72)     at org.apache.openejb.core.stateless.StatelessInstanceManager$BackgroundThread.run(StatelessInstanceManager.java:507) Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:      java.net.SocketException: Connection reset     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:209)     at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)     at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:124)     at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)     at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)     at $Proxy43.createService(Unknown Source)     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:340)     ... 22 more Caused by: java.net.SocketException: Connection reset     at java.net.SocketInputStream.read(SocketInputStream.java:168)     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)     at java.io.DataInputStream.readByte(DataInputStream.java:248)     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)     ... 28 more </Msg>



As additional information:

  • All firewall systems on my machine are fully deactivated

  • No recent modification has been made to the networking in my LAN

  • I installed a few Windows updates last week

Can anyone help me in solving the issue?


Another question: could this problem be tied to a "dirty" Python installation? I'm not sure that GP services rely on Python (I'm getting a Java exception!)
Tags (2)
14 Replies
SimonJackson
Occasional Contributor III

Thanks  loads - this helped me also.

I did try updatingmy registry settings with those from another machine that had an identical ArcGIS for Server installed, but something did not work.

Easier option for me was to simply run a Repair on the ArcGIS for Server install - fixed everything up!

MalcolmJ
Occasional Contributor II

Thanks this solved my problem completely.

I have Server 10.3 and I had to set this as the value in the registry for PythonPath.

"C:\Python27\ArcGISx6410.3\Lib;C:\Python27\ArcGISx6410.3\DLLs;C:\Python27\ArcGISx6410.3\Lib\lib-tk"

0 Kudos
ClaireBrill
New Contributor III

So helpful! Thanks for the detailed question and solution.

0 Kudos
GaryPrior
New Contributor II

Thanks so much! Solved my same issue! Even on AGS 10.5

0 Kudos
SandeepPastapure
New Contributor

If you are getting problem to start only GP services and using Windows Server 2008 R2 and Windows 7 then need to be install the patch.

download from this link https://support.esri.com/en/download/7576 

Thanks.