[SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

[ClaudioSparpaglione]

Hi,

I've been running ArcGIS Server 10.1 with SP1 on a Windows Server 2012 64-bit machine for months without any problems in publishing map services and geoprocessing services. I have an ArcMap 32-bit installation on the same machine of the server, which I use to view maps and publish services on the server.

Yesterday, this crazy situation happened to me: I tried to publish a new map service through ArcMap, but it was stuck and said "Not responding" for minutes, so I killed it. Tried again and again: no fortune. Then I examined the server log file, which in my machine is at the path "C:\arcgisserver\logs\<machine_name>\server" and I found lines like the following ones:

< ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. Utilities/PrintingTools.GPServer instance has crashed.</Msg>  < ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg>

The log file contained one crash-report line for each of the geoprocessing services running on the server.

I therefore tried to investigate: I opened the ArcGIS Manager web application and tried to restart the PublishingTools service (which comes bundled with the server): it took a lot of time, then the service  correctly stopped but in the end this message popped up:

System.PublishingTools.GPServer: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset

I tried to restart all of the other geoprocessing (either System's and my custom ones), and for each of them I got similar messages printed on screen.
I then tried to restart all of the other services (map services, geometry services) and everything went fine!
So it seems to be the just an error on the geoprocessing services.

I further investigated by setting the server log's verbosity to DEBUG (that is: print everything!) and this is the full stack-trace of the exception:

<Msg time="..." type="SEVERE" code="8273" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg> <Msg time="..." type="DEBUG" code="9999" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">java.lang.Exception: System/PublishingTools.GPServer instance has crashed.     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:363)     at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:218)     at com.esri.arcgis.discovery.ejb.impl.GPServerSyncBean.init(GPServerSyncBean.java:47)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164)     at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122)     at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237)     at org.apache.openejb.core.stateless.StatelessInstanceManager.createInstance(StatelessInstanceManager.java:448)     at org.apache.openejb.core.stateless.StatelessInstanceManager.access$100(StatelessInstanceManager.java:72)     at org.apache.openejb.core.stateless.StatelessInstanceManager$BackgroundThread.run(StatelessInstanceManager.java:507) Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:      java.net.SocketException: Connection reset     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:209)     at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)     at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:124)     at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)     at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)     at $Proxy43.createService(Unknown Source)     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:340)     ... 22 more Caused by: java.net.SocketException: Connection reset     at java.net.SocketInputStream.read(SocketInputStream.java:168)     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)     at java.io.DataInputStream.readByte(DataInputStream.java:248)     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)     ... 28 more </Msg>

As additional information:

• All firewall systems on my machine are fully deactivated


• No recent modification has been made to the networking in my LAN


• I installed a few Windows updates last week

Can anyone help me in solving the issue?


Another question: could this problem be tied to a "dirty" Python installation? I'm not sure that GP services rely on Python (I'm getting a Java exception!)

[ClaudioSparpaglione · Accepted solution]

After a lot of tries and speculations, I finally managed to find a solution.

I discovered that in the Windows Registry the following hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7

was completely empty.

This hive was related to a standalone Python 2.7 installation that I made aside to ArcGIS's ones (32-bit for ArcMap and 64-bit for ArcGIS Server - both are in C:\Python27\ArcGIS) and that I removed weeks ago. So, those keys were cleaned but the main hive remained.
As I expected, ArcGIS Server relies on Python for Geo Processing services execution, and it looks like it is referencing the Python interpreter using the "PythonPath" key found in the hive.

So you just have to set its value according to your Python environment.

That's all: simple to say but damn hard to spot...

Hope this can be helpful for others!

[LisaCasey]

Thank you so much for this post! I was having the exact same problem, and your advice really helped me so much.

[FrancoisRousseau]

Thanks for sharing. Helped me a lot!

[JuliaDalton]

How did you reset the "PythonPath" value?
I'm having this same issue!

Thanks,
Julia

[ChuckWright]

You just saved me hours!  Thank you!

[DylanKennard]

Thank you so much I have been fighting this for 2 days now.  Changing permissions, double checking etc.  Need to check the Server log more often and quicker!

To answer the question on where to change the Hive use regedit then go HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\PythonPath as stated before.  Click on the default key and change the Values of the data.  To something like C:\Python27\ArcGIS10.1\Lib;C:\Python27\ArcGIS10.1\DLLs;C:\Python27\ArcGIS10.1\Lib\lib-tk  I was getting the issue for 2 reasons.  I originally installed ArcServer on the E drive and it appears it wrote the python key to my system registry. All the file paths seen above were pointed to the E drive. Due to my failed ArcServer install on the E drive I moved it to the C Drive.  I also install Desktop which installs Python 2.7 but made sure to use the same folder so it would merge the receptive files and libraries. This still caused me other python scripting issues (i.e. failed to import ArcPy) before I even arrived at this problem.  I also installed Python from Python.org for reasons to troubleshoot something else before this problem which causes other problems.

To solve this specific problem I confirmed the "old E" file paths and hot swapped out the E for C and restarted all the Geo processing Services.  Then instantly the geoprocessing service I built on my own finally published and started.

In essence it would be best to double check and make sure this registry path matches a real python Library with ArcPy and all the other stuff.  You also may want to double check if you have multiple version of the Python IDE installed and which is the default program.  If the IDE that points to ArcPy is not default you will get a import arcpy failure.  That will be a good test.

In essence I continue to feel that ESRI should not install Python by default but ask you to point to an install already or have you install it in the typical location that is supplied by python.org.  It becomes a real headache when putting Python.org, Desktop, and Server on one computer.

[DavidMuthami]

This post is extremely useful. I have been able to resolve the same error while trying to publish a map document with parcel fabric data.

Thanks,

David Muthami

[MichelleMestrovich]

Thanks so much.  This totally solved my problem

[SeanTondre]

Thanks for posting this!