topic Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart in ArcGIS Enterprise Questions

[SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

ClaudioSparpaglione — Fri, 18 Jan 2013 10:09:46 GMT

Hi,

I've been running ArcGIS Server 10.1 with SP1 on a Windows Server 2012 64-bit machine for months without any problems in publishing map services and geoprocessing services. I have an ArcMap 32-bit installation on the same machine of the server, which I use to view maps and publish services on the server.

Yesterday, this crazy situation happened to me: I tried to publish a new map service through ArcMap, but it was stuck and said "Not responding" for minutes, so I killed it. Tried again and again: no fortune. Then I examined the server log file, which in my machine is at the path "C:\arcgisserver\logs\<machine_name>\server" and I found lines like the following ones:

< ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. Utilities/PrintingTools.GPServer instance has crashed.</Msg>  < ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg>

The log file contained one crash-report line for each of the geoprocessing services running on the server.

I therefore tried to investigate: I opened the ArcGIS Manager web application and tried to restart the PublishingTools service (which comes bundled with the server): it took a lot of time, then the service correctly stopped but in the end this message popped up:

System.PublishingTools.GPServer: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset

I tried to restart all of the other geoprocessing (either System's and my custom ones), and for each of them I got similar messages printed on screen.
I then tried to restart all of the other services (map services, geometry services) and everything went fine!
So it seems to be the just an error on the geoprocessing services.

I further investigated by setting the server log's verbosity to DEBUG (that is: print everything!) and this is the full stack-trace of the exception:

<Msg time="..." type="SEVERE" code="8273" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg> <Msg time="..." type="DEBUG" code="9999" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">java.lang.Exception: System/PublishingTools.GPServer instance has crashed.     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:363)     at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:218)     at com.esri.arcgis.discovery.ejb.impl.GPServerSyncBean.init(GPServerSyncBean.java:47)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164)     at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122)     at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237)     at org.apache.openejb.core.stateless.StatelessInstanceManager.createInstance(StatelessInstanceManager.java:448)     at org.apache.openejb.core.stateless.StatelessInstanceManager.access$100(StatelessInstanceManager.java:72)     at org.apache.openejb.core.stateless.StatelessInstanceManager$BackgroundThread.run(StatelessInstanceManager.java:507) Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:      java.net.SocketException: Connection reset     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:209)     at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)     at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:124)     at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)     at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)     at $Proxy43.createService(Unknown Source)     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:340)     ... 22 more Caused by: java.net.SocketException: Connection reset     at java.net.SocketInputStream.read(SocketInputStream.java:168)     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)     at java.io.DataInputStream.readByte(DataInputStream.java:248)     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)     ... 28 more </Msg>

As additional information:

All firewall systems on my machine are fully deactivated

No recent modification has been made to the networking in my LAN

I installed a few Windows updates last week

Can anyone help me in solving the issue?

Another question: could this problem be tied to a "dirty" Python installation? I'm not sure that GP services rely on Python (I'm getting a Java exception!)

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

ClaudioSparpaglione — Fri, 18 Jan 2013 13:28:15 GMT

After a lot of tries and speculations, I finally managed to find a solution.

I discovered that in the Windows Registry the following hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7

was completely empty.

This hive was related to a standalone Python 2.7 installation that I made aside to ArcGIS's ones (32-bit for ArcMap and 64-bit for ArcGIS Server - both are in C:\Python27\ArcGIS) and that I removed weeks ago. So, those keys were cleaned but the main hive remained.
As I expected, ArcGIS Server relies on Python for Geo Processing services execution, and it looks like it is referencing the Python interpreter using the "PythonPath" key found in the hive.

So you just have to set its value according to your Python environment.

That's all: simple to say but damn hard to spot...

Hope this can be helpful for others!

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

FrancoisRousseau — Wed, 06 Mar 2013 03:59:58 GMT

Thanks for sharing. Helped me a lot!

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

JuliaDalton — Wed, 12 Jun 2013 15:00:07 GMT

How did you reset the "PythonPath" value?
I'm having this same issue!

Thanks,
Julia

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

ChuckWright — Wed, 11 Sep 2013 21:46:13 GMT

You just saved me hours! Thank you!

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

DylanKennard — Wed, 22 Jan 2014 17:40:33 GMT

Thank you so much I have been fighting this for 2 days now. Changing permissions, double checking etc. Need to check the Server log more often and quicker!

To answer the question on where to change the Hive use regedit then go HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\PythonPath as stated before. Click on the default key and change the Values of the data. To something like C:\Python27\ArcGIS10.1\Lib;C:\Python27\ArcGIS10.1\DLLs;C:\Python27\ArcGIS10.1\Lib\lib-tk I was getting the issue for 2 reasons. I originally installed ArcServer on the E drive and it appears it wrote the python key to my system registry. All the file paths seen above were pointed to the E drive. Due to my failed ArcServer install on the E drive I moved it to the C Drive. I also install Desktop which installs Python 2.7 but made sure to use the same folder so it would merge the receptive files and libraries. This still caused me other python scripting issues (i.e. failed to import ArcPy) before I even arrived at this problem. I also installed Python from Python.org for reasons to troubleshoot something else before this problem which causes other problems.

To solve this specific problem I confirmed the "old E" file paths and hot swapped out the E for C and restarted all the Geo processing Services. Then instantly the geoprocessing service I built on my own finally published and started.

In essence it would be best to double check and make sure this registry path matches a real python Library with ArcPy and all the other stuff. You also may want to double check if you have multiple version of the Python IDE installed and which is the default program. If the IDE that points to ArcPy is not default you will get a import arcpy failure. That will be a good test.

In essence I continue to feel that ESRI should not install Python by default but ask you to point to an install already or have you install it in the typical location that is supplied by python.org. It becomes a real headache when putting Python.org, Desktop, and Server on one computer.

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

DavidMuthami — Mon, 05 May 2014 16:24:45 GMT

This post is extremely useful. I have been able to resolve the same error while trying to publish a map document with parcel fabric data.

Thanks,

David Muthami

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

MichelleMestrovich — Tue, 15 Jul 2014 16:15:19 GMT

Thanks so much. This totally solved my problem

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

SeanTondre — Fri, 08 Aug 2014 00:11:53 GMT

Thanks for posting this!

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

SimonJackson — Wed, 11 Feb 2015 00:18:03 GMT

Thanks loads - this helped me also.

I did try updatingmy registry settings with those from another machine that had an identical ArcGIS for Server installed, but something did not work.

Easier option for me was to simply run a Repair on the ArcGIS for Server install - fixed everything up!

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

MalcolmJ — Mon, 14 Sep 2015 21:18:05 GMT

Thanks this solved my problem completely.

I have Server 10.3 and I had to set this as the value in the registry for PythonPath.

"C:\Python27\ArcGISx6410.3\Lib;C:\Python27\ArcGISx6410.3\DLLs;C:\Python27\ArcGISx6410.3\Lib\lib-tk"

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

ClaireBrill — Fri, 14 Apr 2017 14:43:39 GMT

So helpful! Thanks for the detailed question and solution.

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

GaryPrior — Fri, 11 Aug 2017 15:57:24 GMT

Thanks so much! Solved my same issue! Even on AGS 10.5

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

SandeepPastapure — Fri, 09 Mar 2018 11:59:47 GMT

If you are getting problem to start only GP services and using Windows Server 2008 R2 and Windows 7 then need to be install the patch.

download from this link https://support.esri.com/en/download/7576

Thanks.

Re: [SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restart

LisaCasey — Wed, 19 Jan 2022 21:18:52 GMT

Thank you so much for this post! I was having the exact same problem, and your advice really helped me so much.