*Background: Windows Server 2016 running ArcGIS Server 10.5.1 Web Adapter run on a separate server. Publishing occurred via 10.3.1 ArcGIS Desktop clients.
So I've been getting these odd messages for a bit and I have been trying to trace it down. Outside of having some search engine spiders that occasionally hit my rest endpoints that no longer exist, I get this odd series of messages that start off like this:
| 09/09/2018 4:40 PM | 09/09/2018 4:36 PM | 9999 | SEVERE | | | com.esri.arcgis.discovery.ejb.ArcGISServiceException: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.ejb.impl.GenericServerBean.handleRequest(GenericServerBean.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164) at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:92) at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122) at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:221) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:174) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:136) at org.apache.openejb.server.ejbd.EjbRequestHandler.doEjbObject_BUSINESS_METHOD(EjbRequestHandler.java:238) at org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:129) at org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196) at org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71) at org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213) at org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66) at org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91) at org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.esri.arcgis.discovery.servicelib.AGSException: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.servicelib.impl.RemoteServiceImpl.handleRequest(RemoteServiceImpl.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy44.handleRequest(Unknown Source) at com.esri.arcgis.discovery.ejb.util.EJBBase.handleRequestBase(EJBBase.java:586) at com.esri.arcgis.discovery.ejb.impl.GenericServerBean.handleRequest(GenericServerBean.java:70) ... 30 more Caused by: java.lang.Exception: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.servicelib.impl.ServiceImpl.handleRequest(ServiceImpl.java:244) at com.esri.arcgis.discovery.servicelib.impl.RemoteServiceImpl.handleRequest(RemoteServiceImpl.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) ... 3 more Caused by: AutomationException: 0x0 - null at com.esri.arcgis.interop.NativeObjRef.nativeVtblInvokeNative(Native Method) at com.esri.arcgis.interop.NativeObjRef.nativeVtblInvoke(Unknown Source) at com.esri.arcgis.interop.NativeObjRef.invoke(Unknown Source) at com.esri.arcgis.interop.Dispatch.vtblInvoke(Unknown Source) at com.esri.arcgis.system.IRequestHandlerProxy.handleStringRequest(Unknown Source) at com.esri.arcgis.discovery.servicelib.impl.SOThread.a(SOThread.java:949) at com.esri.arcgis.discovery.servicelib.impl.SOThread.a(SOThread.java:713) at com.esri.arcgis.discovery.servicelib.impl.SOThread.run(SOThread.java:466) |
| 09/09/2018 4:40 PM | 09/09/2018 4:36 PM | 9527 | SEVERE | | | Error handling request to service 'Services/ServiceName.MapServer'. AutomationException: 0xc00cee3a - |
| 09/09/2018 4:40 PM | 09/09/2018 4:36 PM | 8001 | SEVERE | | | Instance of service 'Services/ServiceName.MapServer' failed to process a request. AutomationException: null |
| 09/09/2018 4:40 PM | 09/09/2018 4:36 PM | 100007 | SEVERE | anonymous | MapServer.HandleStringRequest | IMessage::ReadXML failed. |
This error only shows for a handful of services and appear to be random in nature as to which ones toss the error.
So the questions are:
1) Has anyone seen this before?
2) How have you fixed the issue when it came up?
So far, triage has included republishing the service and increasing the available resources to the service.
