Connection to callback URL '[serivces url]' failed

[norie]

I've been getting this error:

Connection to callback URL 'http://[SERVER NAME]:6080/arcgis/services' failed. For more information, see the ArcGIS Server help topic �?Ports used by ArcGIS Server�?. You can access this topic in the table of contents by navigating to Administering ArcGIS for Server > Securing your ArcGIS Server site > Configuring a secure environment for ArcGIS Server.

I am managing the ArcGIS Server directly from the server itself. The server is hosting ArcGIS Server with GEP. The server had no problems last week, so not really sure what has happened since then.

The error initially started appearing during service publishing this week. Failing with "Failed to Execute (UploadServiceDefinition)" and "Failed to Execute (Publish Service Definition)" in ArcMap and the Connection to callback failure message on the server logs. Along with this Exception/StackTrace below.

Since this instance wasn't hosting anything of value, I mistakenly decided to reinstall. Now I get the Connection to callback url failure message during the site creation.

Any ideas on how to fix this?

com.esri.arcgis.discovery.servicelib.AGSException: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.catalog.RemoteCatalog.handleRequest(RemoteCatalog.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148) at com.sun.proxy.$Proxy47.handleRequest(Unknown Source) at com.esri.arcgis.discovery.ejb.impl.ServiceCatalogBean.handleRequest(ServiceCatalogBean.java:105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164) at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:92) at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122) at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:221) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:174) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:136) at org.apache.openejb.server.ejbd.EjbRequestHandler.doEjbObject_BUSINESS_METHOD(EjbRequestHandler.java:238) at org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:129) at org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196) at org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71) at org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213) at org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66) at org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91) at org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.Exception: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.catalog.Catalog.handleRequest(Catalog.java:147) at com.esri.arcgis.discovery.catalog.RemoteCatalog.handleRequest(RemoteCatalog.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) ... 3 more Caused by: AutomationException: 0x0 - null at com.esri.arcgis.system.Message.readXML(Unknown Source) at com.esri.arcgis.discovery.catalog.Catalog$a_.a(Catalog$a_.java:376) at com.esri.arcgis.discovery.catalog.Catalog$a_.a(Catalog$a_.java:351) at com.esri.arcgis.discovery.catalog.Catalog$a_.run(Catalog$a_.java:288)

[KennethOGuinn]

Hi S R,

You are right, these questions are really IT centric. In fact, it might be best to point your IT staff to this post so they can take a look.

The following talks about how to check certificates View or manage your certificates

Also, with regards to the trust, https://support.microsoft.com/en-us/kb/174360  shows a bit about it as well.

Long story short though, if you run your browser as the service account and it cannot connect via HTTPS, I would treat it as any other real user, and contact the IT dept. and let them know that there is an account that should be able to access the URL, but cannot. Then they can take it from there.

[AshleyPeters]

Ken,

I'm having a very similar issue on a Windows 2008 R2 server. I have a single machine setup, and my internal and external IP addresses are different from one another. When I run my print service, I receive WinINet error #12029, the attempt to connect to the server failed.

I've been trying to find a resolution for this for about 6 months. Could this be caused by the same problem?

Thanks!

[KennethOGuinn]

Hi Ashley,

I apologize for the delay, I just saw your message now!

It does look to be in the same vein.

Essentially the WinINet errors are errors thrown back up the stack to ArcGIS Server when the server asks the Operating System to make a network connection to something, and the OS says "hey, I tried to connect like you asked, but I couldn't and here is the best explanation I can give to you; do with it what you will" Then ArcGIS Server logs that error in its logs.

I see on https://support.microsoft.com/en-us/kb/193625

12029 ERROR_INTERNET_CANNOT_CONNECT

  The attempt to connect to the server failed.

This error is not very specific, but I suspect it is because its not even able to connect to the machine at all, so it can't give you a specific error as a result. Which is somewhat handy, because it implies you don't have a problem with SSL or proxies.

The best method to troubleshoot this is going to be to log into the machine as the account that is running the arcgis server. (This is critical because you really need to be able to test under the same conditions as arcgis server is running under, to truely see what is affecting it, and you can't get any closer than being the user that is running ArcGIS Server!)

Next you need to start checking why you can't connect. I apologize for some nuts and bolts network troubleshooting stuff, but there is no real way to avoid it.

You're going to want to check a few things, I wont go into too many details, but if this stuff is unfamiliar, see if you can point your local friendly IT person to this and see if they can help.

- Do a quick sanity check to make sure everything is the way you think it is. I use a combiation of the following two commands on windows.

---- ipconfig /all

---- Just to see what IP addresses you have and who you are using as your DNS server to resolve names to IP addresses.

---- netstat -aon

---- This will list the ports on windows.

---- If you see your Web Server is listening only on port 443, but running fiddler you see requests trying to resolve to 80, that would explain the 12029 error.

- Is my ArcGIS Server resolving the DNS name to the correct IP address?

---- nslookup <yourserver.yourdomain.com>

---- This command will let you compare the IP address ArcGIS Server think yourserver.yourdomain.com resolves to against what your actual ip address is from the other commands. If you find it's resolving to the wrong IP address, this is the problem and why it can't connect!

- try opening up a combination of the different URLs/ports/IP addresses to see if you can isolate which one isn't working, if you can find out which one isn't working, you can probably assume that is one that AGS is trying to use that causes the error.

- http://ags.yourdomain.com:6080/arcgis/rest

- https://ags.yourdomain.com:6443/arcgis/rest

- http://webadaptor.yourdomain.com:80/arcgis/rest

- http://webadaptor.yourdomain.com:443/arcgis/rest

- ping <yourserver.yourdomain.com>

- ping <both of your IP addresses>

---- This is really just a way to make sure you can reach your server at its name and IP address in general.

- What path is my machine using to to reach that IP address

---- tracert <yourserver.yourdomain.com>

---- This shows you the different IP addresses along the way you are going to. I dont find this too useful, but if you find it going places it shouldn't it might give you a hint as to the problem.

Hopefully that should help at least start you on your way. Lastly, if you have access to Esri Tech Support, please feel free to call them as they might have some additional insights as well.

Thanks and good luck!