topic Re: Connection to callback URL '[serivces url]' failed in ArcGIS Enterprise Questions

Connection to callback URL '[serivces url]' failed

norie — Wed, 02 Oct 2013 21:35:33 GMT

I've been getting this error:

Connection to callback URL 'http://[SERVER NAME]:6080/arcgis/services' failed. For more information, see the ArcGIS Server help topic �?Ports used by ArcGIS Server�?. You can access this topic in the table of contents by navigating to Administering ArcGIS for Server > Securing your ArcGIS Server site > Configuring a secure environment for ArcGIS Server.

I am managing the ArcGIS Server directly from the server itself. The server is hosting ArcGIS Server with GEP. The server had no problems last week, so not really sure what has happened since then.

The error initially started appearing during service publishing this week. Failing with "Failed to Execute (UploadServiceDefinition)" and "Failed to Execute (Publish Service Definition)" in ArcMap and the Connection to callback failure message on the server logs. Along with this Exception/StackTrace below.

Since this instance wasn't hosting anything of value, I mistakenly decided to reinstall. Now I get the Connection to callback url failure message during the site creation.

Any ideas on how to fix this?

com.esri.arcgis.discovery.servicelib.AGSException: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.catalog.RemoteCatalog.handleRequest(RemoteCatalog.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148) at com.sun.proxy.$Proxy47.handleRequest(Unknown Source) at com.esri.arcgis.discovery.ejb.impl.ServiceCatalogBean.handleRequest(ServiceCatalogBean.java:105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164) at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:92) at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122) at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:221) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:174) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:136) at org.apache.openejb.server.ejbd.EjbRequestHandler.doEjbObject_BUSINESS_METHOD(EjbRequestHandler.java:238) at org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:129) at org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196) at org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71) at org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213) at org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66) at org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91) at org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.Exception: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.catalog.Catalog.handleRequest(Catalog.java:147) at com.esri.arcgis.discovery.catalog.RemoteCatalog.handleRequest(RemoteCatalog.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) ... 3 more Caused by: AutomationException: 0x0 - null at com.esri.arcgis.system.Message.readXML(Unknown Source) at com.esri.arcgis.discovery.catalog.Catalog$a_.a(Catalog$a_.java:376) at com.esri.arcgis.discovery.catalog.Catalog$a_.a(Catalog$a_.java:351) at com.esri.arcgis.discovery.catalog.Catalog$a_.run(Catalog$a_.java:288)

Re: Connection to callback URL '[serivces url]' failed

MorganHarvey — Thu, 03 Oct 2013 17:42:44 GMT

Aaron:

I'm running into what looks like the exact same problem on two of my servers.

I've disabled selinux, puppet, and iptables on that server and our networking guys confirm that the port is open (and that I can get to machine:6080 to get into the manager helps illustrate that also), so I'm rather bumfuzzled. Did two completely fresh installations of AGS and still no dice.

We've opened a support request and I'll update here with anything we figure out.

Morgan Harvey
GIS Application Developer
Portland State University

Re: Connection to callback URL '[serivces url]' failed

MorganHarvey — Mon, 07 Oct 2013 20:47:55 GMT

Aaron:

I've resolved my version of this, at least.

What wound up fixing it was me removing and reinstalling the 'X Window System' group package, which I did out of pure desperation. I'm not sure what, exactly, happened, but perhaps a package was updated since it was initially installed. (I did not to an update first to see what was different though)

Hope that helps,
-morgan

Re: Connection to callback URL '[serivces url]' failed

norie — Wed, 09 Oct 2013 22:38:15 GMT

Aaron:

I've resolved my version of this, at least.

What wound up fixing it was me removing and reinstalling the 'X Window System' group package, which I did out of pure desperation. I'm not sure what, exactly, happened, but perhaps a package was updated since it was initially installed. (I did not to an update first to see what was different though)

Hope that helps,
-morgan

Thanks for the update, Morgan. Unfortunately, I'm having this problem on a Windows 2008 R2 server.

Re: Connection to callback URL '[serivces url]' failed

norie — Thu, 10 Oct 2013 03:45:31 GMT

A little update�??

I changed the ArcGIS Server Account to my own admin account and everything worked as intended. I attempted to revert back to the AD service account (with local admin privs), but service requests started failing again.

The service account that ArcGIS Server was running under is local admin and was working previously. As far as I know, nothing has changed and I�??ve logged in as the service account interactively to verify it�??s rights.

The challenge now is getting the AD service account working again.

Re: Connection to callback URL '[serivces url]' failed

StephanieSnider — Mon, 28 Oct 2013 17:35:05 GMT

aaronnoriega,
Were you able to resolve this error message on Windows 2008 R2 server and still use the AD service account? I'm getting the same error message along with a couple more:

ERROR_INTERNET_SEC_CERT_CN_INVALID, URL = https://servername:6443/arcgis/services, HTTP Status code = 0 and Status Text = (WinInet error code = 12038)

Error while using HTTPS security, URL = https://servername:6443/arcgis/services, HTTP Status code = 0 and Status Text = (WinInet error code = 12057)

Re: Connection to callback URL '[serivces url]' failed

norie — Tue, 29 Oct 2013 18:37:52 GMT

I have a ticket open with ESRI, so it's still in progress. I have verified that AGS works with an alternate account and rather than recreating the AD account I'd like to narrow down the problem should the issue arise again. Especially since my environment hasn't been pushed to a production state.

However, from the looks of you error you have an SSL certificate with a bad common name. Check out these links:

http://resources.arcgis.com/en/help/main/10.2/index.html#//0154000005wq000000

http://resources.arcgis.com/en/help/main/10.2/#/Suppressing_warnings_from_self_signed_certificates/0154000005q6000000/

Re: Connection to callback URL '[serivces url]' failed

StephanieSnider — Tue, 29 Oct 2013 19:16:16 GMT

We've created organizational trusted certificiates using the server name as well as the server name with the domain. Both are throwing back the same errors. The web browser doesn't have a problem with the certificate. I can access manager and rest services fine. However, ArcGIS Server will not start the Printing Tools service and gives these errors. We are using the same setup method in production (where we are getting the errors) that we did in development and test. The trusted cert without domain worked fine in dev and test. I suspect its a blocked port on the DMZ where our production server is located. I sent the list of required ports to our IT dept, but I haven't confirmed with them that they actually did open those ports - and I don't have access to that server. I'll keep trying. Let me know if you hear anything good back from ESRI tech support.

Re: Connection to callback URL '[serivces url]' failed

StephanieSnider — Tue, 05 Nov 2013 22:39:00 GMT

Just to followup. For my situation, we created a new site and restored the directories. This fixed the error messages. ESRI said that something must have been corrupted during the initial site creation.

Re: Connection to callback URL '[serivces url]' failed

BenRay — Sun, 29 Dec 2013 22:05:36 GMT

I am also getting this error on an Ubuntu 64 bit 12.04 machine..(I know this is not on the approved OS list). However, I am getting this error during site creation.

Connection to callback URL 'http://servername:6080/arcgis/services' failed. For more information, see the ArcGIS Server help topic �?Ports used by ArcGIS Server�?. You can access this topic in the table of contents by navigating to Administering ArcGIS for Server > Securing your ArcGIS Server site > Configuring a secure environment for ArcGIS Server.

It is on ArcGIS 10.2 Server for Linux.

Any fix on this yet?

Re: Connection to callback URL '[serivces url]' failed

VadivelanPalanichamy — Mon, 11 Aug 2014 13:55:36 GMT

Hi, is there any solution for this call back error, other than repairing and re-creating the site. ?? (in windows server)

Re: Connection to callback URL '[serivces url]' failed

BubbaHey — Tue, 12 Aug 2014 14:05:26 GMT

What operating system? Check Firewall - and make sure ports are open, especially port 6080

See:

Ports used by ArcGIS Server

ArcGIS Help (10.2, 10.2.1, and 10.2.2)‌

Re: Connection to callback URL '[serivces url]' failed

VadivelanPalanichamy — Tue, 12 Aug 2014 14:40:33 GMT

I am also facing same issue, everything working fine with admin account, but not with service account.. anyone got fix or workaround (without doing repair or re create the site) for this issue ? (windows 2008 r2, server 10.2)

Re: Connection to callback URL '[serivces url]' failed

KennethOGuinn — Fri, 29 Aug 2014 21:50:56 GMT

Hello,

When a callback URL error is being thrown, it is because the ArcGIS Server itself cannot connect back to itself on the URL specified in the error.

This can be caused by a number of things, but the best method I have found for troubleshooting this particular error is to run your browser as the user that is running your ArcGIS Server service. Then browse to the URL in the error (Linux or Windows.) The reason why this is helpful is this allows you to see what the ArcGIS Server would see when connecting to the ArcGIS Server URL specified in the error message.

For example I have seen two things cause this error in the last month, though there could be many more reasons in the same way that many things can network connections to fail for specific users. Both cases were diagnosed using and resolved using the above method. So far I have yet to see this error caused by a bad install or corrupt data, instead it's been environmental.

1) The proxy settings for the service account running the ArcGIS Server Service did not have the same proxy settings as other users on the domain applied to it. (Normal users were setup to have exceptions for local addresses, but the service account was not.
As a result, the proxy was interfering with the ArcGIS Server reaching its own address while other users could browse the manager page normally. So when attempting to start/stop a service, the callback URL error would be thrown as the server attempts to connect to its HTTP/HTTPS listener.
To change this we ran Internet Explorer as the service account. In Windows Shift+Right click Internet Explorer to bring up the pop up menu to allow us to select "Run as different user." (In Linux you could "su - " to the account running your service, and run your preferred browser.)
Since this case was Windows, we Opened up: Tools > Internet Options > LAN Settings > Advanced, then set our proxy settings to match the settings of the other users. In this case adding an exception for the URL of the ArcGIS Server.

2) The account running the ArcGIS Server Windows Service could not download the Certificate Revocation list for the certificate being used in ArcGIS Server.
So the effect was users could browse the manager page via https and could check that the CRL was fine and the certificate had not been revoked. However, when the account that ArcGIS Server was running under tried to check the CRL, it was not allowed to download the CRL as a result of Group Policy. When ArcGIS Server is setup to use HTTPS, if it cannot verify a Certificate Revocation list (assuming the certificate has one,) it assumes to error on the side of caution and not make the connection to the server with an unverifiable certificate.

Two additional tests to help resolve this issue:

1) Rerun the Configure ArcGIS Server account to have the ArcGIS Server run under your domain account, this way you know AGS has the same settings as you in terms of what you can browse.

2) If you are getting Callback URL errors that reference 6443. You can tell if it's HTTPS related by switching the HTTPS settings for ArcGIS Server back to HTTP only. If the problem goes away, it is something related to your HTTPS/Certificates.

Additionally, issues with HTTPS/SSL will also tend to have to be followed by (WinInet error code = xxxxx) If you google the WinInet error code, it often will tell you what is the problem .

I hope this information helps!

Re: Connection to callback URL '[serivces url]' failed

LakshmananVenkatesan — Mon, 10 Aug 2015 18:34:42 GMT

Hi, I am facing same issue in ArcGIS server 10.3.1 . Any additional clues or directions?

Re: Connection to callback URL '[serivces url]' failed

KennethOGuinn — Mon, 10 Aug 2015 19:35:38 GMT

Hi S R,

Did you try switching the account running the ArcGIS Service to your domain account? If your user account can access the server via 6443, if you run ArcGIS Server under those same credentials it should also be able to reach the callback URL.

I'd double check with your IT folks to make sure this is ok to do as a test, but I suspect you'll find it fixes it. (Conversely, if you were to log in as the account running the ArcGIS Server Service, I suspect you will see that if you browse to http://server.domain.com:6080/arcgis/manager that it will not open the page.)

Hope this helps,

Ken O.

Re: Connection to callback URL '[serivces url]' failed

LakshmananVenkatesan — Mon, 10 Aug 2015 19:45:36 GMT

Ken, I have created new thread and provided my results and observation https://community.esri.com/message/542623#542623 Summary: a) if ArcGIS server service running with local admin account, services are working fine in HTTP and HTTPS mode. b) if ArcGIS server service running with Service account (the account specifically created for running services like ArcGIS server, FME and others softwares) does not work in HTTPS mode. The call back error is thrown. I need to run ArcGIS server using SERVICE account because this account has privilege to connect to internet, which is must for running my Geoevent processor which connects to third party data. It looks like Service account not able to make a call back in HTTPS mode for some reason. How to find what was exact reason for the call back issue on HTTPS mode for some accounts? I tried deleting site and recreated ; recreated new self signed certificate.

Re: Connection to callback URL '[serivces url]' failed

KennethOGuinn — Mon, 10 Aug 2015 19:53:55 GMT

Hi S R,

Probably the easiest way to see what is happening is to run a web browser as the service account, and then open up https://server.domain.com:6443/arcgis/manager.

This will let you see what the AGS sees when it tries to reach the callback URL.

It could be a few different things, but it does sound like the service account is differently configured from the regular users.

Below is a list of things I've seen cause the issue, though there are probably many other things that could cause a problem:

- Service account does not have proxy configured, but regular account does. (You can check this by running Internet Explorer as the service account, then Checking the Internet options for that Service Account.)

- Service account does not have the Domain CA that sighed the server cert in it's trusted CAs. (you can check this in certmgr.msc if it's in Windows.)

- The trust settings have an excpetion for http://server.domain.com but not https://server.domain.com

Hope this helps!

Ken O.

Re: Connection to callback URL '[serivces url]' failed

LakshmananVenkatesan — Mon, 10 Aug 2015 20:05:57 GMT

Ken,

I will check all these and get back to you. I need to take a help of IT team for this to test.

Re: Connection to callback URL '[serivces url]' failed

LakshmananVenkatesan — Mon, 10 Aug 2015 20:45:06 GMT

Ken, Would you mind explaining Point 3 and 4 little in detail. a) how to check domain CA in trusted CA? b) Do you mean Trust settings in IE?. how to check?. Generally many of IE settings are controlled by IT admin folks.