topic Re: Unable to log into ArcGIS Server Manager after Federating in ArcGIS Enterprise Portal Questions

Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Fri, 02 Jun 2017 20:40:07 GMT

I have federated Portal 10.5 with a single-machine ArcGIS Server 10.5 site. Portal is configured to use Active Directory for its Identity Store. Portal and ArcGIS Server are on different virtual servers and their respective web adaptors are on a separate Web Server and configured under different IIS sites. I have done ArcGIS Enterprise Basic deployment in the past successfully.

This table summarized the configuration

URL	Description
https://ags-web-dev.mydomain.org/server	Web Adaptor pointing to the ArcGIS Server
https://ags-web.mydomain.org/portal	Web Adaptor pointing to the Portal for ArcGIS Site
https://ags-dev.mydomain.org:6443/arcgis	Direct URL to the ArcGIS Server machine
https://ags-portal.mydomain.org:7443/arcgis	Direct URL to the Portal for ArcGIS

Almost everything appears to work correctly. For example

I can log into Portal using the primary site administrator account
I can log into Portal using my Domain account
After Federating, all of the ArcGIS Server services appeared as Portal items, as expected
I can log into the ArcGIS Server Administrative service directory using the ArcGIS Server site administrator credentials, or by manually generating a Portal token.

The only issue if that attempting to open ArcGIS Server Manager fail. The interface hangs on the "Please wait..." progress bar and the network traffic shows repeated failures to POST to the portal generateToken page via the ArcGIS Server proxy

https://ags-dev.mydomain.org:6443/arcgis/manager/proxy?_proxyUrl=https%3A%2F%2Fags-web.mydomain.org%2Fportal%2Fsharing…

The network inspector shows that ArcGIS Server returned a 500 server error caused by a "Connection closed" (see screenshot).

Are there any good ways to go about diagnosing the root cause of an issue like this?

Update

Installing Fiddler and enabling HTTPS traffic snooping shows that request is being sent from the ArcGIS Server machine to the Portal Web Adaptor.

This may be a side-effect of enabling Fiddler as a MITM proxy, but the Portal logs show this WARNING:

ArcGIS Server services URL 'https://ags-web-dev.mydomain.org/server' cannot be validated against 'https://ags-web-dev.mydomain.org/server/rest/info'. If the service URL is a proxy URL verify it is accessible to clients.

The JSON at the /info endpoint is

{  "currentVersion": 10.5,  "fullVersion": "10.5.0",  "soapUrl": "https://ags-web-dev.mydomain.org/server/services",  "secureSoapUrl": null,  "owningSystemUrl": "https://ags-web.mydomain.org/portal",  "authInfo": {   "isTokenBasedSecurity": true,   "tokenServicesUrl": "https://ags-web.mydomain.org/portal/sharing/generateToken"  } }

What information is being used to attempt this 'validation'?

Re: Unable to log into ArcGIS Server Manager after Federating

JonathanQuinn — Fri, 02 Jun 2017 21:28:54 GMT

Hm, so I think that when either Server or Portal use their respective proxies, the request is actually made from the software itself as the user running the software's service. I'm more certain this is the case for the Portal going through the sharing proxy, so I'm guessing that it's the same for Server. So, if we assume that's correct, what you can try to do is reach the Sharing API through the Portal web adaptor on the Server machine when you're logged in as the account running the ArcGIS Server Windows service. That will tell you if the URL it's proxying to can be reached by the Server as the account running the service. Another thing you can try is to sign into the Portal website and then bring up Server Manager, as that'll automatically sign you in with the cookies already present in the browser.

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Sun, 04 Jun 2017 13:36:33 GMT

I'll try logging in as the ArcGIS Service account and see if that makes a difference.

Regarding logging into Portal first, that is actually how I discovered the issue. I had just Federated the server and clicked on the server link in the Portal Server setting page, which pointed to the server manager page.

I may not have verified that I was logged into Portal on subsequent tests, so I'll double-check that being logged into Portal does not change the behavior.

Thanks for the suggestions.

Re: Unable to log into ArcGIS Server Manager after Federating

CarolSousa — Mon, 05 Jun 2017 15:53:05 GMT

I have seen the "Please wait..." message hang, too. I believe it to be an issue with the browser. Do you get the "Not secure" message at the top of your browser? I'm using Chrome, and sometimes when I get that message I get the "Your connection is not private" message, then I go on to 'Advanced' and 'Proceed', then I get to Server Manager. When the browser hangs at "Please wait", I think it is due to that error message, except that it isn't presented to me to proceed around it. If I go back or refresh I'll usually get the privacy error, then I can move on.

Re: Unable to log into ArcGIS Server Manager after Federating

JonathanQuinn — Mon, 05 Jun 2017 16:00:48 GMT

If you're using the default self-signed certificate on 6443, that is actually related to Chrome losing the exception you've made for the self signed certificate. What happens is the following:

1) You try to reach Manager over 6443. Chrome sees that this is a self-signed certificate so you're asked if you want to proceed or not. You select Proceed.

2) Since you're trying to reach a federated Server, you're redirected to the Portal sign in page. This is where the initial exception for the self signed certificate is lost.

3) Once provide your credentials, you're redirected back to Server Manager. Since the initial exception has been lost, and for whatever reason Chrome doesn't prompt you to proceed again, the rest of the requests to Server Manager don't go through. You should see INSECURE_RESPONSE errors or something similar in the dev tools in Chrome.

This is reproducible outside of Server Manager and Portal as well. IE and Firefox don't have this problem.

Re: Unable to log into ArcGIS Server Manager after Federating

CarolSousa — Mon, 05 Jun 2017 16:12:19 GMT

I've got domain certificates on all the Enterprise machines. But I should use IE or Firefox. Thanks!

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Mon, 05 Jun 2017 16:17:33 GMT

I have seen the behavior you describe caused by self-signed or incorrect SSL certificate, however in this case we do have proper certificates installed on both the ArcGIS Server, Portal and Web Adaptors in IIS.

I have added a screenshot of the browser when connected to the Manager page with the SSL credentials shown. Please let me know if you see anything amiss in this settings.

The issue seems to be pretty clearly caused by a network connection being terminated when the ArcGIS Server proxy page attempts to connect to the Portal generateToken page via the Portal Web Adaptor. This behavior is confirmed by the ArcGIS Server error page that was captured from the Web Server network inspector, and by monitoring the traffic between the ArcGIS Server and Web Adaptors via Fiddler.

I am waiting on IT to grant remote login privileges to the ArcGIS Server domain account in order to test Johnathan Quinn's suggestion of verifying that the ArcGIS Server process identity is not being blocked due to external policy considerations.

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Wed, 07 Jun 2017 04:06:45 GMT

New information:

Logging in as the ArcGIS Server service account made no difference.
I created two python scripts on the ArcGIS Server, one that POSTs to the /proxy page and one that POST directly to the generateToken URL that is trying to be proxied. Each has appropriate headers and body content. The first script returns the same ArcGIS Server error page as shown in the network panel screenshot and the second succeed and returns a token from Portal.
For a few hours today things started working and I was able to log into ArcGIS Server Manager. The IT staff I'm in contact with stated that they had not made any changes. A few hours later and the error returned. Very strange.
Turning on DEBUG level logging in the ArcGIS Server doesn't show anything specific, but there are these entries that appear to be somewhat related to the issue at hand,
1. Token is not a valid Admin token. Trying portal token next. Token = 8_AptoTJ86nrEVK5hBnnfD-9C7LzVwrcQNoFgnLNkMh1PkFTV8ssh4EyaDOknlAYQmygXFCykSiYl2_xnp178FKqk2tNBc0tvQ8JpV05r5dFXvElHufzydxeMsIfImSk3QNklmn1obIstorWPo2s_U8GFhk4gq3LLUpu8KcClOo8f7vAhgta4C3d2Fl09OA0, referrer = https://ags-dev.mydomain.org:6443/arcgis/manager/Could not decrypt token. Token may not be valid.
2. ARCGIS_PORTAL_TOKEN Authentication, Token is not available in the request, request is treated as anonymous

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Fri, 09 Jun 2017 13:23:44 GMT

More information.

I’ve attached a network trace of the traffic captured between our ArcGIS Server and the Web Server that hosts the Web Adaptor for Portal. It appears that the Web Server (192.168.103.50) is sending a TCP Reset packet in response to the CLIENT HELLO send by the ArcGIS Server. This process repeats a few times and then, apparently, the ArcGIS Server side gives up.

We actually have three ArcGIS Servers set up right now (1 dev, 2 production) and the same error is happening in all cases, so it appears to be something systemic and specific to how the proxy page and/or tomcat/geronimo/Java is handling it's networking.

Might be time to write a test client in Java and see if that behaves the same as ArcGIS Sever....

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Fri, 09 Jun 2017 14:34:12 GMT

Getting closer to a resolution. It appears that this is likely an IIS/SSL issue related to certificate negotiation. The IIS Web Server we are using has an ECC certificate installed (versus the more common RSA signed certificate) and that appears to be problematic.

We had an issue when installing ECC certificates on ArcGIS Server, as well. Chrome and other broswers reported that they could not negotiate a common cipher and the connection failed. We ended up using RSA certificates instead. The same type of issue appears to be in play here.

Some relevant stackexchange posts on this issue

https://security.stackexchange.com/questions/96804/server-sends-rst-after-receiving-client-hello-when-binding-certain-certificate

https://serverfault.com/questions/774826/tls-1-2-client-hello-triggers-tcp-reset-from-2012-r2

Re: Unable to log into ArcGIS Server Manager after Federating

JonathanQuinn — Fri, 09 Jun 2017 17:43:08 GMT

Ah, ok, I have a basic understanding of certificates, not to the degree to know the difference between ECC and RSA. You may want to reach out to Technical Support so they can investigate whether there's a problem with a certain certificate type in the software.

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Mon, 12 Jun 2017 15:36:04 GMT

I have had an open support case concurrent with this thread and have cross-referenced customer support to this topic as well. Hopefully the specific failure mechanism will be found soon...

Re: Unable to log into ArcGIS Server Manager after Federating

RandallWilliams — Mon, 12 Jun 2017 17:25:15 GMT

Portal and server support ECDHE (Elliptic-curve Diffie Hellman) key exchanges. Is your listed here:

https://server.arcgis.com/en/server/latest/administer/windows/restrict-arcgis-server-ssl-protocols-and-cipher-suites.htm

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Mon, 12 Jun 2017 17:42:57 GMT

The original certificates that were installed on IIS were ECDHE_ECDSE which were not on the list. After discovering that discrepancy, IT installed new certificates that are ECDHE_RSA with P384.

My understanding is that the block cipher and message authentication are negotiated and, according to the Chrome Security toolbar, it can connect with AES_256_CBC_SHA1 (AES_256_CBC with HMAC_SHA1, specifically). That would appear to match the TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA entry in the list of supported ArcGIS Server ciphers, so our expectation was that ArcGIS Server would (at least) be able to use that cipher as well.

Unfortunately, the new certificate did not resolved the issue. So we are looking more closely at the details of the certificates and network traces to try and figure out where the TLS handshake breaks down.

I am certainly not a certificate/TLS/Networking expert by any means, so I may be misunderstanding the specifics of how this is supposed to work and any quirks of the Java networking stack when talking to Windows/IIS web servers.

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Tue, 13 Jun 2017 17:02:45 GMT

A workaround!

The actual issue appears to be a lack of support for SNI in the ArcGIS Server proxy. Our web server was set up with multiple sites, each bound to a FQDN. On a hunch, I added a default HTTPS binding to the site that contains the Portal Web Adaptor and things began working.

At one point I did have a default binding on the site in order to configure the Web Adaptor (so I could use https://localhost/ to get at the configuration page), which is why it worked for a while and then "stopped". At the time, I didn't associate removing the default binding with the loading failures.

There is an Esri BUG-000093827 that discusses an issue with the Portal proxy and SNI support that was fixed in 10.4.1, but nothing I could find that discussed the state of SNI support in the ArcGIS Server proxy.

I'm hoping to get a resolution from Esri support that either

Confirms this as a bug in the AGS proxy, or
Provides some details on how to enable SNI support

For now, moving on from this issue....

TLS Extension List sent from ArcGIS Server

extensions [extension_type: elliptic_curves,extension_type: ec_point_formats,extension_type: signature_algorithms]

TLS Extension List sent from Chrome Browser

extensions [extension_type: 31354,extension_type: renegotiation_info,extension_type: server_name,extension_type: extended_master_secret,extension_type: SessionTicket_TLS,extension_type: signature_algorithms,extension_type: status_request,extension_type: 18,extension_type: application_layer_protocol_negotiation,extension_type: 30032,extension_type: ec_point_formats,extension_type: elliptic_curves,extension_type: 10794]

Re: Unable to log into ArcGIS Server Manager after Federating

RandallWilliams — Tue, 13 Jun 2017 17:07:39 GMT

OK, yes. That makes sense.

There's a second issue that's fixed in 10.5 - [#BUG-000099854 A proxy call within Portal for ArcGIS does not support Server Name Indication (SNI): GET /arcgis/sharing/proxy?https://server.domain.com/arcgis/rest/services/Hosted/MyService/FeatureServer/0?f=json }

Re: Unable to log into ArcGIS Server Manager after Federating

MatthiasFries — Wed, 18 Oct 2017 09:29:00 GMT

Thanks for this workaround! I had the same issue but never thought that the default https binding could be the cause. I tried many things and contacted support without success, we have a similar setup with multiple sites each bound to a FQDN. Editing the default https binding with an EV SSL certificate solved this issue.

Re: Unable to log into ArcGIS Server Manager after Federating

RobertDriessen2 — Thu, 23 Nov 2017 10:45:17 GMT

Hi Lucas

Thank you for your post. Its the only thing I have found that comes close to solving my problem.

I am getting the exact same problem of not being able to access server manager after federation. I even get a log message which includes "If the service URL is a proxy URL verify it is accessible to clients".

I am having a frustrating time trying to solve this. Have you had a response from ESRI regarding your 2 points? We have two portals joined together in primary/secondary failover mode. Web adapters are on both servers but arcgis is the only site on these servers (so I assume that would make it default anyway). However we do have a f5 load balancer which directs traffic to the primary server if it is up or the secondary server if it is down. I am trying to work out if you fix applies to the load balancer or the servers with the web adapters?

Thanks again, Rob

Re: Unable to log into ArcGIS Server Manager after Federating

JonathanQuinn — Tue, 28 Nov 2017 18:41:49 GMT

I don't have an answer for the SNI problem, but just wanted to chime in and mention it's unnecessary to only send traffic to the primary server. The Web Adaptors are aware of the health of the portal machines and will balance requests to each machine.

Re: Unable to log into ArcGIS Server Manager after Federating

LucasScharenbroich — Mon, 11 Dec 2017 15:38:32 GMT

Rob,

Esri support did confirm the lack of SNI support in the ArcGIS Server proxy and filed BUG-000106097. It is listed as fixed in 10.6.

BUG-000106097: A proxy call within ArcGIS GIS Server does not suppo..