Unable to log into ArcGIS Server Manager after Federating

[LucasScharenbroich]

I have federated Portal 10.5 with a single-machine ArcGIS Server 10.5 site.  Portal is configured to use Active Directory for its Identity Store. Portal and ArcGIS Server are on different virtual servers and their respective web adaptors are on a separate Web Server and configured under different IIS sites. I have done ArcGIS Enterprise Basic deployment in the past successfully.

This table summarized the configuration

URL	Description
https://ags-web-dev.mydomain.org/server	Web Adaptor pointing to the ArcGIS Server
https://ags-web.mydomain.org/portal	Web Adaptor pointing to the Portal for ArcGIS Site
https://ags-dev.mydomain.org:6443/arcgis	Direct URL to the ArcGIS Server machine
https://ags-portal.mydomain.org:7443/arcgis	Direct URL to the Portal for ArcGIS

Almost everything appears to work correctly.  For example

• I can log into Portal using the primary site administrator account
• I can log into Portal using my Domain account
• After Federating, all of the ArcGIS Server services appeared as Portal items, as expected
• I can log into the ArcGIS Server Administrative service directory using the ArcGIS Server site administrator credentials, or by manually generating a Portal token.

The only issue if that attempting to open ArcGIS Server Manager fail.  The interface hangs on the "Please wait..." progress bar and the network traffic shows repeated failures to POST to the portal generateToken page via the ArcGIS Server proxy 

https://ags-dev.mydomain.org:6443/arcgis/manager/proxy?_proxyUrl=https%3A%2F%2Fags-web.mydomain.org%... 

The network inspector shows that ArcGIS Server returned a 500 server error caused by a "Connection closed" (see screenshot).

Are there any good ways to go about diagnosing the root cause of an issue like this?

Update

Installing Fiddler and enabling HTTPS traffic snooping shows that request is being sent from the ArcGIS Server machine to the Portal Web Adaptor.

This may be a side-effect of enabling Fiddler as a MITM proxy, but the Portal logs show this WARNING:

ArcGIS Server services URL 'https://ags-web-dev.mydomain.org/server' cannot be validated against 'https://ags-web-dev.mydomain.org/server/rest/info'. If the service URL is a proxy URL verify it is accessible to clients.

The JSON at the /info endpoint is

{  "currentVersion": 10.5,  "fullVersion": "10.5.0",  "soapUrl": "https://ags-web-dev.mydomain.org/server/services",  "secureSoapUrl": null,  "owningSystemUrl": "https://ags-web.mydomain.org/portal",  "authInfo": {   "isTokenBasedSecurity": true,   "tokenServicesUrl": "https://ags-web.mydomain.org/portal/sharing/generateToken"  } }

What information is being used to attempt this 'validation'?

[JonathanQuinn]

Hm, so I think that when either Server or Portal use their respective proxies, the request is actually made from the software itself as the user running the software's service.  I'm more certain this is the case for the Portal going through the sharing proxy, so I'm guessing that it's the same for Server.  So, if we assume that's correct, what you can try to do is reach the Sharing API through the Portal web adaptor on the Server machine when you're logged in as the account running the ArcGIS Server Windows service.  That will tell you if the URL it's proxying to can be reached by the Server as the account running the service.  Another thing you can try is to sign into the Portal website and then bring up Server Manager, as that'll automatically sign you in with the cookies already present in the browser.

[LucasScharenbroich]

I'll try logging in as the ArcGIS Service account and see if that makes a difference.

Regarding logging into Portal first, that is actually how I discovered the issue.  I had just Federated the server and clicked on the server link in the Portal Server setting page, which pointed to the server manager page.

I may not have verified that I was logged into Portal on subsequent tests, so I'll double-check that being logged into Portal does not change the behavior.

Thanks for the suggestions.

[LucasScharenbroich]

New information:

1. Logging in as the ArcGIS Server service account made no difference.
2. I created two python scripts on the ArcGIS Server, one that POSTs to the /proxy page and one that POST directly to the generateToken URL that is trying to be proxied. Each has appropriate headers and body content.  The first script returns the same ArcGIS Server error page as shown in the network panel screenshot and the second succeed and returns a token from Portal.
3. For a few hours today things started working and I was able to log into ArcGIS Server Manager.  The IT staff I'm in contact with stated that they had not made any changes.  A few hours later and the error returned. Very strange.
4. Turning on DEBUG level logging in the ArcGIS Server doesn't show anything specific, but there are these entries that appear to be somewhat related to the issue at hand,
   1. Token is not a valid Admin token. Trying portal token next. Token = 8_AptoTJ86nrEVK5hBnnfD-9C7LzVwrcQNoFgnLNkMh1PkFTV8ssh4EyaDOknlAYQmygXFCykSiYl2_xnp178FKqk2tNBc0tvQ8JpV05r5dFXvElHufzydxeMsIfImSk3QNklmn1obIstorWPo2s_U8GFhk4gq3LLUpu8KcClOo8f7vAhgta4C3d2Fl09OA0, referrer = https://ags-dev.mydomain.org:6443/arcgis/manager/Could not decrypt token. Token may not be valid.
   2. ARCGIS_PORTAL_TOKEN Authentication, Token is not available in the request, request is treated as anonymous

[LucasScharenbroich]

More information.

I’ve attached a network trace of the traffic captured between our ArcGIS Server and the Web Server that hosts the Web Adaptor for Portal.  It appears that the Web Server (192.168.103.50) is sending a TCP Reset packet in response to the CLIENT HELLO send by the ArcGIS Server.  This process repeats a few times and then, apparently, the ArcGIS Server side gives up.

We actually have three ArcGIS Servers set up right now (1 dev, 2 production) and the same error is happening in all cases, so it appears to be something systemic and specific to how the proxy page and/or tomcat/geronimo/Java is handling it's networking.

Might be time to write a test client in Java and see if that behaves the same as ArcGIS Sever....

[LucasScharenbroich]

Getting closer to a resolution.  It appears that this is likely an IIS/SSL issue related to certificate negotiation.  The IIS Web Server we are using has an ECC certificate installed (versus the more common RSA signed certificate) and that appears to be problematic.

We had an issue when installing ECC certificates on ArcGIS Server, as well.  Chrome and other broswers reported that they could not negotiate a common cipher and the connection failed. We ended up using RSA certificates instead. The same type of issue appears to be in play here.

Some relevant stackexchange posts on this issue

https://security.stackexchange.com/questions/96804/server-sends-rst-after-receiving-client-hello-whe...

https://serverfault.com/questions/774826/tls-1-2-client-hello-triggers-tcp-reset-from-2012-r2

[JonathanQuinn]

Ah, ok, I have a basic understanding of certificates, not to the degree to know the difference between ECC and RSA.  You may want to reach out to Technical Support so they can investigate whether there's a problem with a certain certificate type in the software.

[LucasScharenbroich]

I have had an open support case concurrent with this thread and have cross-referenced customer support to this topic as well.  Hopefully the specific failure mechanism will be found soon...

[RandallWilliams]

Portal and server support ECDHE (Elliptic-curve Diffie Hellman) key exchanges. Is your listed here:

https://server.arcgis.com/en/server/latest/administer/windows/restrict-arcgis-server-ssl-protocols-a...

[LucasScharenbroich]

The original certificates that were installed on IIS were ECDHE_ECDSE  which were not on the list. After discovering that discrepancy, IT installed new certificates that are ECDHE_RSA with P384.

My understanding is that the block cipher and message authentication are negotiated and, according to the Chrome Security toolbar, it can connect with AES_256_CBC_SHA1 (AES_256_CBC with HMAC_SHA1, specifically). That would appear to match the TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA entry in the list of supported ArcGIS Server ciphers, so our expectation was that ArcGIS Server would (at least) be able to use that cipher as well.

Unfortunately, the new certificate did not resolved the issue.  So we are looking more closely at the details of the certificates and network traces to try and figure out where the TLS handshake breaks down.  

I am certainly not a certificate/TLS/Networking expert by any means, so I may be misunderstanding the specifics of how this is supposed to work and any quirks of the Java networking stack when talking to Windows/IIS web servers.