High availability Portal/Server in Azure with Federated and Hosting server issues

[JYI]

We have followed arcgis-powershell-dsc/BaseDeployment-MultiMachine.json at master · Esri/arcgis-powershell-dsc · GitH...  to deploy the high available system in Azure. The system consists of 2 Web Adaptors, 2 Portal for ArcGIS, 2 ArcGIS Server, one ArcGIS Data Store, and One ARR 3.0 load balancer. The system is up and running. 

The wild card SSL certificate is installed on the Load Balancer. The public url registered with the certificate works well.

There is a problem with this federated server and hosting server though. 

Every time I can only make either Federated Server or Hosting server available, not both. I.e., once the server is federated successfully, I then choose this server as the Hosting Server and then Save, it will throw 502 Bad Gateway error and the Hosting Server cannot be added.

To fix this, I go into Portal admin ->  Federation ->  Servers ->  select the federated server -> Update the server role to be Hosting Server -> Update server, now the hosting server is good and the federated server becomes not valid anymore!

If I remove the Hosting Server from Portal, i.e., change the Hosting Server to be No Hosting Server, then the Federated server becomes valid again. 

From the surface, the issue seems just that the Federated Server and the Hosting Server cannot be valid at the same time.

Any ideas? Thanks!

[JonathanQuinn]

What is the error when you validate the federated Servers in Portaladmin? If you were to use the services or admin URL you use to federate and check the machines list for each Server site, does refreshing a couple times return the correct machine names each time?

[JYI]

The error:

502 - Web server received an invalid response while acting as a gateway or proxy server.

There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.

No, there are no errors with the machine names. I used the registered public server url to federate. 

[CodyMarsh]

Hello,

Would you be able to check the validation of your ArcGIS Relational Data Store from your ArcGIS Server Manager? If the Data Store is not validating or throwing an error then that may result in not being able to add the ArcGIS Server as a hosting server.

[JYI]

Yes, I have checked each time if the Relational Data Store is valid in ArcGIS Server, before federating a server and adding the hosting server. Same thing happened to the Relational data store though, that it is not stable either.  Not each time it shows valid. 

[CodyMarsh]

Interesting, since the Data Store is not stable this may be what is causing the hosting behavior. Are there any logs in the Server Manager that mentions the Data Store validation?

Also one additional question: What version of the software have you installed?

[JYI]

I am trying to find the logs in ArcGIS server for Data store validation. Nothing there for data store validation. The version is 10.6.1. People are saying this is SSL certificate issue as the errors are more related to 502 Bad Gateway? 

[JonathanQuinn]

If this was a certificate problem, then the Server could never be validated regardless if it was hosting or non-hosting.

When I've seen the 502 error in Azure, it was because I hadn't mapped the internal DNS to the external URL correctly. I doubt that's the issue in your case as you can likely reach the services URL and admin URL you're using within the federation. The LB/ARR, (which is returning the 502), should be able to handle a JSON response that contains a failure. I don't see a reason why if the Data Store can't be validated, it would return a 502. If the Data Store can't be validated, (for example, when validating the hosted server in Portaladmin, when validating the relational Data Store in Manager or directly through the Admin API), an error should definitely be logged in the Server.

[JYI]

I just tried to deploy the wildcard certificate to the trusted certificate store on each machine, bind the certificate to the port 443 on the load balancer and two web adaptors, and then import the certificate through sslcertificates -> importExistingServerCertificate  on each portal and server machines, still got the same issue: Federated server and Hosting Sever cannot be valid at the same time. The data store is always valid at this moment.

Could you explain more on this "When I've seen the 502 error in Azure, it was because I hadn't mapped the internal DNS to the external URL correctly. "? I would like to know the solution to it. 

Thanks. 

[JYI · Accepted solution]

Thanks all for help. 

The final solution is:

"

.  There are 2 causes of 502 Gateway errors

1. SSL certificate on the server being proxied is not trusted
   1. Fix is to import the certificate into the trusted cert store – can be done using Internet explorer or PowerShell
2. The request timeout in IIS needs to be adjusted

In IIS Manager, right click on the site and go to Manage Web Site -> Advanced Settings. In there, under Connection Limits you should see Connection Time-out.

"