Validate servers failing in Portal

[ClintonBallandis1]

Hi,

When I attempt to validate our federated servers in Portal (Enterprise 10.5.1) I'm getting red exclamation marks indicating that it is failing. When I log into the Portal Admin and do a validate the status of Failure is returned on both Servers with the Following Message Error: java.lang.Exception: Server returned status code 500. If I go and inspect the Log files in the Portal Administrator I get the following Warning Validation failed for Hosting Server  Codes 207053 and  Warning validation for federated server failed 207051. These failures have only started happening recently.

I can still access the server managers for both ArcGIS Servers via the Portal without any issues, however I can no longer publish from ArcGIS Pro or created hosted featurelayers from Portal. If I try and add a shapefile from my computer I'm getting ERROR Publish exception 'Exception: Internal Server Error'. 

We have an externally signed SSL certificate on the Web Adaptor and ESRI Self Signed Certificate on the individual server components. 

I'm wanting to get this sorted before upgrading our environment to 10.6.1.

Any help would be appreciated

Thanks

Clinton

[JonathanQuinn]

Are you able to reach the Admin API via the URL you used to federate? Can you validate the ArcGIS Data Store manually? There's a specific URL that's likely returning the 500 but there are a few requests that are made when validating a federated Server, so it's tough to know which that is without getting Fiddler or similar to intercept the traffic.

[BillBott]

You said this only started happening recently - do you know of any system or environment changes that occurred? 

You can access all of these endpoints without error?

• <server host>/<server web adaptor>/manager e.g. https://yourservermachine.yourdomain/server/manager/
• <server host>/<server web adaptor>/admin https://yourservermachine.yourdomain/server/admin/ (may prompt for token, or server admin login)
• <server host>:6443/arcgis/manager (e.g. https://yourservermachine.yourdomain:6443/arcgis/manager/
• <server host>:6443/arcgis/admin (e.g. https://yourservermachine.yourdomain:6443/arcgis/admin/

Since the error is in java, that sounds like it's coming from ArcGIS Server instead of the Web Adaptor, what does your ArcGIS Server log say? You tried publishing to server direct (6443) or through WA? 

[ClintonBallandis1]

Hi Jonathan,

If I try and reach the server using the manager URL: eg.https://gisserver.domain.com:6443/arcgis/manager I get the following:

URL: webadaptor.domain.com/arcgis/sharing/oauth2/autorize

If I try and reach the server using the admin URL e.g https://gisserver.domain.com:6443/arcgis/admin I taken to the ArcGIS Server Directory.

 

If I try and generate a portal token for https://gisserver.domain.com:6443/arcgis/admin  I get the following message:

URL: webadaptor.domain.com/arcgis/sharing/generateToken

If I run fiddler then log into the Portal and select My Content I'm getting a 500 error

https://webadaptor.domain.com/arcgis/sharing/proxy?https://webadaptor.domain.com/server/admin/data/f...=US304..................

If I go to the server manager using webadaptor.domain.com/server/manager I am able to successfully login in and validate the datastore.

If I go to the server admin webadaptor.domain.com/server/admin I am able to generate a token through the ArcGIS Portal and subsequently log into the server adim directory.

The ArcGIS Server logs aren't returning anything regarding log in or authorization errors.

When my colleague tries to log in to the Portal he is getting the following error

With regard to system changes we recently renewed our licences and I've had firewall rules implemented to block external access to any admin pages, I'm now wondering if a rule is now blocking internal traffic?

Thanks,

Clinton

[JonathanQuinn]

I agree with Bill Bott‌, that's not an error thrown from Server. I'd speak with your IT staff on sorting out firewall rules.

I'd also discourage unfederating. If you need to update the Admin URL of the federated Server/hosting Server, do it through the Sharing API:

1) Sign into the Sharing API as an administrator (https://portal.domain.com/webadaptor/sharing/rest/)

2) Click the Org ID link 0123456789ABCDEF

3) Scroll to the bottom and click the Servers child resource

4) Click on the Server ID you need to change

5) Click Update Server

6) Enter the new Admin URL.

Note that the services URL can be changed, but there are a lot of other places you'd need to update, ex operational and basemap layers in webmaps, any items referencing the old server URL. Hosted services would have to be republished.

[BillBott]

It's a guess, but it looks like the rules got locked down on port 6443 perhaps for everything but the Web Adaptor machine. Or, if the Web Adaptor is on the same machine as ArcGIS Server then it would likely be unaffected by the changes.

So I would try....

1. Check firewall rules  - specifically, HTTPS traffic on port 6443 is permitted internally. Keep everything else as is. Access to that port is required to publish (either directly, or through the Web Adaptor) or....
2. Unfederate Server and re-federate through Web Adaptor (both URLs go through Web Adaptor instead of one going to 6443).

However, be aware if you choose to unfederate then everything that is published may have to be to be republished and you must have administrative access enabled on your Web Adaptor.  If those are not to big of a deal, I'd go with unfederate. You have the firewall tightened up and access will be through one port, not two.

[ClintonBallandis1]

Hi,

Thank you for the responses so far. I've had someone look at the firewall and I can now hit the following URL's

https://gisserverdomain.com:6443/arcgis/admin

and

https://gisserverdomiain.com:6443/arcgis/manager (note trying to access this URL gets stuck on please wait icon)

Unfortunately I can still not validate the servers. I did some further investigation and noticed that I can no longer access services through the  server manager via the web adaptor.

Everthing looks normal on the Server manager however when I go to view a service nothing is returned. I'm getting the following error in Chrome Dev tools  

I checked the server logs and I'm getting warnings regarding permissions and General geodatabase errors. This is odd as I have an admin account and should be able to access everything ?

Any ideas ? 

Thanks,

Clinton

[BillBott]

Hi Clinton, my first thought is to uninstall and reinstall the server web adaptor and make sure "allow administrative access" is checked when you do so. 

Admin access will be restricted to administrative accounts only, but that should work.

[JonathanQuinn]

When you federate Portal and Server, you'll only be able to use either the services URL or the admin URL to reach Manager. If you federate using the web adaptor URL for both, the internal machine name likely won't work and you'll see the scrolling bar indefinitely. The network traffic should have a bunch of generateToken requests and the response has an error about being unable to generate a token for this server. You may not be running into this but it's good to know.

The error does seem to be related to admin access, though, as Bill Bott‌ suggests. You don't need to uninstall and reinstall, though. You can either:

1) Re-register it and click the checkbox to allow admin access

2) Navigate to https://server.domain.com:6443/arcgis/admin/system/webadaptors/<id> and append update to the URL, then check the box to enable admin access

[ClintonBallandis1]

Hi I've checked the ArcGIS Server Admin Directory and the webadaptor is already configured to allow admin access.

Do I unregister the webadaptor here? Then  how do I re register ? Do I go to the server hosting the webadaptor and and just configure again ?

Thanks,

Clinton