Federated server unavailable

[Dixie_MDavis]

Good morning, all.

Yesterday our power went out while we were running the webgisdr utility on our Enterprise 10.6.1 system. The webgisdr
process had not completed - it was combining the individual backups from the component software pieces (Portal, Server and Data Store) when the VMs were shut down. We have three VMs participating in the site - Portal and AGS are running on one server, the Data Store is on a separate server and the web adaptors are on another server running IIS. The AGS is federated with the Portal and acts as its hosting server. We are only using the relational data store. We do not have a great deal of content in the Portal, but do have a nearly completed web app, map services and hosted feature services.

After the power went out it flickered, came back on for a 1/2 hour and then went out again. About an hour later it
eventually come back up. After the power was stable, we tried to re-run the webgisdr utility to create a new backup.
The webgisdr failed and reported that it was unable to connect to the ArcGIS server. We refreshed the NT services for
all the software and even restarted the servers, but since then our Portal and ArcGIS server are not functioning
correctly.

Our current state is that the Portal is running, we can login, but if we try to view any hosted services it reports that
the layers are unavailable. The Portal's Organization page indicates there is a problem with the federated server as
there is a red exclamation point next to the AGS' url. The Portal administrator site does report the correct details
about the AGS server when clicking Home > Federation > Servers. But, when we click "Validate" it reports a failure
status and provides a Java exception that indicates a 403 status code when trying to access the AGS server. Attempting
to access the AGS Manager site reports that the server is not currently participating in an ArcGIS Server site. Trying
to access the REST end point for the AGS server on the host, returns a 503 status with the following error: "ArcGIS
Server Site is not initialized."

Regarding logs, no logs have been created for AGS (in the content directory\logs\<hostname>\server\) since yesterday
when the power went out. The last statements written were "Verify machine registration observers: Disconnecting from
site". The Portal logs (from the content directory) report a 403 error for the URL: https://<hostname>/<ags
wa>/admin/data/findItems?managed=true...

There are no tomcat logs in the framework\tomcat location for AGS and the only thing written in the latest Portal
framework\tomcat log (catalina.2019-10-31.log) is 31-Oct-2019 08:52:17.314 SEVERE [https-jsse-nio-7443-exec-1]
com.esri.commons.web.rest.providers.BaseProvider.getValue Token Required.

Does anyone know if there is a way to get our AGS server back and re-established as our federated server? We do not have any existing Enterprise backups. We do have backups of our VMs from early last week, but before we go to a great deal of effort to restore the system and lose work, we want to find out if there is another way to restore the AGS server and its relationship to the enterprise.

Thanks for sticking with me if you have made it to the end of this long post and we appreciate any insight folks may
have.

Best, Dixie.