Hi
In AWS Ive deployed ArcGIS Enterprise 10.8.1 using Windows in HA and using ELB, both templates available here.
It works on first deployment and successful CloudFormation create-stack.
However when stopping/restarting Portal service on a given instance, or rebooting the Portal instances (either of them), they fail to restart Portal service, citing internal database not started properly, and symptoms like passwords rejected at /portaladmin login.
Here's one instance's set of logs:
Windows EventViewer
PostgreSQL : The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. (repeated, initially Information class, but further back is Error class)
Portal: The description for Event ID 5 from source Portal for ArcGIS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. (repeated, initially Information class, but further back is Error class)
"C:\Program Files\ArcGIS\Portal\framework\service\logs\service-error-0.log"
Mon Aug 24 13:01:13.237 UTC 2020 Exception in thread "I/O dispatcher 17"
Mon Aug 24 13:01:13.237 UTC 2020 java.lang.NoSuchMethodError: org.apache.http.ConnectionClosedException: method 'void <init>()' not found
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
Mon Aug 24 13:01:13.237 UTC 2020 at java.base/java.lang.Thread.run(Unknown Source)
"C:\arcgisportal\logs\10.72.4.42\portal\portal-20200824.123347-12940-0.0.log"
<Msg time="2020-08-24T12:34:34,908" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:34:34,908" type="WARNING" code="218012" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Error at check and update urls. The portal is currently unavailable. Please contact your portal administrator.</Msg>
<Msg time="2020-08-24T12:35:12,375" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:35:18,695" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:35:22,903" type="WARNING" code="218014" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Starting Index Service.</Msg>
<Msg time="2020-08-24T12:36:28,437" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:36:49,886" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:37:44,483" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:38:09,93" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:38:41,750" type="WARNING" code="218015" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Started Index Service.</Msg>
<Msg time="2020-08-24T12:38:41,828" type="WARNING" code="217060" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The database server was found to be stopped. Re-starting it.</Msg>
<Msg time="2020-08-24T12:38:51,974" type="WARNING" code="217064" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The web server was found to be stopped. Re-starting it.</Msg>
<Msg time="2020-08-24T12:43:09,108" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:12,517" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:15,819" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:20,950" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:31,158" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T13:01:13,481" type="WARNING" code="217068" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Error in the process monitor watchdog thread. sleep interrupted</Msg>
"C:\arcgisportal\logs\database\pg_log\postgresql-2020-08-24_060707.log"
2020-08-24 05:38:42 PDT: [6272]: LOG: database system was shut down at 2020-08-24 05:32:36 PDT
2020-08-24 05:38:42 PDT: [5824]: LOG: database system is ready to accept connections
2020-08-24 05:38:42 PDT: [10060]: LOG: incomplete startup packet
2020-08-24 05:38:42 PDT: [10180]: FATAL: no pg_hba.conf entry for host "::1", user "arcgis", database "arcgis", SSL off
2020-08-24 05:39:12 PDT: [12016]: LOG: incomplete startup packet
2020-08-24 05:40:12 PDT: [12196]: LOG: incomplete startup packet
2020-08-24 05:41:12 PDT: [12120]: LOG: incomplete startup packet
2020-08-24 05:42:12 PDT: [11232]: LOG: incomplete startup packet (repeated)
2020-08-24 05:42:33 PDT: [11348]: LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. (repeated)
Thanks for any help.
ArcGIS Enterprise Admin Professional, AWS Solutions Architect Professional
