Portal for ArcGIS 10.7 issue: internal portal database does not appear to be running or accepting connections

[MwalimuKaranja]

Hi! I was running Portal for ArcGIS on a Windows Server 2012 R2 server for a while until up to a week ago when it suddenly went down. The Portal for ArcGIS Windows Service shows that the service is running but it is not possible to access the portal site on port 7443. I took a look at some of the logs and this is what I got from the Portal logs in the arcgisportal folder:

Failed to start the portal. The observer's beforeStart() function returned a failure.

Also had a look at the logs in the C:\Program Files\ArcGIS\Portal\framework\service\logs folder and got the following in the service-0.log:

<Msg time="2019-07-28T18:13:49,381" type="SEVERE" code="218010" source="Portal Admin" process="1592" thread="1" methodName="" machine="" user="" elapsed="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>

Got the following on the service-error-0.log:

Sun Jul 28 18:13:54.019 EAT 2019 java.lang.NullPointerException
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.observers.ConfigObserver.c(ConfigObserver.java:793)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.observers.ConfigObserver.e(ConfigObserver.java:900)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.observers.ConfigObserver.beforeStart(ConfigObserver.java:191)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.impl.ObserverManager.beforeStart(ObserverManager.java:159)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.impl.NodeAgent.start(NodeAgent.java:246)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.impl.Main.start(Main.java:83)

Might anyone have an idea how to solve this issue. What might I be doing wrong?

[deactivated-user-ChrisBeaudett]

We're having the same issue:  After stopping/starting the Portal for ArcGIS Windows service, not all java executables were started, the postgres.exe process is not running on the machine, and the portal (including the portaladmin endpoint) became unresponsive.  The java.exe process was started (I'm guessing this is the Windows service), but no javaw.exe processes.  Eventually the indexer javaw.exe process started (after a few minutes). 

The portal logs contained serveral instances of this error:

The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) ...

And the Windows Event Viewer had half a dozen PostgreSQL errors of two different variations...

1)

The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

pg_ctl: PID file "E:/arcgisportal/db/postmaster.pid" does not exist

2)

The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Is server running?

... both of which we've been told to ignore in the previous post.

The same conditions persisted after rebooting the machine.

As the local PostgreSQL db is installed as an Esri component of the portal and is software-managed without the usual PostgreSQL client applications, perhaps Esri could provide additional feedback on how to troubleshoot this issue.

[deactivated-user-ChrisBeaudett]

Update to my previous post...

We were able to get our portal operational again, but not ideal:  we were able to determine the flags passed to the postgres.exe command and started it in a command prompt window, and all started working again.  The command that we used is:

postgres.exe -D "E:/arcgisportal/db" -p 7654

where -D is the database directory and -p is the port number to listen on.  (You can see all flags by running 'postgres.exe --help')

Once we got pgsql running we tried bouncing the Portal for ArcGIS Windows service again, hoping that whatever condition existed to prevent pgsql from starting before was now resolved, but PostGreSQL still did not start.

We're running ArcGIS Enterprise v10.7.1.

I don't know if that additional info is meaningful enough to point the Esri folks to a possible resolution.  Any help with troubleshooting the pgsql component would be helpful to the user community.

[JonathanQuinn]

If processes don't start, it could either be a problem with that process, or something with the processes that starts those processes. When the Portal Windows service starts, the ArcGISPortal.exe process starts first. There is logic within that process to check whether the java.exe, postgres.exe, and javaw.exe processes are started. If they're not, it'll start them. Does the RAM usage of the ArcGISPortal.exe climb to more than 100 MB? If not, then that's a problem. You can look at the service logs under framework\etc\service to see if errors are written to the logs there.

[AndresEcheverri]

Hi All

Could you fix the problem?. We have a similar issue after IT applied a security patch for Windows. The Portal is not updating or showing up new created web services  (I can see them in ArcGIS Server) or updating Users or Groups. Users still can see the items, web maps and apps.

We are getting the same messages:

• The index service seems to be unavaliable. Please check the index service.
• The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).

We run the Reindex in the Portal Admin Directory but didn't fix the problem after checking that the numbers were not matching. The numbers are not matching after running ReIndex. 

We haven't rebuild the Index replacing the Folders waiting for ESRI Support answers. Not sure is this will create further issues.

We also have the same errors with Postgres and Portal in the Windows Event Logger as described by Mwalimu Karanja. 

Is there any solution found for this issue?.

Regards,

Andres

[JonathanQuinn]

I'm surprised that you are able to sign into Portaladmin if the logs are indicating the index service and database are not available. Those errors would indicate that tokens couldn't be generated, which are necessary for accessing Portaladmin. I'd continue working with Support, given the multiple issues being presented in the logs.

[AndresEcheverri]

Thanks Jonathan.

Finally, we solved the problem rebuilding the Index following this Technical Article suggested by ESRI Suport How To: Rebuild the index for Portal for ArcGIS 10.6.1 . We are running Portal 10.7. After rebuilding the Index, the numbers matched, and all the web services that were created and were available in ArcGIS Server showed up in Portal. 

Regards,

Andres

[DuongHB]

For me I just copy content from empty_index.zip and replace at C:\arcgisportal\index\nodes\0\indices (like Craig guide prior)

When the Portal service restart, it automatically change the name of the folders, for example from ago-groups to C_Qm8eEMTKmlBGiXrKmfdA

[Mandalorian]

@JonathanQuinn FYI - I also ran into this same issue and the same log messages in my 10.7.1 Enterprise. Recreating the index service didn't fix it. The issue for me was that my Portal VM's network adapter somehow became disconnected (unchecked) in vSphere. Once this was re-checked, the issue was resolved:

[udai]

I ran into the same issue  with Portal for ArcGIS 10.9.1 on windows server

<Msg time="2023-04-24T12:47:38,623" type="SEVERE" code="217003" source="Portal" process="12556" thread="1" methodName="" machine="XXXXXX" user="" elapsed="" requestID="">Error before starting configuration observer. null</Msg>
<Msg time="2023-04-24T12:47:38,626" type="SEVERE" code="217075" source="Portal" process="12556" thread="1" methodName="" machine="XXXX" user="" elapsed="" requestID="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>
<Msg time="2023-04-24T14:12:18,406" type="SEVERE" code="218010" source="Portal Admin" process="11832" thread="1" methodName="" machine="XXXXX" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>

Is there any way to manually rebuild the index for Portal version 10.9.1 ??

seems The procedure stated here

https://support.esri.com/en-us/knowledge-base/how-to-manually-rebuild-the-index-in-portal-for-arcgis...  works only with versions prior to 10.7.1