Portal for ArcGIS 10.7 issue: internal portal database does not appear to be running or accepting connections

MwalimuKaranja · ‎07-28-2019

Hi! I was running Portal for ArcGIS on a Windows Server 2012 R2 server for a while until up to a week ago when it suddenly went down. The Portal for ArcGIS Windows Service shows that the service is running but it is not possible to access the portal site on port 7443. I took a look at some of the logs and this is what I got from the Portal logs in the arcgisportal folder:

Failed to start the portal. The observer's beforeStart() function returned a failure.

Also had a look at the logs in the C:\Program Files\ArcGIS\Portal\framework\service\logs folder and got the following in the service-0.log:

<Msg time="2019-07-28T18:13:49,381" type="SEVERE" code="218010" source="Portal Admin" process="1592" thread="1" methodName="" machine="" user="" elapsed="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>

Got the following on the service-error-0.log:

Sun Jul 28 18:13:54.019 EAT 2019 java.lang.NullPointerException
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.observers.ConfigObserver.c(ConfigObserver.java:793)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.observers.ConfigObserver.e(ConfigObserver.java:900)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.observers.ConfigObserver.beforeStart(ConfigObserver.java:191)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.impl.ObserverManager.beforeStart(ObserverManager.java:159)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.impl.NodeAgent.start(NodeAgent.java:246)
Sun Jul 28 18:13:54.019 EAT 2019 at com.esri.arcgis.portal.impl.Main.start(Main.java:83)

Might anyone have an idea how to solve this issue. What might I be doing wrong?

CraigRussell · ‎07-28-2019

Hi Mwalimu

The first thing to check would be disk space, although Portal should include specific errors/warnings in the logs when this is becoming a problem.

I came across the 218010 error code last week, which in that instance turned out to be caused by corrupt indexes preventing the indexing service (and consequently Portal itself) from starting properly. While the indexing service javaw.exe process was visible in task manager, the Portal java.exe process wasn't running; do you see at least two Java processes?

Also, do you also see this warning coming up repeatedly in the portal logs?

<Msg time="2019-07-24T17:38:19,196" type="WARNING" code="218014" source="Portal" process="9980" thread="1" methodName="" machine="portal.domain.com" user="" elapsed="">Starting Index Service.</Msg>

Assuming this looks like your issue and you can login to the Portal Administrator Directory then you could try reindexing your portal by going to system > indexer > reindex with Mode=Full. If you can't access the administrator directory (I couldn't with my issue last week) then try the following:

Stop the Portal service
Backup your Portal index folder - e.g. C:\arcgisportal\index
Replace the content of C:\arcgisportal\index\nodes\0\indices with content of C:\Program Files\ArcGIS\Portal\framework\template\empty_index.zip
Start Portal service
Recreate indexes by running https://portal.domain.com/arcgis/portaladmin/system/indexer/reindex with Mode=Full

Note that I didn't see anything in the service-error-0.log, so while there might be similar contributing factors, this doesn't look like it's exactly the same issue.

Hope this helps.

Craig

MwalimuKaranja · ‎07-29-2019

Hi Craig, thanks for taking the time to try and answer my query. I tried to follow your instructions but got stuck because none of the java services for the Portal were running. Only java services running on Task Manager were the Server and DataStore services.

Also, this was the only error in the Portal logs:

<Msg time="2019-07-29T23:32:28,848" type="SEVERE" code="217003" source="Portal" process="1644" thread="1" methodName="" machine="XXXXX" user="" elapsed="">Error before starting configuration observer. null</Msg>
<Msg time="2019-07-29T23:32:28,849" type="SEVERE" code="217075" source="Portal" process="1644" thread="1" methodName="" machine="XXXXX" user="" elapsed="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>

Tried the re-indexing procedure but still couldn't access the Portal admin directory.

FrédéricPRALLY · ‎07-30-2019

Hi Mwalimu,

Please check the following items about this file C:\Program Files\ArcGIS\Portal\framework\runtime\tomcat\conf\server.xml

1. if it's existing => if it's missing copy paste my template file version 10.7.1,

2. if the size of this file is correct (not 0 ko)=> if it's missing copy paste my template file version 10.7.1,

3. if the size is less than 17 ko, edit it and check at the end to see if there are lines missing such as closing tags.

4. after that restart Portal Windows service

Hope this help,

Fred

MwalimuKaranja · ‎07-31-2019

Hi Frederic,

Thanks for replying. Unfortunately, the server.xml file that is in the tomcat/conf folder is in great shape - opening and closing tags are fine. There are absolutely no issues with the XML file, sadly. Still having the same problem.

VHolubec · ‎10-07-2019

Hello,

we are having similar issue. JAVA process is not running. JAVAW seems to be ok as well as internal DB is running (connection via pgadmin works fien). But, we are unable to open home or portaladmin page.

The log starts with - The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections and it´s full of - >

<Msg time="2019-10-07T12:20:23,404" type="WARNING" code="218014" source="Portal" process="3364" thread="1" methodName="" machine="GISCENTR002.AD.LOCAL" user="" elapsed="" requestID="">Starting Index Service.</Msg>
<Msg time="2019-10-07T12:20:23,438" type="WARNING" code="218015" source="Portal" process="3364" thread="1" methodName="" machine="GISCENTR002.AD.LOCAL" user="" elapsed="" requestID="">Started Index Service.</Msg>

We are not able to loging to reindex - is there any way how to reindex without portaladmin or reinstall the indexisng service?

Thank you.

Vladimir

JonathanQuinn · ‎10-07-2019

Check the logs, they may tell you why the index service can't start:

C:\Program Files\ArcGIS\Portal\framework\runtime\ds\usr\logs\<machine>\elasticlog\bds_<id>.log

VHolubec · ‎10-08-2019

Dear Jonathan,

there is not so much in the log - just these four lines:

[2019-10-01T16:12:45,697][INFO ][o.e.n.Node ] [GISCENTR002.AD.LOCAL] stopping ...

[2019-10-01T16:12:46,927][INFO ][o.e.n.Node ] [GISCENTR002.AD.LOCAL] stopped

[2019-10-01T16:12:46,954][INFO ][o.e.n.Node ] [GISCENTR002.AD.LOCAL] closing ...

[2019-10-01T16:12:47,315][INFO ][o.e.n.Node ] [GISCENTR002.AD.LOCAL] closed

Other files here mention just a time - month ago - when site location was changed from 😧 drive to G: drive. I think this could be the trigger of the issue, because we found, that in some portal configs file the 😧 drive was not changed to G: drive. On the other hand, the Portal was running fine for the whole month after site location was changed.

Any idea?

Thank you.

Vladimir

JosephBuckles1 · ‎12-01-2021

Did you ever find a solution to your JAVA issues? We are experiencing that now on 10.8.1. its like JAVA just disappeared from our server.

StephenKaranja · ‎12-28-2020

Thanks a lot for your suggestion Fred! It resolved a problem we had starting the Portal for ArcGIS service after a power failure interrupted it during its startup. The C:\Program Files\ArcGIS\Portal\framework\runtime\tomcat\conf\server.xml file had been corrupted (filled with NULL bytes). Replacing it with a valid file and restarting the server machine resolved the issue.