ArcGIS Portal Issue

[LeeKum_Cheong]

Hi all,

Need some insights on the following issue.

<Msg time="2017-09-29T10:46:45,92" type="SEVERE" code="218010" source="Portal Admin" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2017-09-29T10:46:45,94" type="WARNING" code="218012" source="Portal Admin" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error at check and update urls. The portal is currently unavailable. Please contact your portal administrator.</Msg>
<Msg time="2017-09-29T10:46:45,396" type="WARNING" code="217014" source="Portal" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error at URL updater. C:\Program Files\ArcGIS\Portal\apps\GeoList\index.html</Msg>
<Msg time="2017-09-29T10:46:50,192" type="WARNING" code="218014" source="Portal" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Starting Index Service.</Msg>

Subsequently after restart.

<Msg time="2017-09-29T10:49:41,498" type="SEVERE" code="217003" source="Portal" process="4048" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error before starting configuration observer. null</Msg>
<Msg time="2017-09-29T10:49:41,498" type="SEVERE" code="217075" source="Portal" process="4048" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>

I am on ArcGIS Enterprise 10.5.1, windows 7. Have tried repairing Portal as well, but still faced with same issue. Also, I tried to perform a netstat but couldn't see 7443, 6443 is present.

Thanks.

[JonathanQuinn]

Under the <install dir>\framework\runtime\tomcat\logs folder, are there any errors within the localhost.<date>.log file?

[LeeKum_Cheong]

Hi Jonathan,

Thanks for reaching out. The latest log was localhost.2017-09-29.log and the file is empty. The 2nd latest log was localhost.2017-09-26.log.

I see some errors within the log files, could you advise on the details?

Thanks.

[AllanAxon1]

I have the same error (code 218010) from the portal log about the database on startup. In the localhost log in the <TomcatHome>/logs I have the following (shortened here, but attached the whole log). I also cannot login to the portal with the initial admin user.

Oct 06, 2017 8:21:57 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'gwCommunity' while setting bean property 'community'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwCommunity' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'securityPermissions' while setting bean property 'securityPermissions'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityPermissions' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'gwContent' while setting bean property 'content'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwContent' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'gwSearch' while setting bean property 'search'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwSearch' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'searchIndexRemoteServiceProxy' while setting bean property 'index'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexRemoteServiceProxy' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceDispatcher' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceDispatcher' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceProxyEs' while setting bean property 'indexingServices' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceProxyEs' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Invocation of init method failed; nested exception is NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

[Anonymous User]

Hi Jonathan,

I am facing the same issue after our upgrade to 10.6.1 to 10.8. Need your help on this. Please find the attached log files. 

Thanks in Advance

Naveen.

[JonathanQuinn]

The index service isn't starting. Are you on a one core machine? If so, you'll need to be on a machine with at least 2 cores starting at 10.8. If you are on a machine with at least 2 cores, troubleshooting index related issues can get complicated; I suggest you reach out to Support.

[Anonymous User]

Thanks for your reply. our server is 4core machine. I think issue is not related with cores on machine. regarding the indexing it was running fine and records got matched. (done full mode indexing). our major issue is federation is getting failed  on both hosted server and geoevent. I have raised the ticket already for support but the response is very slow. So i try to reach out in our community.

Thanks,

Naveen 

[JonathanQuinn]

You could look at the logs under <install directory>\framework\runtime\ds\usr. Sometimes they can provide useful information about why the index service won't start.

[Anonymous User]

Hi Jonathan,

I am with NAVBE onto this issue. Please find the logs. It is very difficult for us to understand these logs messages.

We were having HA portal setup but for now we are living with only one portal only as we have seen issues switching over the roles and to be true HA ( as per product documentation postgres within portal can never be HA and work as Active-PAssive)

Currently we have 2 java processes running as shown in below image and yes our federation is failing because some component of ESRI portal is not working properly ( when i say some component yes because portal is using many things internally like Tomcat, Python, pgsql , elastisearch etc) but the logs shared by NAVBE was taken from following path D:\ArcGIS\Portal\framework\runtime\tomcat\logs and localhost clearly logs some errors.

Can you please give us some more insight on this issue? Is it related to permissions? We have already granted full control on service level user to all gis folders. Thanks

[JonathanQuinn]

In regards to internal components not being HA as they're active passive, HA in general means no single points of failure. A primary and standby database, with automatic failover, means that there isn't a single point of failure at the database tier. An environment doesn't need to be active/active at every tier for it to be HA.

In regards to the problem, you're at 10.8, correct? Did you attempt to rebuild the index manually? In your case, there are two problems: split brain issue with the internal index service, and it appears you may have rebuilt the index manually, which further corrupts it.

Do you have a support case open? If so, can you provide the case number?