topic Re: ArcGIS Portal Issue in ArcGIS Enterprise Portal Questions

ArcGIS Portal Issue

LeeKum_Cheong — Tue, 03 Oct 2017 08:30:37 GMT

Hi all,

Need some insights on the following issue.

<Msg time="2017-09-29T10:46:45,92" type="SEVERE" code="218010" source="Portal Admin" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2017-09-29T10:46:45,94" type="WARNING" code="218012" source="Portal Admin" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error at check and update urls. The portal is currently unavailable. Please contact your portal administrator.</Msg>
<Msg time="2017-09-29T10:46:45,396" type="WARNING" code="217014" source="Portal" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error at URL updater. C:\Program Files\ArcGIS\Portal\apps\GeoList\index.html</Msg>
<Msg time="2017-09-29T10:46:50,192" type="WARNING" code="218014" source="Portal" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Starting Index Service.</Msg>

Subsequently after restart.

<Msg time="2017-09-29T10:49:41,498" type="SEVERE" code="217003" source="Portal" process="4048" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error before starting configuration observer. null</Msg>
<Msg time="2017-09-29T10:49:41,498" type="SEVERE" code="217075" source="Portal" process="4048" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>

I am on ArcGIS Enterprise 10.5.1, windows 7. Have tried repairing Portal as well, but still faced with same issue. Also, I tried to perform a netstat but couldn't see 7443, 6443 is present.

Thanks.

Re: ArcGIS Portal Issue

JonathanQuinn — Tue, 03 Oct 2017 17:51:35 GMT

Under the <install dir>\framework\runtime\tomcat\logs folder, are there any errors within the localhost.<date>.log file?

Re: ArcGIS Portal Issue

LeeKum_Cheong — Wed, 04 Oct 2017 01:29:23 GMT

Hi Jonathan,

Thanks for reaching out. The latest log was localhost.2017-09-29.log and the file is empty. The 2nd latest log was localhost.2017-09-26.log.

I see some errors within the log files, could you advise on the details?

Thanks.

Re: ArcGIS Portal Issue

AllanAxon1 — Fri, 06 Oct 2017 12:33:51 GMT

I have the same error (code 218010) from the portal log about the database on startup. In the localhost log in the <TomcatHome>/logs I have the following (shortened here, but attached the whole log). I also cannot login to the portal with the initial admin user.

Oct 06, 2017 8:21:57 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'gwCommunity' while setting bean property 'community'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwCommunity' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'securityPermissions' while setting bean property 'securityPermissions'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityPermissions' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'gwContent' while setting bean property 'content'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwContent' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'gwSearch' while setting bean property 'search'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwSearch' defined in ServletContext resource [/WEB-INF/gw-config.xml]: Cannot resolve reference to bean 'searchIndexRemoteServiceProxy' while setting bean property 'index'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexRemoteServiceProxy' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceDispatcher' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceDispatcher' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceProxyEs' while setting bean property 'indexingServices' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceProxyEs' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Invocation of init method failed; nested exception is NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

Re: ArcGIS Portal Issue

Luiz_AmadeuCoutinho — Fri, 06 Oct 2017 15:49:45 GMT

I got a similar problem here, few weeks ago. The datastore was reinstalled and the problem was fixed.

And I don´t know what happend.

Re: ArcGIS Portal Issue

AllanAxon1 — Fri, 06 Oct 2017 16:41:22 GMT

Luiz,

Thanks for the reply. I'm new to portal and trying to script the installation. Are you talking about the local postgres db (gwdb)? This database seems to be running and is accessible locally through port 7654 on the portal machine. How do you re-install just the datasource. I've uninstalled/re-installed portal several times and keep getting the errors reported by OP and myself.

I assume we are not talking about installing the ArcGIS Data Store, but let me know if that's what you did.

--Allan

Re: ArcGIS Portal Issue

Luiz_AmadeuCoutinho — Tue, 10 Oct 2017 09:27:53 GMT

Hi Allan

Yes, We did the reinstallation of ArcGIS Datastore Software only (Not Portal or Server).

I´m not sure but here our Server was full and I think the problem were related with because the source db was inacessible .

So We cleaned a lot of trash (old services and files) and thousands of .gz files from Datastore Backup Walarchive on ...arcgis/datastore/usr/arcgisdatastore/backup/walarchive

Re: ArcGIS Portal Issue

AllanAxon1 — Tue, 10 Oct 2017 13:16:42 GMT

After some more digging, it looks like my problem is related to the elasticsearch used by Portal and the fact that the process for this that listens on 7120 and 7220 is not starting correctly.

We are not using DataSource yet.

Thanks for the help.

Re: ArcGIS Portal Issue

JonathanQuinn — Tue, 10 Oct 2017 16:41:26 GMT

It seems the problem is that elastic doesn't know to configure itself on 7120 and 7220:

[{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

The 9300 indicates the port it's trying to start on. Is this a new install or an upgrade?

Re: ArcGIS Portal Issue

AllanAxon1 — Tue, 10 Oct 2017 16:49:14 GMT

This is a new install. I originally thought this was a firewall problem, but some other error messages suggested to me that the 9300 may be a process id? I was able to install and run Portal on my workstation and there is a java.exe process that is running there that listens on 7120 and 7220. This process doesn't exist on the server and there are several logs that seem to indicate the index or search function is not working when I try to login with the initial administrator account. I can connect to the Postgres database and see the initial admin account record in the user database.

In response to whether elasticsearch is trying to use 7120 and 7220, I do see these ports in the elastic.yml file, so hopefully this process is trying to start up listening on the correct ports.

I'm less convinced it's a firewall problem now after comparing the running portal install on the workstation with the server portal install.

Thanks,

Re: ArcGIS Portal Issue

PaulDavidson1 — Tue, 10 Oct 2017 17:18:12 GMT

Luiz: I have seen the problem of a full disk take down the DataStore server.

I worked a bit with Esri to try to recover the server but in the end, it was easiest to do a new re-install.

DataStore can chew up space, especially with its default backup settings (at least in 10.4.1)

I have since put monitors on the servers to alert when disks are reaching certain capacities.

Re: ArcGIS Portal Issue

Luiz_AmadeuCoutinho — Wed, 11 Oct 2017 09:26:36 GMT

Yes We did the same here. 🙂

Re: ArcGIS Portal Issue

DanMallett — Fri, 08 Dec 2017 00:41:42 GMT

@Allan Axon did you find a way to fix this? I'm having the same issue.

Re: ArcGIS Portal Issue

RyanBumm — Thu, 14 Dec 2017 04:43:28 GMT

We are currently seeing a similar issue on a new portal install. Did you ever find a fix?

Re: ArcGIS Portal Issue

AllanAxon1 — Fri, 15 Dec 2017 14:29:28 GMT

Ryan,

Some more details on our problem and resolution. When we installed portal manually on a workstation, everything worked. When we ran the scripted install (we use Powershell to run installs on our servers), the portal would not work. We could see errors related to connecting to public maven repositories in the server which once we thought about this it made sense because our servers have restricted firewall access to public sites. Also, I had checked for patches but had missed a patch (See: http://support.esri.com/en/download/7535) that when I included this patch in our install script, we were able to access the portal manager.

--Allan

Re: ArcGIS Portal Issue

DeanMoiler — Tue, 17 Apr 2018 18:24:28 GMT

Hi All,

I've come across a fix that might resolve this issue. In searches relating to a similar sounding portal issue("There is not enough space on the disk") I came across this thread, so in case this is the fix, here goes:

I'm running two Portal 10.5.1 Servers, one primary and one failover.

The second server would not start TomCat, but the Portal for ArcGIS Service would start and stay running, and would only populate the following in the Portal log:

************************************************************************

"<Msg time="2018-04-15T02:13:09,352" type="WARNING" code="217014" source="Portal" process="1636" thread="1" methodName="" machine="**YOURPORTALNAME**" user="" elapsed="">Error at URL updater. C:\Program Files\ArcGIS\Portal\apps\GeoList\index.html</Msg>
<Msg time="2018-04-15T02:13:10,179" type="SEVERE" code="217003" source="Portal" process="1636" thread="1" methodName="" machine="**YOURPORTALNAME** user="" elapsed="">Error before starting configuration observer. XML document structures must start and end within the same entity.</Msg>
<Msg time="2018-04-15T02:13:10,179" type="SEVERE" code="217075" source="Portal" process="1636" thread="1" methodName="" machine="**YOURPORTALNAME**" user="" elapsed="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>"

************************************************************************

I tried installing the latest Portal patch:

Esri Support 10.5 (10.5.1)

Unfortunately this did not resolve the issue, and I stumbled upon the following wonderful solution discovered by Jing:

https://jingthegisguy.wordpress.com/2017/02/14/portal-for-arcgis-10-5-fatal-error-fatal-error-11-premature-end-of-file/c…

For some reason it seems the failover server configuration files had been updated and missed the following trailing text from the:

C:\Program Files\ArcGIS\Portal\framework\runtime\tomcat\conf\server.xml

When I checked the production vs failover versions of these files the following text was missing from the failover:

************************************************************************

Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->

</Host>
</Engine>
</Service>
</Server>

************************************************************************

When I copied this content into the failover server.xml file and restarted the Portal for ArcGIS Service, it started working again. Luckily I didn't need to perform a reinstall on this one!

This is the line in the tomcat error log that made it seem relevant for this issue.

C:\Program Files\ArcGIS\Portal\framework\service\logs\service_error.log

************************************************************************

Mar 09, 2018 10:23:49 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [Astron] modules [], plugins [], sites []
ERROR: 'There is not enough space on the disk'
[Fatal Error] :130:4: XML document structures must start and end within the same entity.

************************************************************************

Hopefully this helps someone and Esri provide a solution / patch to stop this from occurring.

Dean

Re: ArcGIS Portal Issue

ChugachElectric — Wed, 15 Aug 2018 23:42:38 GMT

Dean - Thank you for posting this info, this was our fix after running out of disk space on a single server deployment of 10.5.1.

Re: ArcGIS Portal Issue

StefanKoch — Wed, 22 May 2019 16:51:44 GMT

In my portal 10.7 I also had this problem after deactivating one of my network cards and I fixed it by looking at

the log file in the folder

C:\Program Files\ArcGIS\Portal\framework\runtime\ds\usr\logs\GIS.KFE.BMVG.ORG\elasticlog\<yourcluster>.log

and then configuring

C:\Program Files\ArcGIS\Portal\framework\runtime\ds\framework\runtime\elasticsearch\config\elasticsearch.yml
from network.host: _site_,myserver,_local_ to network.host: 127.0.0.1

Also check the doc under Network Settings | Elasticsearch Reference [6.4] | Elastic

This worked for me.

Hope you find this useful.

Re: ArcGIS Portal Issue

HannesFleischer — Wed, 13 Nov 2019 06:09:05 GMT

Had the same Error Massage in Portal 10.7 after a Bluescreen Crash caused by hardware failure.

C:\Program Files\ArcGIS\Portal\framework\service\logs\service-error-0.log showed:

java.lang.NullPointerException
at com.esri.arcgis.portal.observers.ConfigObserver.d(ConfigObserver.java:830)
at com.esri.arcgis.portal.observers.ConfigObserver.f(ConfigObserver.java:937)
at com.esri.arcgis.portal.observers.ConfigObserver.beforeStart(ConfigObserver.java:192)
at com.esri.arcgis.portal.impl.ObserverManager.beforeStart(ObserverManager.java:159)
at com.esri.arcgis.portal.impl.NodeAgent.start(NodeAgent.java:246)
at com.esri.arcgis.portal.impl.Main.start(Main.java:83)

After some Investigation i found out that the NPE was caused during startup while accessing the following property file:

C:\Program Files\ArcGIS\Portal\framework\webapps\arcgis#sharing\WEB-INF\classes\resources\gw-config.properties

That file had not survived the Bluescreen crash in a valid state and contained a single property rather than a valid list of many (~270 in my case) properties.

Solution:

Copy that file from another installation.
Find the shared key property associated with the corrupt installation in "C:\Program Files\ArcGIS\Portal\framework\etc\portal-config.properties (security.shared.key)
Copy the value of that property to the gw-config.properties shared key property (security.shared-key)

Now my gw-config.properties appears to be valid again. Portal starts OK and can be accessed though GUI and Rest API.

I hope somebody can use this information in a similar situation.

Hannes

Re: ArcGIS Portal Issue

Anonymous User — Wed, 27 May 2020 10:53:26 GMT

Hi Jonathan,

I am facing the same issue after our upgrade to 10.6.1 to 10.8. Need your help on this. Please find the attached log files.

Thanks in Advance

Naveen.