ArcGIS Portal Issue

LeeKum_Cheong · ‎10-03-2017

Hi all,

Need some insights on the following issue.

<Msg time="2017-09-29T10:46:45,92" type="SEVERE" code="218010" source="Portal Admin" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2017-09-29T10:46:45,94" type="WARNING" code="218012" source="Portal Admin" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error at check and update urls. The portal is currently unavailable. Please contact your portal administrator.</Msg>
<Msg time="2017-09-29T10:46:45,396" type="WARNING" code="217014" source="Portal" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error at URL updater. C:\Program Files\ArcGIS\Portal\apps\GeoList\index.html</Msg>
<Msg time="2017-09-29T10:46:50,192" type="WARNING" code="218014" source="Portal" process="3832" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Starting Index Service.</Msg>

Subsequently after restart.

<Msg time="2017-09-29T10:49:41,498" type="SEVERE" code="217003" source="Portal" process="4048" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Error before starting configuration observer. null</Msg>
<Msg time="2017-09-29T10:49:41,498" type="SEVERE" code="217075" source="Portal" process="4048" thread="1" methodName="" machine="XXX.XXX.COM" user="" elapsed="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>

I am on ArcGIS Enterprise 10.5.1, windows 7. Have tried repairing Portal as well, but still faced with same issue. Also, I tried to perform a netstat but couldn't see 7443, 6443 is present.

Thanks.

Luiz_AmadeuCoutinho · ‎10-10-2017

Hi Allan

Yes, We did the reinstallation of ArcGIS Datastore Software only (Not Portal or Server).

I´m not sure but here our Server was full and I think the problem were related with because the source db was inacessible .

So We cleaned a lot of trash (old services and files) and thousands of .gz files from Datastore Backup Walarchive on ...arcgis/datastore/usr/arcgisdatastore/backup/walarchive

AllanAxon1 · ‎10-10-2017

After some more digging, it looks like my problem is related to the elasticsearch used by Portal and the fact that the process for this that listens on 7120 and 7220 is not starting correctly.

We are not using DataSource yet.

Thanks for the help.

JonathanQuinn · ‎10-10-2017

It seems the problem is that elastic doesn't know to configure itself on 7120 and 7220:

[{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

The 9300 indicates the port it's trying to start on. Is this a new install or an upgrade?

AllanAxon1 · ‎10-10-2017

This is a new install. I originally thought this was a firewall problem, but some other error messages suggested to me that the 9300 may be a process id? I was able to install and run Portal on my workstation and there is a java.exe process that is running there that listens on 7120 and 7220. This process doesn't exist on the server and there are several logs that seem to indicate the index or search function is not working when I try to login with the initial administrator account. I can connect to the Postgres database and see the initial admin account record in the user database.

In response to whether elasticsearch is trying to use 7120 and 7220, I do see these ports in the elastic.yml file, so hopefully this process is trying to start up listening on the correct ports.

I'm less convinced it's a firewall problem now after comparing the running portal install on the workstation with the server portal install.

Thanks,

DanMallett · ‎12-07-2017

@Allan Axon did you find a way to fix this? I'm having the same issue.

PaulDavidson1 · ‎10-10-2017

Luiz: I have seen the problem of a full disk take down the DataStore server.

I worked a bit with Esri to try to recover the server but in the end, it was easiest to do a new re-install.

DataStore can chew up space, especially with its default backup settings (at least in 10.4.1)

I have since put monitors on the servers to alert when disks are reaching certain capacities.

Luiz_AmadeuCoutinho · ‎10-11-2017

Yes We did the same here. 🙂

RyanBumm · ‎12-13-2017

We are currently seeing a similar issue on a new portal install. Did you ever find a fix?

AllanAxon1 · ‎12-15-2017

Ryan,

Some more details on our problem and resolution. When we installed portal manually on a workstation, everything worked. When we ran the scripted install (we use Powershell to run installs on our servers), the portal would not work. We could see errors related to connecting to public maven repositories in the server which once we thought about this it made sense because our servers have restricted firewall access to public sites. Also, I had checked for patches but had missed a patch (See: http://support.esri.com/en/download/7535) that when I included this patch in our install script, we were able to access the portal manager.

--Allan

DeanMoiler · ‎04-17-2018

Hi All,

I've come across a fix that might resolve this issue. In searches relating to a similar sounding portal issue("There is not enough space on the disk") I came across this thread, so in case this is the fix, here goes:

I'm running two Portal 10.5.1 Servers, one primary and one failover.

The second server would not start TomCat, but the Portal for ArcGIS Service would start and stay running, and would only populate the following in the Portal log:

************************************************************************

"<Msg time="2018-04-15T02:13:09,352" type="WARNING" code="217014" source="Portal" process="1636" thread="1" methodName="" machine="**YOURPORTALNAME**" user="" elapsed="">Error at URL updater. C:\Program Files\ArcGIS\Portal\apps\GeoList\index.html</Msg>
<Msg time="2018-04-15T02:13:10,179" type="SEVERE" code="217003" source="Portal" process="1636" thread="1" methodName="" machine="**YOURPORTALNAME** user="" elapsed="">Error before starting configuration observer. XML document structures must start and end within the same entity.</Msg>
<Msg time="2018-04-15T02:13:10,179" type="SEVERE" code="217075" source="Portal" process="1636" thread="1" methodName="" machine="**YOURPORTALNAME**" user="" elapsed="">Failed to start the portal. The observer's beforeStart() function returned a failure.</Msg>"

************************************************************************

I tried installing the latest Portal patch:

Esri Support 10.5 (10.5.1)

Unfortunately this did not resolve the issue, and I stumbled upon the following wonderful solution discovered by Jing:

https://jingthegisguy.wordpress.com/2017/02/14/portal-for-arcgis-10-5-fatal-error-fatal-error-11-pre...

For some reason it seems the failover server configuration files had been updated and missed the following trailing text from the:

C:\Program Files\ArcGIS\Portal\framework\runtime\tomcat\conf\server.xml

When I checked the production vs failover versions of these files the following text was missing from the failover:

************************************************************************

Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->

</Host>
</Engine>
</Service>
</Server>

************************************************************************

When I copied this content into the failover server.xml file and restarted the Portal for ArcGIS Service, it started working again. Luckily I didn't need to perform a reinstall on this one!

This is the line in the tomcat error log that made it seem relevant for this issue.

C:\Program Files\ArcGIS\Portal\framework\service\logs\service_error.log

************************************************************************

Mar 09, 2018 10:23:49 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [Astron] modules [], plugins [], sites []
ERROR: 'There is not enough space on the disk'
[Fatal Error] :130:4: XML document structures must start and end within the same entity.

************************************************************************

Hopefully this helps someone and Esri provide a solution / patch to stop this from occurring.

Dean