How to start Portal for ArcGIS 10.7.1 internal portal database processes (postgres.exe, java.exe, javaw.exe)?

[WilliamDeGraw]

10.7.1 base deployment.

Portal for ArcGIS internal database is postgresql 10.6

Background: The install directory ran out of space during pandemic stay at home orders. The database looks like it didn't shutdown properly. Figured out the space issue and corrected it. The ArcGISPortal.exe restarts ok, but internal database doesn't start. Symptoms - no access to Portal home page, PortalAdmin, webgisdr doesn't run, Server Manager not accessible, cannot serve out any maps or apps. No access to Portal via Web Adaptor, no access via localhost:7443 on the server.

How do you do a database backup with postgres tools installed with Portal?

How do you check the health of the database?

What has to be cleaned up and how?

How do you restart the database manually?

When ArcGISPortal.exe starts up successfully, it takes up 100-115MB of memory, but there are no processes visible for postgres.exe, java.exe, javaw.exe.

cmd line, running tasklist shows no processes for the database.

Running "postgres.exe -d 5 -P -D "data dir" results in:

PANIC: could not open control file "global/pg_control": permission denied

I'm asking for help fixing the database so normal operations can start again.

Last log entry for database was before space issue was fixed and says:

startup process was terminated by exception 0xC0000409

See C include file "ntstatus.h" for a description of the hexadecimal value

aborting startup due to startup process failure

database system is shut down

[WilliamDeGraw]

Thanks for the input, good value.

my indexing comment was not clear. Manually reindexing using clean files from template folder from install files in Portal install directory. I did this once with 10.6.1 where support walked me through the steps.

ESRI case #02404696 Reindex does not match user store and index

I'll try the command with postgres.exe again. My denied attempt was my regular user account.

[WilliamDeGraw]

Notes for all readers:

My next attempt at troubleshooting and log research is 8/24 due to pandemic rules in place at the office.

[WilliamDeGraw]

update: Next office visit is 8/28 to continue troubleshooting.

Looking forward to checking a few things.

For future readers needing an education in postgresql (like me, I'm 10 hours deep searching the Internet for valuable information and finally found the golden ticket):

Best resource I found to learn the basics (concepts and demonstrations) in short bursts, highly recommend watching YouTube videos by CHENNAKESHAVA RAMESH, here's an example - pg ctl for starting stopping cluster - YouTube  . You will learn how to admin from pgAdmin and the command line.

Things on my to do list:

pg_ctl -D "path to cluster" status
pg_ctl -D "path to cluster" start
pg_ctl -D "path to cluster" -m <smart | fast | immediate> stop

If cluster is shutdown, but there is a postmaster.pid file in the cluster folder...thats a problem. A shutdown cluster should not have the postmaster.pid file. Said another way...visible postmaster.pid means your cluster is started and running while no postmaster.pid file means cluster is shutdown. So I can't remember if that file was in the cluster folder, this is a check for next office visit.

Thanks,

Ernie D.

[WilliamDeGraw]

Successfully started postgres via command prompt (runas, needs to be same account that's running Port for ArcGIS service).

pg_ctl start -m fast -o "-p 7654" -D E:\arcgisportal\db

Portal's internal Tomcat web server -  This may not be running. How do you manually start this? 

Going to ...\Portal\framework\runtime\tomcat\bin to run startup.bat didn't do anything to help the situation.

Now researching how to manually start java.exe and javaw.exe . Note, just clicking on the .exe doesn't do anything in directory ...\Portal\framework\runtime\jre\bin.

[WilliamDeGraw]

Close out remark:

So we manually restarted the database. This did not fix the issue as the Portal web server was not running. At this point I asked ESRI to give guidance since all items not running should be controlled by the their code, logically executing all from ArcGISPortal.exe. ESRI recommended uninstalling/reinstalling Portal and it's Web Adaptor and an import site from latest backup via Portal Admin Dir. After taking precautions such as snapshots and manually copying important items (config.js, etc.), performed the recommendation. Still the ArcGIS Enterprise base deployment was not 100% operational. Key point here, Portal was now running as a version from last backup while Server and Datastore were running at current time which caused validations to fail. To fix, ran webgisdr import from last backup to restore all parts. Then replaced default copy of config.js with my copy with custom settings. At this time, all validations passed (green check mark), did some tests in Portal Admin directory, successfully ran a webgisdr export, published to Server, created hosted feature layer in Portal. All stakeholders confirmed all looked good. Closed the case with ESRI support.