How to start Portal for ArcGIS 10.7.1 internal portal database processes (postgres.exe, java.exe, javaw.exe)?

[WilliamDeGraw]

10.7.1 base deployment.

Portal for ArcGIS internal database is postgresql 10.6

Background: The install directory ran out of space during pandemic stay at home orders. The database looks like it didn't shutdown properly. Figured out the space issue and corrected it. The ArcGISPortal.exe restarts ok, but internal database doesn't start. Symptoms - no access to Portal home page, PortalAdmin, webgisdr doesn't run, Server Manager not accessible, cannot serve out any maps or apps. No access to Portal via Web Adaptor, no access via localhost:7443 on the server.

How do you do a database backup with postgres tools installed with Portal?

How do you check the health of the database?

What has to be cleaned up and how?

How do you restart the database manually?

When ArcGISPortal.exe starts up successfully, it takes up 100-115MB of memory, but there are no processes visible for postgres.exe, java.exe, javaw.exe.

cmd line, running tasklist shows no processes for the database.

Running "postgres.exe -d 5 -P -D "data dir" results in:

PANIC: could not open control file "global/pg_control": permission denied

I'm asking for help fixing the database so normal operations can start again.

Last log entry for database was before space issue was fixed and says:

startup process was terminated by exception 0xC0000409

See C include file "ntstatus.h" for a description of the hexadecimal value

aborting startup due to startup process failure

database system is shut down

[George_Thompson]

I would reach out to technical support on this issue to see if there is any solution. I have seen in the past where the site is unrecoverable after this type of issue.

Do you have a backup of the site before the disk space ran out?

[WilliamDeGraw]

Yes we do have a backup. No webgisdr is not accessible. PortalAdmin URL is not reachable...reason unknown, but a good guess is that it needs the database to be running.

[George_Thompson]

Here is another similar post - https://community.esri.com/thread/237613-portal-for-arcgis-107-issue-internal-portal-database-does-n... 

[WilliamDeGraw]

I've read that post a few times before creating my post and my details in my post answers/shows my results of some of the suggestions.

Already have a case open with ESRI support. My first contact said the re-indexing was not supported in 10.7.1 so did not attempt.

I thought GeoNet would be a chance to find someone with PostgreSQL experience since getting the database started manually is the primary objective.

[George_Thompson]

Well just getting the DB started again, may not get you the desired results. It sounds like the Postgres DB for Portal may be corrupted (i.e. unrecoverable).

Do you have a working Portal to compare files to?

In my Portal (C:\arcgisportal\db) it has all of the necessary files to start PG.

Are any of these files at the bottom 0kb?

Did your IT have a snapshot of the machine before the space issues?

Can you verify that the account running that command for the postgres.exe has permissions on the folders needed?

[WilliamDeGraw]

Quick question: is the account running postgres.exe the same account running ArcGISPortal.exe? If the same, I've got that on my to do list. 

When running the following I didn't know what account was needed to run postgres.exe (denied due to running cmd as my user account):

Running "postgres.exe -d 5 -P -D "data dir" results in:

PANIC: could not open control file "global/pg_control": permission denied

If it's a secret account as part of the install, does ESRI known the username and password?

[George_Thompson]

I do not know which account runs that postgres service from Portal, not sure it is the same. I know that there is a way to get that information and the support analyst might be able to help with that.

I did notice that you had some other questions in your original post, see my comments below. I am by no means the "expert" on all these but can provide some guidance.

How do you do a database backup with postgres tools installed with Portal?

    - I am not sure this is directly supported we would usually recommend this ArcGIS Enterprise backups—Portal for ArcGIS (10.7 and 10.7.1) | Documentation for ArcGIS Enterprise as a best practice.

How do you check the health of the database?

       - This is usually managed by Portal. This is usually the beauty of Portal is that you do not worry about this portion of the administration.

What has to be cleaned up and how?

      - If you restore the DB it will be based on the date/time of the backup. I am not sure you can "clean up" outside of what Portal allows you via the available tools.

How do you restart the database manually?

 - This is what you are working on.

Hope this helps.

[WilliamDeGraw]

appreciate your part of the conversation. Always good to talk this out.

People > docs

[JonathanQuinn]

Already have a case open with ESRI support. My first contact said the re-indexing was not supported in 10.7.1 so did not attempt.

Reindexing is absolutely supported at 10.7.1, and every version before and after that, forever. I'm not sure how that information was relayed, but that is not correct.

The database will run as the same account that runs the Portal for ArcGIS Windows service. Starting the database yourself may help, as long as you attempt to start it as the same account used for the Portal for ArcGIS service. Starting the database as a different account may return errors that are not relevant.

PANIC: could not open control file "global/pg_control": permission denied

If you see the above error when attempting to start the database as the Portal for ArcGIS Windows service, then it's clear there's a permissions problem. You can also look at the Event Viewer which should have some errors about attempts to start. There may be a lot of other noise in there (postmaster.pid not found, something about making sure things are installed, etc), but there may be some explicit errors about not being able to start PG.