Highly available Portal - Standby Server DB issues

[rajeshc]

Hi,

We have a Highly available setup of Portal for ArcGIS 10.6.1 with one Primary and one Standby server each. The OS is Wiondows Server 2016. We are running into Disk space issues on the standby server due to numerous DB folders being created automatically under the portal installation path C:\arcgisportal. Sometimes every 2-3 minutes, a new db folder is created. This issue is appearing only on the standby server. Is this behavior normal for the product?

Thanks,

Rajesh C

[JonathanQuinn]

No, that's not expected behavior. The only time a dbXXXX folder will be created is when the standby comes back from a failure or when you run importSite on Portal. During normal operation, it shouldn't create those folders.

It seems like it is continually thinking the standby is coming back from a failure. Can you set the logs to at least Verbose and then post the logs? Feel free to scrub the machine names out of them. Do you see a promote.dat file within the db directory on primary?

[rajeshc]

Hi Jonathan,

Our setup was a fresh installation and new site, there was no site import operation.  

The primary db directory contains a promote.dat file with zero bytes.

I have added the portal log to the main post as an excel file.

Thanks for the reply.

[JonathanQuinn · Accepted solution]

The promote.dat file is the trigger file used to tell the software to promote the database to primary. This shouldn't be in the db directory on the primary and is likely the reason for the behavior you're seeing. Delete the promote.dat file and see if that improves things.

[rajeshc]

We have deleted the promote.dat file from the db directory on the primary server. I will update on the effect later.

It is neither present on the secondary.  Curious to know how the promotion will work if this file is not present..

Thanks

[JonathanQuinn]

It shouldn't be present anywhere. The promote.dat file is created once the software determines that the standby should be promoted. What you may have been running into is a bug, an indefinite loop of promotion/demotion:

1) Portal 1 is primary and portal 2 is standby. Portal 1 has the promote.dat file. If this file is present in the db directory on standby, it will be promoted to primary. At some point the standby came back from being down and the file was present in the db directory on primary.

2) When standby comes back, it needs to get a db snapshot from primary. This is basically copying the db directory from portal 1 to portal 2.

3) When portal 2 gets the db directory, the promote.dat file tells it to promote itself to primary.

4) Logic in portal 1 is validating whether portal 2 is standby. When it isn't, it backs up the db directory to dbXXXX and sends over another db directory.

5) Now the deployment is back at step 3, and so on.

There are two issues: 1) the promote.dat file is left on primary and 2) the promote.dat file is moved over to standby during the db snapshot. I'm not sure how 1 could have happened but 2 is easily handled by deleting the file when we create the db snapshot. We plan to do this within the software.

[rajeshc]

By deleting the promote.dat file from the Primary Server's db directory seems to solve the issue so far.  Now is it safe to delete those unwanted  dbxxxxx folders ?

Thanks

[JonathanQuinn]

Yes, you can delete the dbXXXX folders.