Failed to validate the server managed database

[LavanyaVasudevanNES]

Hello Everyone, 

We are running into issues at 11.3, and 11.5 where clients performed the latest Windows Server 2019, 2022 patches, restarted the machines and datastore no longer validates hence "breaking" Federation.  I have opened an ESRI ticket and we are seeing nothing unusual.   

1.  We have ensured all folders have service account access, no luck

2. We have restarted the the services and machines ( portal, datastore, server) , no luck 

3. Describedatastore shows healthy status, still no luck

4.  We even tried to restart a broken hosted service via portal admin end point to see if it will fix the issue, no luck 

5. Ensured all ports were open, disabled firewall, still no luck 

This is happening to machines both on the AWS environment and On -Premise Hyper-V boxes.   Since this is a production environment, we are hoping to use the Repair Datastore as the last resort. 

This is broken at 11.5 even on a machine that hasnt been updated since January so  makes me thing something else is happening.   Three different IT departments, three different Clients are involved.   All of the machines were functional before Wednesday the 12th of November 2025.  

The machines worked perfectly last Wednesday.  No other changes were made except for the update patches to Windows.   I monitor a few more machines that are going to get the patches applied this next weekend and am hoping to find a root cause/solution before then.   I am attaching a screenshot of the updates that were applied, machines restarts and AGE is broken.   

 I am really running out of resources and time at this point.  Any help is appreciated. 

thankyou 

lavanya

[JakeSkinner]

Hi @LavanyaVasudevanNES,

You should have some datastore backups in the following location \arcgisserver\backup\relational\dbbackup.  You can try performing a fresh install of ArcGIS Data Store and restore one of the backups located here for the relational store.  Take a look at the below video for steps on how to do this:

https://community.esri.com/t5/arcgis-enterprise-documents/restore-and-upgrade-relational-data-store-...

[LavanyaVasudevanNES]

Thankyou Jake. 

This is a great solution but it still doesnt get to the root cause of why all of a sudden the clients are losing federation/validation.   

[JakeSkinner]

@LavanyaVasudevanNES are the datastore logs (i.e. C:\arcgisdatastore\logs) reporting anything?

[LavanyaVasudevanNES]

We set the log more to Debug on the server and here is what is returning: 

<Msg time="2025-11-19T11:05:06,179" type="DEBUG" code="9999" source="Data Store" process="8844" thread="1" methodName="" machine="MachineName" user="" elapsed="" requestID="">Health check replication....</Msg>
<Msg time="2025-11-19T11:05:06,180" type="DEBUG" code="9999" source="Data Store" process="8844" thread="1" methodName="" machine="MachineName" user="" elapsed="" requestID="">Replication health check is not enabled.</Msg>
<Msg time="2025-11-19T11:05:06,180" type="DEBUG" code="9999" source="Data Store" process="8844" thread="1" methodName="" machine="MachineName" user="" elapsed="" requestID="">Replication is not enabled for this relational data store.</Msg>

[JakeSkinner]

There will be additional logs specifically for the Data Store located at \arcgisdatastore\logs.  I would recommend checking there as well.

[LavanyaVasudevanNES]

Hey Jake, 

This is from the datastore.  The logs were empty on 11, 12,13 and then 14th it start filling up with this.  I do have an ESRI case where I have sent the complete log file to:  Esri Case #04048032  

Thankyou again for providing such quick responses.  

[A_Wyn_Jones]

Can you check the Windows Services start up type, I have found Windows update to changes these sometimes.

Try restarting ArcGIS server whilst everything else is running - If this works, you may need to change ArcGIS Server and Datastore to start up type to Automatic (Delayed)

[LavanyaVasudevanNES]

Thankyou Wyn,  they were set to Automatic.  I had to reset them back to Automatic(Delayed) and that still didnt fix the federation/validation.    I still dont understand why Windows Updates reset this and service account permissions to folders.  

I am now looking into:  \arcgisportal\logs\database\pg_log and seeing a whole lot of 

2025-11-13 08:00:20.878 PST [4800] FATAL: terminating connection due to administrator command
2025-11-13 08:00:20.878 PST [4060] FATAL: terminating connection due to administrator command
2025-11-13 08:00:20.878 PST [3096] FATAL: terminating connection due to administrator command
2025-11-13 08:00:20.878 PST [3428] FATAL: terminating connection due to administrator command
2025-11-13 08:00:20.888 PST [2112] LOG: background worker "logical replication launcher" (PID 1252) exited with exit code 1
2025-11-13 08:00:20.908 PST [4088] LOG: shutting down
2025-11-13 08:00:22.271 PST [2112] LOG: database system is shut down

and then at about 4:30 CST, the log file shows: 

2025-11-13 14:37:27.785 PST [10908] LOG: starting PostgreSQL 15.5, compiled by Visual C++ build 1931, 64-bit
2025-11-13 14:37:27.786 PST [10908] LOG: listening on IPv6 address "::", port 7654
2025-11-13 14:37:27.786 PST [10908] LOG: listening on IPv4 address "0.0.0.0", port 7654
2025-11-13 14:37:27.843 PST [10188] LOG: database system was shut down at 2025-11-13 14:20:23 PST
2025-11-13 14:37:27.870 PST [10908] LOG: database system is ready to accept connections

and all other log files  after this are EMPTY.  

[LavanyaVasudevanNES]

Okay we have tried to unregister the datastore, restore from backup and it has failed.   

The server itself was restored to Pre-windows update phase and everything validated and worked perfectly fine.   Unfortunately, due to IP conflicting issues, we have had back to the post Windows Update phase. Now Datastore is broken/not validating and federation gives an error.  Luckily this is sufficient to get the GIS team working on stuff, but we are back to the original problem.  This is surely something from the windows update that is messing with the GIS elements.     We suspect:  https://support.microsoft.com/en-us/topic/november-11-2025-kb5068791-os-build-17763-8027-485c3175-0d...  is the problem child.