ArcGIS Data Store has detected an issue with 'db'

[Sehafiz]

I am using ArcGIS Enterprise 11.3 with ArcGIS Portal, Web Adaptors, Hosting Server, Data Store and another federated ArcGIS Server - each installed on separate virtual machines with Windows Server 2019 OS.

From ArcGIS Hosting Server Manager log, I can see there is a warning that says ArcGIS Data Store has detected an issue with 'db'. This message actually comes from the Data store log that I confirmed when I checked the Data Store log file from within the Data Store machine. Usually it appears after a certain interval. Not the same interval every day. It may appear every 5 minutes or every 9 minutes and sometimes there is a big 3-4 hours gap and then starts appearing.

I raised this issue with local Esri and they could not replicate the issue and could not find the cause of the issue. During the investigation the machine resources were checked like CPU, RAM, Disk Space etc. Nothing found insufficient. I had unique SSL certificates for each machine with the required Subject Alternative Names (SANs) and the machines were trusting each other. Someone else advised to use the same single SSL certificate across all the machines with one common name and the required SANs. I implemented that. Still I get the same warning message.

Is there anything I can do to find the root cause of the message thereby find a solution to fix it?

Following is a screenshot of the error message. 

Thank you.

[OzanCanT]

Hi @Sehafiz ,

Under normal circumstances, PFX should contain the entire chain, so you can add PFX. You don't need to add it from here. Just double-click on the PFX file, select Local Machine, make sure Include All extended properties is selected, and it will be imported to the right location if you select automatic. If you want to do it here, it would be better to add it to the Trusted Root Certification and Personal folders.

The important point is that this certificate must be the same as the certificate used in the initial site (tomcat) of the Portal, Server, and DataStore. To verify this, you can check the endpoints specified in the documentation below for the Portal and Server:
https://enterprise.arcgis.com/en/server/latest/administer/windows/configuring-https-using-an-existin...
https://enterprise.arcgis.com/en/portal/latest/administer/windows/import-a-certificate-into-the-port...

Additionally, you mentioned that you updated the SSL. Did you also update the SSL certificate in ArcGIS Data Store using the updatesslcertificate utility? Did you update the SSLs only for IIS (ArcGIS Web Adaptor) or for the initial sites of the Portal, Server, and data store (tomcat)?

[Sehafiz]

Hi @OzanCanT ,

Since the beginning the certificate was in the "Personal" folder. According to your advice, I have also installed it selecting "Automatic". Also manually I have added that in the "Trusted Root Certification" folder. I have done this in all the participating machines - Web Adaptor, Portal, Hosting Server, Data Store and the other federated ArcGIS Server. I have restarted all of those. Unfortunately I am still getting the Warning message, "ArcGIS Data Store has detected an issue with 'db'." and the "could not accept SSL connection: An established connection was aborted by the software in your host machine."

I can confirm that I am using the same pfx for all the machines, when I am installing it by double clicking on it and when I am importing to ArcGIS Server and Portal. How I am importing are shown in the following screenshots.

Root and Intermediate Certificate import into PortalImport Server Certificate into Portal

Import Server Certificate into ArcGIS Server

Also I have updated the Data Store certificate via replacesslcertificate utility. Since I am using ArcGIS Enterprise 11.3, updatesslcertificate utility is deprecated and Esri suggests to use replacesslcertificate. The link is https://enterprise.arcgis.com/en/portal/11.3/administer/windows/whats-new-data-store.htm

I would like to thank you again for your assistance.

[OzanCanT]

Hey @Sehafiz ,

Thanks again for the details, based on what you shared earlier. I noticed that you're now using a single SSL certificate with the necessary SANs across all sites. That's generally OK, but I’m wondering if the single certificate you are using contains the FQDNs of all components (portal,server,datastore etc.) in the SAN.

From the SSL warning message you're seeing, it doesn’t seem like a full handshake failure, more like a trust or hostname mismatch issue that might be popping up occasionally. ArcGIS Data Store can log these as a warning even if the overall communication works.

To rule out any cert trust or SAN-related problems, you might want to validate the SSL handshake from both directions using OpenSSL (you must install it first for use on the command line). 

• From Hosting Server to Data Store:
  Open Command Prompt on the hosting server and run:
  openssl s_client -connect <datastore-fqdn>:2443

• Do the same from the Data Store machine back to the Hosting Server:
  openssl s_client -connect <hostingserver-fqdn>:6443
  

If the results includes "Verify return code: 0 (ok)" thats great! But if you see something like this: "Verify return code: 18 (self-signed certificate)". That means the cert is not trusted by the machine you're testing from. It will not always breaks the connection but you might get a warning. Also, you can check if all the SANs are there and valid with same command. 

Let me know your findings, and we can dig deeper after that. 

[Sehafiz]

Hi @OzanCanT ,

The SANs in the certificate I have used are

DNS Name: example.domain.com (The domain that we use to access Portal, ArcGIS Server and Hosting Server. For example, to access Portal we use example.domain.com/portal)
DNS Name: Web Adaptor FQDN
DNS Name: Portal for ArcGIS FQDN
DNS Name: Federated ArcGIS Server FQDN
DNS Name: Hosting Server FQDN
DNS Name: DataStore FQDN

To install the OpenSSL, I will need permission from a board of members. It will take some time. Unfortunately I will be away for few weeks and I will not be able to finish the installation before I go.

Do the above SANs give you any hint or still you want me to test using OpenSSL?

Thank you very much.

[OzanCanT]

Hi @Sehafiz ,

Thank you for the update.

The SANs listed in the certificate look appropriate. Could you also confirm the Common Name (CN) value of the certificate? Ideally, the CN should also be example.domain.com, which seems to be your primary domain used for accessing Portal and Server.

Additionally, can you clarify whether the SSL certificate in use for the initial sites is:

• A self-signed certificate
• An internal domain certificate (issued by your organization’s internal CA)
• Or a public CA-signed certificate

This could help us better assess if trust issues or handshake failures may be occurring due to certificate trust chains.

As for the OpenSSL command — it’s a very lightweight and effective tool to manually verify whether a proper SSL handshake can occur between your ArcGIS Server and Data Store endpoints. However, we understand your situation and appreciate the constraints.

Once you're back, I'd strongly recommend doing an OpenSSL test both:

• From the Server machine to Data Store (openssl s_client -connect <datastore-fqdn>:2443)

• And from the Data Store machine to the Server (openssl s_client -connect <server-fqdn>:6443)

Let me know the CN and certificate type when possible, and safe travels!

Best regards,
Ozan

[Sehafiz]

Hi @OzanCanT ,

Thank you for the prompt reply.

Yes you are right. The CN is example.domain.com.

We are using two certificates.

Internally - When the sites are accessed internally - it uses domain certificate, issued by the organization.

Externally - When the sites are accessed externally - it uses public certificate issued by Azure Front Door (AFD).

Thank you again for helping me.