nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Clean Install Object Store Cluster 11.5 Unhealthy

Oiligriv

Hi,
I'm performing a clean installation of ArcGIS Datastore Object Store on dedicated virtual machines to then migrate the tile caches.
The setup went smoothly, and the service user is the same for all three VMs.
I performed the configuration via the web and via command line, but the result is always the same.
Via the web on port 2443/arcgis/datastore, I noticed that the number of VMs in the Cluster is counted correctly.
When I validate via ArcGIS Server Manager, it loops. But if I validate via the Server Rest Admin, I see the following error.

I'm currently very hesitant to launch the 3D cache migration.

Is this behavior normal in Rest Admin?

{
"datastore.name": "oz_hct7a6e",
"datastore.isConfigured": "true",
"datastore.isRegistered": "true",
"deploymentMode": "cluster",
"finalized": true,
"machines": [
{
"machine.overallhealth": "Transitional",
"platform": "Windows",
"name": "PVW-AGOSGEO-AS1.CED.******",
"role": "CLUSTER_MEMBER",
"heapSize": "5246MB",
"healthcheck.enable": "false",
"status": "Starting",
"isSCMHealthy": true,
"isOMHealthy": false,
"isDataNodeHealthy": false,
"s3gStatus": {
"s3gHttpsPort": 29879,
"isS3GHealthy": false
}
},
{
"machine.overallhealth": "Failed",
"platform": "Windows",
"name": "PVW-AGOSGEO-AS2.CED.******",
"role": "CLUSTER_MEMBER",
"healthcheck.enable": "false",
"status": "StartFailed",
"isSCMHealthy": false,
"isOMHealthy": false,
"isDataNodeHealthy": false,
"s3gStatus": {
"s3gHttpsPort": 29879,
"isS3GHealthy": false
}
},
{
"machine.overallhealth": "Unknown",
"platform": "Windows",
"name": "PVW-AGOSGEO-AS3.CED.******",
"role": "CLUSTER_MEMBER",
"healthcheck.enable": "false",
"status": "Configured",
"isSCMHealthy": false,
"isOMHealthy": false,
"isDataNodeHealthy": false,
"s3gStatus": {
"s3gHttpsPort": 29879,
"isS3GHealthy": false
},
"messages": [{"msg": "'PVW-AGOSGEO-AS3.CED.******' is not reachable. Ensure that the machine is powered on and its hostname is accessible."}]
}
],
"datastore.overallhealth": "Unhealthy",
"status": "success"
}

Finde mehr Beiträge mit dem Tag

Accepted answers

All comments

Priya_Das

Hi @Oiligriv

Are you trying to remove the Object datastore from your Environment? If so, I believe we wont be able to remove Object Datastore from ArcGIS Server Manager.

We need to unregister the Object Datastore from Server Admin endpoint.

Copy the Child item (Image 1)

Then to Unregister the object datastore: Home>Data>unregisterItem

In Item path: Paste the item path (Image 2)

Once completed check if the object datastore still exists or not. You can run describe datastore as well.

Before running this take a VM snapshot for backup.

1.png

2.png

Oiligriv

Hi Priya97317
To unregister the ArcGIS Datastore Object Store, I can easily do it using the unregisterdatastore.bat tool.
The object store was just installed and I'm registering it for the first time, but before migrating, I'd like to make sure the validation is OK.

Thanks for the information, but that's not my problem.

Virgilio

s.schuette_ct

Did you configure the objects stores one after the other? Installing them at the same time will cause issues. If you have new machines and no data stored, you can unregister the object store and clear the datastorecontent directory, than you will be able to cleanly register the object store with the cluster (new id will be generated).

There also may be a bug in the Server Manager validation, Object Store validation always fails for me but when directly validating in the server admin it validates successfully. When I open the dev console before starting the object store validation in server manager I see the validation call and it is using a proxy parameter for some reason (no, not set by me).

Justin_Greco

Were you able to resolve this issue? I am setting up Object Store for the first time as well, also on a cluster of 3 machines. I am seeing similar behavior to what you are seeing, except all of my machines are showing an overallHealth of "Transactional" with a status of "Starting".

I have tried unregistering and reconfiguring, but always get the same results.

I am at 11.5 as well, but on Linux.

I do have a premium support case open.

s.schuette_ct

@Justin_Greco Did you start configuration on the Object Store cluster at the same time using command line multi-execution?

FYI there are numerous things you need to be aware of. You can read my lessons learned here: Configure ArcGIS Object datastore in High Availabi... - Esri Community

Justin_Greco

@SimonSchütte_ct Thanks for all the details you have provided about this. Has your configuration been stable?

I thought I had this all fixed twice by removing the datastore and starting fresh, but it seems like it breaks when Linux maintenance is done and the servers are rebooted. Have you had any issues after rebooting servers?

I am already in the process of rebuilding again, which isn't a big deal since we currently only have 1 scene layer published. Also I am configuring them one at a time like you had mentioned, which is what I have been doing all along.

But here are some of my findings:

```
{
  "datastore.name": "oz_u92e5cl",
  "datastore.isConfigured": "true",
  "datastore.isRegistered": "true",
  "deploymentMode": "cluster",
  "finalized": true,
  "machines": [
    {
      "machine.overallhealth": "Transitional",
      "platform": "Linux",
      "name": "SERVER3.xy.xy.xy.xy",
      "role": "CLUSTER_MEMBER",
      "heapSize": "17700MB",
      "healthcheck.enable": "false",
      "status": "Starting",
      "isSCMHealthy": true,
      "isOMHealthy": false,
      "isDataNodeHealthy": false,
      "s3gStatus": {
        "s3gHttpsPort": 29879,
        "isS3GHealthy": false
      }
    },
    {
      "machine.overallhealth": "Failed",
      "platform": "Linux",
      "name": "SERVER2.xy.xy.xy.xy",
      "role": "CLUSTER_MEMBER",
      "heapSize": "18082MB",
      "healthcheck.enable": "true",
      "status": "Started",
      "isSCMHealthy": false,
      "isOMHealthy": true,
      "isDataNodeHealthy": false,
      "s3gStatus": {
        "s3gHttpsPort": 29879,
        "isS3GHealthy": true
      }
    },
    {
      "machine.overallhealth": "Failed",
      "platform": "Linux",
      "name": "SERVER1.xy.xy.xy.xy",
      "role": "CLUSTER_MEMBER",
      "heapSize": "18144MB",
      "healthcheck.enable": "true",
      "status": "Started",
      "isSCMHealthy": false,
      "isOMHealthy": false,
      "isDataNodeHealthy": true,
      "s3gStatus": {
        "s3gHttpsPort": 29879,
        "isS3GHealthy": true
      }
    }
  ],
  "datastore.overallhealth": "Unhealthy"
}

This is what my validate looked like.

SCM is only healthy on SERVER3 (I configured 3 first for some reason)

OM is only healthy on SERVER2

DataNode is only healthy on SERVER2

Since I noticed this the day after the servers were rebooted for maintenance, I am thinking maybe they should be rebooted in a certain order. In my case SERVER3 was the leader. On Wednesday night SERVER1 was rebooted and then on Thursday SERVER2 and SERVER3 were rebooted.

s.schuette_ct

Exactly the question I had: Re: Object Store Cluster Validation 11.5 - Esri Community in my case I could fix it with a fresh configuration, but there should be a proper way to "rebalance" the cluster...

Nimm an der Diskussion teil

Angemeldete Mitglieder können Beiträge verfassen, Updates folgen und mehr. Neu hier? Registriere ein kostenloses Konto.

Schnellzugriffe

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.