The Spatiotemporal Big Datastore is fairly complex and it can be difficult to figure out what setting values you may need without actually looking at the STBDS indices (in Elasticsearch) and getting some information. In general, these are in line with the ES recommendations.
https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html
The Default Number of Shards can exceed the number of nodes, particularly in situations where:
- The spatiotemporal cluster is expected to expand in the future. If you start with 3 STBDS nodes but add more machines over the project to support additional load, recommend going with 7 primary shards. That would support scalability up to 7 machines without requiring re-indexing or re-creating the STBDS data source.
- You have sufficient CPU cores and disk bandwidth on each machine. So if you have a 3-node cluster with 8 cores per node, you should be fine with an index that has 7 primary shards. Source:
https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/
- You are trying to control shard size. If you are getting very large shards for daily rollover but very small shards (<10GB) with hourly rollover, it may be better to increase the primary shards to reach the desired 20-40GB per shard recommendation.
If this number is under 10 million records per some period, then rolling data should be set to that period*.
That is:
Otherwise, divide by 30 and continue to the next step.
* Assuming 3 shards, each record is 4KB, 10 million records ~= 40GB / 3 = 13.3GB per shard,
which falls within ES recommendations.
Record size needs to be estimated empirically, but 2-4KB is common.
See: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
To determine how many shards are created per index interval:
total_shards_per_rolling_period = number_of_shards + (replication_factor*number_of_shards)
To determine the “steady state” shard count:
total_shards = total_shards_per_rolling_period * retention_period
shards_per_machine = total_shards / N
primary_shards_per_machine = shards_per_machine / (replication_factor + 1)
replica_shards_per_machine = (shards_per_machine * replication_factor) / (replication_factor + 1)
e.g.
index is configured with 10 shards, daily rolling period, replication factor of 2, 30 days retention
N (number of machines) = 5
number_of_shards = 10
replication_factor = 2
rolling_period = 1 day
retention_period = 30 days
total_number_of_shards_per_rolling_period = 10 + (10*2) = 30 shards per day
total_shards = 30 shards per day * 30 days = 900 shards
shards_per_machine = 900 / 5 = 180 (60 primary, 120 replicas)
primary_shards_per_machine = 180 / (2 + 1) = 60 primary shards
replica_shards_per_machine = (180 * 2) / (2 + 1) = 120 primary shards
60 primary shards per machine isn’t too bad. If necessary, shards can be lowered by setting the replication factor lower.
Please note that in 10.6+, STBDS will “shrink” smaller rollover indexes/shards to 1 shard to avoid an index explosion.
Typically you don’t need to change this from the default (1,000 milliseconds) unless you are running into write issues
(rejected execution exceptions, or Feature Service document count lagging the GeoEvent STBDS output count, etc.).
The refresh_interval is a separate setting on the STBDS data source, and it defaults to 1 second.
Generally the STBDS output flush interval should be less than or equal to refresh_interval.
Additionally, consider looking at other aspects of the system which may affect query performance:
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.