Log Component Capacity Planning
The log storage component is responsible for storing logs, events, and audit data collected by the log collection component from one or more clusters in the platform. Therefore, you need to assess your log scale in advance and plan the resources needed for the log storage component according to the guidelines in this document.
WARNING
- The following data represents standard figures obtained from tests conducted under laboratory conditions, intended for your reference when planning resources. You must ensure that the actual resources you plan exceed the testing resources described below, and that the log scale does not exceed the corresponding log scale.
- The disk configuration for the data below is:
6000 iops, 250MB/s read and write speed, SSD independent mounting. If your actual storage resources are weaker than the testing resources, please refer to larger scale configuration information and provide more CPU and memory resources as needed.
ElasticSearch
Small Scale 3 Nodes - Total Logs: 6300/s
| Component | Replicas | CPU Limit | Memory Limit |
|---|
| ElasticSearch | 3 | 2C | 4G |
| Kafka | 3 | 2C | 4G |
| Zookeeper | 3 | 2C | 4G |
| Lanaya | 2 | 2C | 4G |
| Razor | 2 | 1C | 2G |
Small Scale 5 Nodes - Total Logs: 9900/s
| Component | Replicas | CPU Limit | Memory Limit |
|---|
| ElasticSearch | 5 | 2C | 4G |
| Kafka | 3 | 2C | 4G |
| Zookeeper | 3 | 2C | 4G |
| Lanaya | 2 | 2C | 4G |
| Razor | 2 | 1C | 2G |
Large Scale 3+5 Nodes - Total Logs: 25000/s
| Component | Replicas | CPU Limit | Memory Limit |
|---|
| ElasticSearch - Master | 3 | 2C | 4G |
| ElasticSearch - Data | 5 | 8C | 16G |
| Kafka | 3 | 2C | 4G |
| Zookeeper | 3 | 2C | 4G |
| Lanaya | 2 | 2C | 4G |
| Razor | 2 | 1C | 2G |
Large Scale 3+7 Nodes - Total Logs: 30000/s
| Component | Replicas | CPU Limit | Memory Limit |
|---|
| ElasticSearch - Master | 3 | 2C | 4G |
| ElasticSearch - Data | 7 | 8C | 16G |
| Kafka | 3 | 2C | 4G |
| Zookeeper | 3 | 2C | 4G |
| Lanaya | 2 | 2C | 4G |
| Razor | 2 | 1C | 2G |
Clickhouse
Single Node - Total Logs: 18000/s
| Component | Replicas | CPU Limit | Memory Limit | Remarks |
|---|
| Clickhouse | 1 | 2C | 4G | 1 replica 1 shard |
| Razor | 1 | 1C | 1G | - |
| Vector | 1 | 2C | 4G | - |
Three Nodes - Total Logs: 20000/s
| Component | Replicas | CPU Limit | Memory Limit | Remarks |
|---|
| Clickhouse | 3 | 2C | 4G | 3 replicas 1 shard |
| Razor | 2 | 1C | 1G | - |
| Vector | 2 | 2C | 4G | - |
Six Nodes - Total Logs: 40000/s
| Component | Replicas | CPU Limit | Memory Limit | Remarks |
|---|
| Clickhouse | 3 | 4C | 8G | 3 replicas 2 shards |
| Razor | 2 | 1C | 1G | - |
| Vector | 2 | 4C | 8G | - |
Nine Nodes - Total Logs: 69000/s
| Component | Replicas | CPU Limit | Memory Limit | Remarks |
|---|
| Clickhouse | 9 | 4C | 8G | 3 replicas 3 shards |
| Razor | 2 | 1C | 1G | - |
| Vector | 2 | 4C | 8G | - |
