Log Component Selection Guide
When installing cluster monitoring, the platform provides two log storage components for your choice: ElasticSearch and Clickhouse. This article will detail the features and applicable scenarios of these two components to help you make the most suitable choice.
- You can only choose one of ElasticSearch or Clickhouse for the cluster log storage component installation.
- Any cluster's log storage component can be selected for log collection to interface with the storage data.
- The current version of the DevOps product does not support archiving Jenkins pipeline execution records using Clickhouse. If you need to use the Jenkins pipeline features, please choose the ACP Log Storage with Clickhouse plugin cautiously.
- The current version of the ServiceMesh service mesh does not support integration with Clickhouse. If you need to use the service mesh features, please choose the ACP Log Storage with Clickhouse plugin cautiously.
- The current version of the ACP Log Storage with Clickhouse plugin does not support IPv6 single stack or IPv6 dual stack workload clusters.
Architecture Comparison
ElasticSearch Architecture
ElasticSearch is an open-source distributed search engine built on Lucene, designed for fast full-text search and analysis. Its advantages include:
- High-performance search: Supports real-time search and can quickly process massive amounts of data.
- Flexible querying capabilities: Offers a powerful query DSL, supporting complex query requirements.
- Scalability: Easily horizontally scalable as needed, suitable for applications of all sizes.
- Diverse data support: Able to handle both structured and unstructured data, widely applicable.
Clickhouse Architecture
Clickhouse is a high-performance columnar database designed for Online Analytical Processing (OLAP). Its advantages include:
- Fast data processing: Supports rapid querying and analysis through columnar storage and data compression.
- Real-time analysis: Capable of processing real-time data streams, suitable for real-time data analysis scenarios.
- High throughput: Optimized for the performance of large-scale data writing and querying, making it very suitable for big data scenarios.
- Flexible SQL support: Compatible with standard SQL, easy to get started, reducing the usage threshold.
Function Comparison
| Clickhouse | Elasticsearch | Explanation | |
|---|---|---|---|
| High Availability | Supported | Supported | |
| Scalability | Supported | Supported | |
| Query Experience | Weak | Strong | Elasticsearch offers more robust search capabilities based on the Lucene language, while Clickhouse only supports SQL queries, limiting its search capabilities. |
| Resource Usage | Low | High | For the same performance requirements, Clickhouse requires fewer resources than Elasticsearch. For example, to support 20,000 logs per second, Elasticsearch needs 3 es-masters and 7 es-nodes (2c4g+8c16g), while Clickhouse only requires 3 2c4g replicas. |
| Performance | High | Low | Under the same resource conditions, the log volume supported by Clickhouse far exceeds that of Elasticsearch. |
| Community Activity | Medium | High | The Elasticsearch community is active with rich documentation, while Clickhouse is a growing and improving community. |
Selection Recommendations
-
If you are accustomed to using Elasticsearch and have a high dependency on the Lucene language, it is recommended that you continue to use the ACP Log Storage with ElasticSearch plugin.
-
If you depend on the platform's Jenkins pipeline or service mesh features, it is recommended that you continue to use the ACP Log Storage with ElasticSearch plugin.
-
If you have high requirements for the performance and resource consumption of the log component but only have basic needs for log querying, it is recommended that you choose to use the ACP Log Storage with Clickhouse plugin.