Logging Docs

Log Component Selection Guide

When installing a log component, the platform provides two log storage options: Elasticsearch and ClickHouse. This article outlines the features and applicable scenarios of these two components to help you choose the most suitable one.

WARNING
  • You can only choose one of Elasticsearch or ClickHouse for the cluster log storage component installation.
  • Any cluster's log storage component can be selected for log collection to interface with the storage data.
  • Now DevOps product does not support archiving Jenkins pipeline execution records using ClickHouse. If you need to use Jenkins pipeline features, please choose Alauda Container Platform Log Storage for ClickHouse cautiously.
  • Now ServiceMesh product does not support integration with ClickHouse. If you need to use service mesh features, please choose Alauda Container Platform Log Storage for ClickHouse cautiously.
  • Now Alauda Container Platform Log Storage for ClickHouse does not support IPv6 single-stack or IPv6 dual-stack workload clusters.

TOC

Architecture ComparisonElasticsearch ArchitectureClickHouse ArchitectureFunction ComparisonSelection Recommendations

Architecture Comparison

Elasticsearch Architecture

Elasticsearch is an open-source distributed search engine built on Lucene, designed for fast full-text search and analysis. Its advantages include:

  • High-performance search: Supports real-time search and can quickly process massive amounts of data.
  • Flexible querying capabilities: Offers a powerful query DSL, supporting complex query requirements.
  • Scalability: Easily horizontally scalable as needed, suitable for applications of all sizes.
  • Diverse data support: Able to handle both structured and unstructured data, widely applicable.

ClickHouse Architecture

ClickHouse is a high-performance columnar database designed for Online Analytical Processing (OLAP). Its advantages include:

  • Fast data processing: Supports rapid querying and analysis through columnar storage and data compression.
  • Real-time analysis: Capable of processing real-time data streams, suitable for real-time data analysis scenarios.
  • High throughput: Optimized for the performance of large-scale data writing and querying, making it very suitable for big data scenarios.
  • Flexible SQL support: Compatible with standard SQL, easy to get started, reducing the usage threshold.
  • Flexible storage planning: Supports S3-backed storage planning for ClickHouse data and tiered cold storage scenarios.

Function Comparison

ClickHouseElasticsearchExplanation
High AvailabilitySupportedSupported
ScalabilitySupportedSupported
Query ExperienceWeakStrongElasticsearch offers more robust search capabilities based on the Lucene language, while ClickHouse only supports SQL queries, limiting its search capabilities.
Resource UsageLowHighFor the same performance requirements, ClickHouse requires fewer resources than Elasticsearch. For example, to support 20,000 logs per second, Elasticsearch needs 3 es-masters and 7 es-nodes (2c4g+8c16g), while ClickHouse only requires 3 2c4g replicas.
PerformanceHighLowUnder the same resource conditions, the log volume supported by ClickHouse far exceeds that of Elasticsearch.
Community ActivityMediumHighThe Elasticsearch community is active with rich documentation, while ClickHouse is a growing and improving community.

Selection Recommendations

  • If you are accustomed to using Elasticsearch and have a high dependency on the Lucene language, it is recommended that you continue to use Alauda Container Platform Log Storage for Elasticsearch.

  • If you depend on the platform's Jenkins pipeline or service mesh features, it is recommended that you continue to use Alauda Container Platform Log Storage for Elasticsearch.

  • If you have high requirements for the performance and resource consumption of the log component but only have basic needs for log querying, it is recommended that you choose Alauda Container Platform Log Storage for ClickHouse.

  • If you need to integrate object storage or use S3-based cold and hot separation for ClickHouse log data, it is recommended that you evaluate Alauda Container Platform Log Storage for ClickHouse first.