#Discovery and Handling of BigKey

BigKeys can significantly impact Redis service performance and stability, potentially leading to service degradation or outages. This document introduces the concept of BigKeys, detection methodologies, and best practices for handling them in production environments.

#What is BigKey

A BigKey in Redis refers not to the key itself being large, but rather to the size of the value associated with the key. BigKeys are generally evaluated based on two criteria: memory consumption and element count. Common thresholds include:

• String: Size exceeds 5MB
• List: Element count exceeds 20,000
• Set: Element count exceeds 10,000
• Sorted Set: Element count exceeds 10,000
• Hash: Field count exceeds 10,000

Note: These thresholds are provided as guidelines and may need adjustment based on your specific environment and requirements.

#Impact of BigKey

BigKeys can adversely affect Redis service performance and stability in several ways:

• Thread Blocking: Redis processes data using a single thread model. Operations on BigKeys can be time-consuming, blocking the execution of other commands and degrading overall service performance.
• Client Timeouts: Due to the single-threaded architecture, operations on BigKeys may cause Redis to become unresponsive to client requests, leading to connection timeouts.
• Network Bandwidth Saturation: Transmitting BigKeys consumes substantial network bandwidth. For example, a 1MB key accessed 1,000 times per second would generate 1GB/s of network traffic, potentially overwhelming standard network infrastructure.

In cluster deployments, additional concerns arise:

• Memory Imbalance: BigKeys can cause disproportionate memory consumption on individual nodes, resulting in suboptimal resource utilization across the cluster.
• Migration Bottlenecks: During scaling operations, migrating BigKeys can block Redis for extended periods, disrupting normal operations and potentially compromising cluster stability.

#Detection of BigKey

#Detecting with Inspection Tools

Alauda Application Services provides Inspections for you to detect BigKey in Redis instances.

1. Edit the inspection tool deployment configuration to enable BigKey detection.

On the cluster where the instance is located, edit the RdsInstaller function to enable BigKey detection.

# First check the RdsInstaller configuration
$ kubectl -n rds-system get RdsInstaller rds -o yaml

The output is as follows:

apiVersion: middleware.alauda.io/v1
kind: RdsInstaller
metadata:
  name: rds
spec:
  ......
  inspection:
    concurrency: "10"
    dependency: true
    env:
    - name: ENABLE_REDIS_KEYS_INDICATOR
    image: xxx
    imagePullPolicy: Always
    instanceReportLimit: "10"
    name: inspection-operator
    namespace: rds-system
    ......

2. Modify the value of the ENABLE_REDIS_KEYS_INDICATOR environment variable and save.

$ kubectl -n rds-system edit RdsInstaller rds

Change the value of ENABLE_REDIS_KEYS_INDICATOR in spec.inspection.env to 1. The modified instance is as follows:

apiVersion: middleware.alauda.io/v1
kind: RdsInstaller
metadata:
  name: rds
spec:
  ......
  inspection:
    concurrency: "10"
    dependency: true
    env:
    - name: ENABLE_REDIS_KEYS_INDICATOR
      value: "1"
    image: xxx
    imagePullPolicy: Always
    instanceReportLimit: "10"
    name: inspection-operator
    namespace: rds-system
    ......

After saving, the inspection Operator tool will restart. You can check the restart progress with the following command:

$ kubectl -n rds-system get pods -l name.operator=inspection-operator

The output is as follows:

$ kubectl -n rds-system get pods -l name.operator=inspection-operator
NAME                                   READY   STATUS    RESTARTS   AGE
inspection-operator-545468bd54-9g65p   1/1     Running   0          8s

Note: The above modification is not permanently effective; the configuration will be overwritten during platform upgrades.

3. Execute the inspection

In Redis -> Details Info, click the Inspection button to start the instance inspection.

Once the inspection starts, the button will display Inspecting.... After completion, the inspection button will become clickable.

Due to the enabled BigKey detection function, the inspection tool will SCAN all data in the cache; hence, this process may take a considerable amount of time.

4. View the inspection results

Upon completion of the inspection, you can click the Query button in the inspection report on the Redis -> Detail Info to view the inspection results.

If BigKey does indeed exist in the instance, the inspection result will list the BigKey Top5 items, as shown in the figure below:

#Usage Limitations

• The inspection tool performs a full scan of Redis data, which can place a certain amount of stress on the Redis service; it is recommended to use during low business peaks.
• Currently, the inspection tool only supports detecting string, list, and zset types.

#Detecting with Command Line

The Redis community offers functionality to detect BigKey using redis-cli. During detection, it traverses/samples all Keys in the Redis instance and returns overall statistics of the Keys and the largest Key of each data type. The bigkeys can currently analyze string, list, set, zset, hash, and stream types. The execution command is as follows:

$ redis-cli -h <host> -a <password> --bigkeys

#Usage Limitations

• The redis-cli detection also performs a full scan of Redis data, which may impose certain pressure on the Redis service; it is recommended to use during low business peaks.

#Optimization of BigKey

It is necessary to choose appropriate optimization schemes based on specific business scenarios and data characteristics, generally focusing on the following aspects:

#Optimization of Storage Data Structures

For collection types with numerous data members (such as list, set, zset, hash, etc.), split them into multiple Keys and ensure that the number of members per Key is within a reasonable range. In a Redis cluster architecture, splitting large Keys can significantly enhance memory balance between data shards.

#Compressing Data

If a substantial amount of JSON or HTML cache is stored, consider compressing the data. Additionally, serialization protocols such as ProtoBuffer or MessagePack can be utilized to reduce the data size.

#Regular Cleanup of Expired Data

For data with a definite expiration time, you can automatically clean up expired data by setting expiration dates. For data with uncertain expiration times, it can be cleaned up periodically. This is particularly relevant for data structures like list, set, zset, and hash, which can accumulate a large amount of expired data; therefore, combining SCAN and DEL to clean up invalid members is advisable.

Regardless of which optimization scheme is chosen, it is crucial to be particularly cautious when using the DEL command to clean up existing BigKey, as the DEL command will block the Redis service. It is recommended to use the UNLINK command instead, as the UNLINK command will asynchronously delete Keys without blocking the Redis service.