#Enabling D3N Cache for Ceph RGW

D3N (Data Delivery Network) cache accelerates access to hot objects by using local NVMe/SSD disks on RGW nodes as a cache layer. It reduces frequent read operations to backend OSDs and significantly improves read performance for hot data.

#Background

The core idea of D3N is simple but effective: instead of fetching object data repeatedly from the Ceph backend storage, RGW serves hot data directly from a local persistent cache located on fast disks (NVMe/SSD) attached to RGW nodes.

This design is especially beneficial for:

• Large object downloads
• Repeated reads of hot objects
• Bandwidth-sensitive workloads

#Prerequisites

• A Ceph cluster is installed
• Alauda Storage is deployed
• An Object Storage (RGW / CephObjectStore) is created
• High-performance local disks (NVMe / SSD) are available on RGW nodes

#Operation Overview

1. Deploy Ceph and create an Object Storage
2. Prepare a high-speed local disk on RGW nodes and mount it to a directory
3. Configure CephObjectStore to:
   • Mount the directory via hostPath
   • Enable D3N-related RGW parameters

#Prepare Local Filesystem for Cache

Mount a directory on each node hosting the RGW service. It is recommended to use a high-performance dedicated disk (rather than a partition), formatted with the XFS file system, and to configure /etc/fstab to ensure the mount persists after a reboot. Replace </path/to/cache/dir> in the subsequent documentation with the actual directory path you have configured.

#Enable D3N Cache in CephObjectStore

Edit the CephObjectStore resource:

kubectl edit cephobjectstore object-store -n rook-ceph

Add the following configuration under gateway:

gateway:
  additionalVolumeMounts:
  - subPath: cache
    volumeSource:
      hostPath:
        path: </path/to/cache/dir>
  rgwConfig:
    rgw_d3n_l1_datacache_persistent_path: /var/rgw/cache/
    rgw_d3n_l1_datacache_size: "10737418240"
    rgw_d3n_l1_local_datacache_enabled: "true"

#Parameters

#Verify D3N Configuration

#Check RGW Configuration via Ceph Tools

In the rook-ceph-tools pod, run:

ceph config dump | grep d3n

Example output:

client.rgw.obj.a  advanced  rgw_cache_enabled                    true
client.rgw.obj.a  advanced  rgw_d3n_l1_datacache_persistent_path /var/rgw/cache/
client.rgw.obj.a  advanced  rgw_d3n_l1_datacache_size            10737418240
client.rgw.obj.a  advanced  rgw_d3n_l1_local_datacache_enabled   true

#Verify RGW Logs

After the RGW pod restarts, D3N initialization logs should appear:

kubectl logs -n rook-ceph rook-ceph-rgw-object-store-a-xxxx | grep -i d3n

Example output:

rgw_d3n: rgw_d3n_l1_local_datacache_enabled=1
rgw_d3n: rgw_d3n_l1_datacache_persistent_path='/var/rgw/cache/'
rgw_d3n: rgw_d3n_l1_datacache_size=10737418240
rgw_d3n: rgw_d3n_l1_evict_cache_on_start=1
rgw_d3n: rgw_d3n_l1_eviction_policy=lru

#Validate Cache Behavior

When downloading objects through RGW, cached files will appear in the host cache directory:

ls </path/to/cache/dir>

Example output:

40b934ab-6c7e-45e7-9fed-a35bc143ce95...multipart_v2v-demo.mkv.1
40b934ab-6c7e-45e7-9fed-a35bc143ce95...multipart_v2v-demo.mkv.2
40b934ab-6c7e-45e7-9fed-a35bc143ce95...shadow_v2v-demo.mkv.3_1
...

The presence of these files confirms that RGW is serving data from the local D3N cache.

Ceph is a trademark or registered trademark of Red Hat, Inc. or its subsidiaries in the United States and other countries. Rook® is a registered trademark of The Linux Foundation. Alauda is an independent vendor. This product is not affiliated with, endorsed by, or sponsored by LF Projects, LLC or The Linux Foundation. All trademarks are the property of their respective owners and are used here for identification purposes only.