CostManagement Docs

How to Define a HAMi Virtual GPU Cost Model

This procedure registers HAMi virtual-device memory allocation as a billing item. The metric is unified at the HAMi scheduler layer and applies to both soft-slice and hard-slice workloads.

TOC

What this configuration provides1. Create the recording rule2. Create the collection configuration3. Create the display configuration4. Add prices to the cost model5. Verify the result

What this configuration provides

  • Memory allocation: scheduler-confirmed virtual memory allocation, accumulated as GiB-hours.
  • Project quota (optional): project virtual-memory quota, accumulated as GiB-hours.
  • Model pricing: allocation records retain modelName, so prices can vary by backing accelerator model.
  • Runtime metrics: WebUI or device-plugin usage metrics may be used for monitoring, but are not the default billing fact.

1. Create the recording rule

INFO

Where to apply this configuration

Apply this PrometheusRule in the cluster that runs HAMi and its scheduler metrics. Confirm that the scheduler exporter target is healthy before applying it.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: alauda-hami-billing-recording
  namespace: kube-system
  labels:
    prometheus: kube-prometheus
spec:
  groups:
    - name: alauda.billing.hami
      interval: 30s
      rules:
        - record: alauda_hami_virtual_memory_allocated_bytes
          expr: |
            label_replace(
              label_join(
                (
                  hami_vgpu_memory_allocated_bytes{namespace!="kube-system"}
                  * on (device_uuid) group_left (device_type)
                    (0 * max by (device_uuid, device_type) (hami_gpu_memory_limit_bytes) + 1)
                ),
                "virtual_device_id", "/", "device_uuid", "namespace", "pod", "container_index"
              ),
              "label_modelName", "$1", "device_type", "(.+)"
            )
        - record: alauda_hami_project_memory_quota_bytes
          expr: |
            avg by (project, cluster) (
              avg_over_time(
                cpaas_project_resourcequota{
                  resource="requests.nvidia.com/gpumem",
                  type="project-hard"
                }[5m]
              ) * 1048576
            )

hami_vgpu_memory_allocated_bytes is the scheduler allocation fact. It is not the runtime memory currently used by a container. The rule creates a unique virtual_device_id for each device, namespace, pod, and container allocation.

WARNING

The resource="requests.nvidia.com/gpumem" selector is only an example. For example, an HAMi Ascend 310P environment may expose NPU memory as requests.huawei.com/Ascend310P-memory. HAMi NPU memory resources are often model-specific. Replace the selector with the exact memory-resource key exposed by the target environment, including the accelerator model when applicable. Check the resource label and unit of cpaas_project_resourcequota in Prometheus before applying the rule. If the resource key or unit is different, update the selector and the MiB-to-bytes conversion accordingly; otherwise ProjectQuota data may be empty or incorrectly scaled.

The project-quota rule is optional. Remove it when project-quota billing is not required.

Verify the normalized metrics in Prometheus:

alauda_hami_virtual_memory_allocated_bytes
alauda_hami_project_memory_quota_bytes

The allocation metric must return the expected series for both soft-slice and hard-slice workloads before continuing.

2. Create the collection configuration

INFO

Where to apply this configuration

Apply this ConfigMap in the HAMi cluster where the Cost Management Agent runs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: slark-agent-hami-config
  namespace: cpaas-system
  labels:
    cpaas.io/slark.collection.config: "true"
data:
  config: |
    - kind: HAMi
      category: HAMi
      item: hamiVirtualMemoryAllocationGiBHours
      period: Hourly
      usage:
        query: alauda_hami_virtual_memory_allocated_bytes
        step: 5m
        mappers:
          name: virtual_device_id
          namespace: namespace
          cluster: ""
          project: ""
      labels:
        query: alauda_hami_virtual_memory_allocated_bytes
        mappers:
          name: virtual_device_id
          namespace: namespace
          cluster: ""
          project: ""
    - kind: Project
      category: HAMi
      item: projectVirtualDeviceMemoryGiBHours
      period: Hourly
      usage:
        query: alauda_hami_project_memory_quota_bytes
        step: 5m
        mappers:
          name: project
          namespace: ""
          cluster: cluster
          project: project

The source value is bytes. Cost Management stores byte-hours and the display divisor converts it to GiB-hours. Reload the Agent after applying the ConfigMap:

kubectl -n cpaas-system delete pod -l service_name=slark-agent

3. Create the display configuration

INFO

Where to apply this configuration

Apply this ConfigMap in the Global cluster where the Cost Management Server runs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: slark-display-config-for-hami
  namespace: kube-public
  labels:
    cpaas.io/slark.display.config: "true"
data:
  config: |
    - name: HAMi
      displayname: {zh: "HAMi 虚拟设备", en: "HAMi Virtual Device"}
      methods:
        - name: MemoryAllocation
          displayname: {zh: "显存分配量", en: "Allocated vMemory"}
          item: hamiVirtualMemoryAllocationGiBHours
          divisor: 1073741824
          unit: {zh: "GiB·时", en: "GiB-hours"}
        - name: ProjectQuota
          displayname: {zh: "项目显存配额", en: "Project vMemory Quota"}
          item: projectVirtualDeviceMemoryGiBHours
          divisor: 1073741824
          unit: {zh: "GiB·时", en: "GiB-hours"}

divisor: 1073741824 means 1 GiB (1024 x 1024 x 1024 bytes). It converts the byte-hours collected by Cost Management into GiB-hours for display and price calculation. Keep the source and collection data in bytes; do not divide the values in Prometheus or in the collection query before they reach the metering system. This preserves precision and keeps the conversion in one place.

Reload the Cost Management Server after applying the ConfigMap:

kubectl -n cpaas-system delete pod -l service_name=slark-server

4. Add prices to the cost model

In the platform console, open Metering and Billing > Cost Models and edit the cost model linked to the HAMi cluster.

Billing itemMethodUnitPrice rule
HAMi Virtual DeviceAllocated vMemoryGiB-hoursDefault price, with optional modelName overrides
HAMi Virtual DeviceProject vMemory QuotaGiB-hoursProject-level default price
WARNING

Project Quota is optional and has no accelerator model dimension. Do not configure modelName price overrides for this method.

For allocation pricing, configure the exact modelName value emitted by the HAMi scheduler label source. The value is case-sensitive. The backing physical device UUID must not be used as the virtual-device name because several virtual devices can share one physical card.

5. Verify the result

Wait for at least one hourly billing window. Verify the result in this order:

  1. Usage: confirm an allocation record exists with the expected virtual_device_id, namespace, and GiB-hours conversion.
  2. Bill: confirm a bill is generated and the expected default or model-specific price is applied.
  3. Console: confirm the same item and amount appear in cost details and statistics.

For shared deployment rules, see Configuration Conventions.