logo
Alauda Container Platform
logo
Alauda Container Platform
Navigation

Overview

Architecture
Release Notes

Install

Overview

Prepare for Installation

Prerequisites
Download
Node Preprocessing
Installing
Global Cluster Disaster Recovery

Upgrade

Overview
Pre-Upgrade Preparation
Upgrade the global cluster
Upgrade Workload Clusters

User Interface

Web Console

Overview
Accessing the Web Console
Customizing the Web Console
Customizing the Left Navigation
CLI Tools

Configure

Feature Gate

Clusters

Overview
Creating an On-Premise Cluster
Cluster Node Planning

How to

Add External Address for Built-in Registry
Choosing a Container Runtime
Updating Public Repository Credentials

Backup and Recovery

Overview
Install
Backup repository

Backup Management

Create an application backup schedule
Hooks

Recovery Management

Run an Application Restore Task
Image Registry Replacement

Networking

Introduction

Architecture

Understanding Kube-OVN
Understanding ALB
Understanding MetalLB

Concepts

Auth
Ingress-nginx Annotation Compatibility
TCP/HTTP Keepalive
ModSecurity
Comparison Among Different Ingress Method
HTTP Redirect
L4/L7 Timeout
GatewayAPI
OTel

Guides

Creating Services
Creating Ingresses
Configure Gateway
Creating a Domain Name
Creating Certificates
Creating External IP Address Pool
Creating BGP Peers
Configure Subnets
Configure Network Policies
Creating Admin Network Policies
Configure Cluster Network Policies

How To

Deploy High Available VIP for ALB
Soft Data Center LB Solution (Alpha)
Preparing Kube-OVN Underlay Physical Network
Automatic Interconnection of Underlay and Overlay Subnets
Use OAuth Proxy with ALB
Creating GatewayAPI Gateway
Configure a Load Balancer
How to properly allocate CPU and memory resources
Forwarding IPv6 Traffic to IPv4 Addresses within the Cluster
Calico Network Supports WireGuard Encryption
Kube-OVN Overlay Network Supports IPsec Encryption
ALB Monitoring

Trouble Shooting

How to Solve Inter-node Communication Issues in ARM Environments?
Find Who Cause the Error

Storage

Introduction

Concepts

Core Concepts
Persistent Volume
Access Modes and Volume Modes

Guides

Creating CephFS File Storage Type Storage Class
Creating CephRBD Block Storage Class
Create TopoLVM Local Storage Class
Creating an NFS Shared Storage Class
Deploy Volume Snapshot Component
Creating a PV
Creating PVCs
Using Volume Snapshots

How To

Setting the naming rules for subdirectories in the NFS Shared Storage Class
Generic ephemeral volumes
Using an emptyDir
How to Annotate Third-Party Storage Capabilities

Troubleshooting

Recover From PVC Expansion Failure

Scalability and Performance

Evaluating Resources for Global Cluster
Evaluating Resources for Workload Cluster
Disk Configuration

Storage

Ceph Distributed Storage

Introduction

Install

Create Standard Type Cluster
Create Stretch Type Cluster
Architecture

Concepts

Core Concepts

Guides

Accessing Storage Services
Managing Storage Pools
Node-specific Component Deployment
Adding Devices/Device Classes
Monitoring and Alerts

How To

Configure a Dedicated Cluster for Distributed Storage
Cleanup Distributed Storage

Disaster Recovery

File Storage Disaster Recovery
Block Storage Disaster Recovery
Object Storagge Disaster Recovery
Update the optimization parameters
Permissions

MinIO Object Storage

Introduction
Install
Architecture

Concepts

Core Concepts

Guides

Adding a Storage Pool
Monitoring & Alerts

How To

Data Disaster Recovery

TopoLVM Local Storage

Introduction
Install

Guides

Device Management
Monitoring and Alerting

Security

Alauda Container Security

Security and Compliance

Compliance

Install

Alauda Compliance with Kyverno

API Refiner

Introduction
Install

Users and Roles

User

Introduction

Guides

Manage User Roles
Create User
User Management

Group

Introduction

Guides

Manage User Group Roles
Create Local User Group
Manage Local User Group Membership

Role

Introduction

Guides

Create Role
Manage Custom Roles

IDP

Introduction

Guides

LDAP Management
OIDC Management

Troubleshooting

Delete User

User Policy

Introduction

Multitenancy(Project)

Introduction

Guides

Create Project
Manage Project
Manage Project Cluster
Manage Project Members

Audit

Introduction

Telemetry

Install

Certificates

cert-manager
OLM Certificates
Certificate Monitoring

Virtualization

Virtualization

Overview

Introduction
Features
Install

Images

Introduction

Guides

Adding Virtual Machine Images
Update/Delete Virtual Machine Images
Update/Delete Image Credentials

How To

Creating Windows Images Based on ISO using KubeVirt
Creating Linux Images Based on ISO Using KubeVirt
Exporting Virtual Machine Images
Permissions

Virtual Machine

Introduction

Guides

Creating Virtual Machines/Virtual Machine Groups
Batch Operations on Virtual Machines
Logging into the Virtual Machine using VNC
Managing Key Pairs
Managing Virtual Machines
Monitoring and Alerts
Quick Location of Virtual Machines

How To

Configuring USB host passthrough
Virtual Machine Hot Migration
Virtual Machine Recovery
Virtual Machine Clone
Physical GPU Passthrough Environment Preparation

Troubleshooting

Pod Migration and Recovery from Abnormal Shutdown of Virtual Machine Nodes
Hot Migration Error Messages and Solutions

Network

Introduction

Guides

Configure Network

How To

Control Virtual Machine Network Requests Through Network Policy
Configuring SR-IOV
Configuring Virtual Machines to Use Network Binding Mode for IPv6 Support

Storage

Introduction

Guides

Managing Virtual Disks

Backup and Recovery

Introduction

Guides

Using Snapshots

Developer

Overview

Introduction

Concepts

Resource Unit Description
Application Types
Workload Types
Features

Quick Start

Creating a simple application via image

Building Applications

Overview

Concepts

Understanding Parameters
Understanding Startup Commands
Understanding Environment Variables

Guides

Namespaces

Creating Namespaces
Importing Namespaces
Resource Quota
Limit Range
Pod Security Admission
Overcommit Ratio
Managing Namespace Members
Updating Namespaces
Deleting/Removing Namespaces

Pre-Application-Creation Preparation

Configuring ConfigMap
Configuring Secrets

Creating Applications

Creating applications from Image
Creating applications from Chart
Creating applications from YAML
Creating applications from Code
Creating applications from Operator Backed
Creating applications by using CLI

Post-Application-Creation Configuration

Configuring HPA
Configuring VerticalPodAutoscaler (VPA)
Configuring CronHPA

Operation and Maintenance

Status Description
Starting and Stopping Applications
Updating Applications
Exporting Applications
Updating and deleting Chart Applications
Version Management for Applications
Deleting Applications
Health Checks

Application Observability

Monitoring Dashboards
Logs
Events

Workloads

Deployments
DaemonSets
StatefulSets
CronJobs
Jobs
Working with Helm charts

Pod

Introduction
Pod Parameters
Deleting Pods

Container

Introduction
Debug Container (Alpha)
Entering the Container via EXEC

How To

Setting Scheduled Task Trigger Rules

Registry

Introduction

Install

Install Via YAML
Install Via Web UI

How To

Common CLI Command Operations
Using Alauda Container Platform Registry in Kubernetes Clusters

Source to Image

Introduction

Install

Installing Alauda Container Platform Builds
Architecture

Guides

Managing applications created from Code

How To

Creating an application from Code

Node Isolation Strategy

Introduction
Architecture

Concepts

Core Concepts

Guides

Create Node Isolation Strategy
Permissions
FAQ

GitOps

Introduction

Install

Installing Alauda Build of Argo CD
Installing Alauda Container Platform GitOps

Upgrade

Upgrading Alauda Container Platform GitOps
Architecture

Concepts

GitOps

Argo CD Concept

Introduction
Application
ApplicationSet
Tool
Helm
Kustomize
Directory
Sync
Health

Alauda Container Platform GitOps Concepts

Introduction
Alauda Container Platform GitOps Sync and Health Status

Guides

Creating GitOps Application

Creating GitOps Application
Creating GitOps ApplicationSet

GitOps Observability

Argo CD Component Monitoring
GitOps Applications Ops

How To

Integrating Code Repositories via Argo CD dashboard
Creating an Argo CD Application via Argo CD dashboard
Creating an Argo CD Application via the web console
How to Obtain Argo CD Access Information
Troubleshooting

Extend

Overview
Operator
Cluster Plugin
Upload Packages

Observability

Overview

Introduction
Features

Monitoring

Introduction
Install

Architecture

Monitoring Module Architecture
Monitoring Component Selection Guide
Concepts

Guides

Management of Metrics
Management of Alert
Management of Notification
Manage Dashboards
Management of Probe

How To

Backup and Restore of Prometheus Monitoring Data
VictoriaMetrics Backup and Recovery of Monitoring Data
Collect Network Data from Custom-Named Network Interfaces
Permissions

Distributed Tracing

Introduction
Install
Architecture
Concepts

Guides

Query Tracing
Query Trace Logs

How To

Non-Intrusive Integration of Tracing in Java Applications
Business Log Associated with the TraceID

Troubleshooting

Unable to Query the Required Tracing
Incomplete Tracing Data

Logs

Introduction
Install

Architecture

Log Module Architecture
Log Component Selection Guide
Log Component Capacity Planning
Concepts

Guides

Logs

How To

How to Archive Logs to Third-Party Storage
How to Interface with External ES Storage Clusters
Permissions

Events

Introduction
Events
Permissions

Inspection

Introduction
Architecture

Guides

Inspection
Component Health Status
Permissions

Hardware accelerators

Overview

Introduction
Features
Install

Application Development

Introduction

Guides

CUDA Driver and Runtime Compatibility
Add Custom Devices Using ConfigMap

Troubleshooting

Troubleshooting float16 is only supported on GPUs with compute capability at least xx Error in vLLM
Paddle Autogrow Memory Allocation Crash on GPU-Manager

Configuration Management

Introduction

Guides

Configure Hardware accelerator on GPU nodes

Resource Monitoring

Introduction

Guides

GPU Resource Monitoring

Alauda Service Mesh

About Alauda Service Mesh

Alauda AI

About Alauda AI

Alauda DevOps

About Alauda DevOps

Alauda Cost Management

About Alauda Cost Management

Alauda Application Services

Overview

Introduction
Architecture
Features
Install
Upgrade

Alauda Database Service for MySQL

About Alauda Database Service for MySQL-MGR
About Alauda Database Service for MySQL-PXC

Alauda Cache Service for Redis OSS

About Alauda Cache Service for Redis OSS

Alauda Streaming Service for Kafka

About Alauda Streaming Service for Kafka

Alauda Streaming Service for RabbitMQ

About Alauda Streaming Service for RabbitMQ

Alauda support for PostgreSQL

About Alauda support for PostgreSQL

Operations Management

Introduction

Parameter Template Management

Introduction

Guides

Parameter Template Management
Permissions

Backup Management

Introduction

Guides

External S3 Storage
Backup Management
Permissions

Inspection Management

Introduction

Guides

Create Inspection Task
Exec Inspection Task
Update and Delete Inspection Tasks

How To

How to set Inspection scheduling?

Inspection Optimization Recommendations

MySQL

MySQL IO Load Optimization
MySQL Memory Usage Optimization
MySQL Storage Space Optimization
MySQL Active Thread Count Optimization
MySQL Row Lock Optimization

Redis

Redis BigKey
High CPU Usage in Redis
High Memory Usage in Redis

Kafka

High CPU Utilization in Kafka
Kafka Rebalance Optimization
Kafka Memory Usage Optimization
Kafka Storage Space Optimization

RabbitMQ

RabbitMQ Mnesia Database Exception Handling
Permissions

Alert Management

Introduction

Guides

Relationship with Platform Capabilities
Permissions

Upgrade Management

Introduction

Guides

Instance Upgrade
Permissions

API Reference

Overview

Introduction
Kubernetes API Usage Guide

Advanced APIs

Alert APIs

AlertHistories [v1]
AlertHistoryMessages [v1]
AlertStatus [v2]
SilenceStatus [v2]

Event APIs

Search

Log APIs

Aggregation
Archive
Context
Search

Monitoring APIs

Indicators [monitoring.alauda.io/v1beta1]
Metrics [monitoring.alauda.io/v1beta1]
Variables [monitoring.alauda.io/v1beta1]

Kubernetes APIs

Alert APIs

AlertTemplate [alerttemplates.aiops.alauda.io/v1beta1]
PrometheusRule [prometheusrules.monitoring.coreos.com/v1]

Inspection APIs

Inspection [inspections.ait.alauda.io/v1alpha1]

Notification APIs

Notification [notifications.ait.alauda.io/v1beta1]
NotificationGroup [notificationgroups.ait.alauda.io/v1beta1]
NotificationTemplate [notificationtemplates.ait.alauda.io/v1beta1]
📝 Edit this page on GitHub
Previous PagePre-Upgrade Preparation
Next PageUpgrade Workload Clusters

View full docs as PDF

#Upgrade the global cluster

consists of a global cluster and one or more workload clusters. The global cluster must be upgraded before any workload clusters.

This document walks you through the upgrade procedure for the global cluster.

If the global cluster is configured with the global DR (Disaster Recovery) solution, follow the global DR procedure strictly. Otherwise, follow the Standard procedure .

#TOC

#Standard procedure

WARNING

If you are upgrading from 3.16.x or 3.18.x and have Application Services installed, please refer to the Application Services upgrade guide for additional steps that must be performed before upgrading the global cluster.

#Upload images

Copy the core package to any control plane node of the global cluster. Extract the package and cd into the extracted directory.

  • If the global cluster uses the built-in registry, run:

    bash upgrade.sh --only-sync-image=true
  • If the global cluster uses an external registry, you also need to provide the registry address:

    bash upgrade.sh --only-sync-image=true --registry <registry-address> --username <username> --password <password>

If you plan to upgrade the Operator and Cluster Plugin together during the global cluster upgrade, you can pre-push their images to the global cluster's registry in advance. For bulk upload instructions, see Push only images from all packages in a directory.

INFO

Uploading images typically takes about 2 hours, depending on your network and disk performance.

If your platform is configured for global disaster recovery (DR), remember that the standby global cluster also requires image upload. Be sure to plan your maintenance window accordingly.

WARNING

When using violet to upload packages to a standby cluster, the parameter --dest-repo <VIP addr of standby cluster> must be specified.
Otherwise, the packages will be uploaded to the image repository of the primary cluster, preventing the standby cluster from installing or upgrading extensions.

Also be awared that either authentication info of the standby cluster's image registry or --no-auth parameter MUST be provided.

For details of the violet push subcommand, please refer to Upload Packages.

#Trigger the upgrade

After the image upload is complete, run the following command to start the upgrade process:

bash upgrade.sh --skip-sync-image

Wait for the script to finish before proceeding.

If you have already pre-pushed the Operator and Cluster Plugin images to the global cluster's registry, you can then follow Create only CRs from all packages in a directory. After running this command, wait about 10–15 minutes until upgrade notifications appear for functional components. You will then be able to upgrade the Operator and Cluster Plugin together as part of the subsequent upgrade steps.

#Upgrade the global cluster

  1. Log in to the Web Console of the global cluster and switch to Administrator view.
  2. Navigate to Clusters > Clusters.
  3. Click on the global cluster to open its detail view.
  4. Go to the Functional Components tab.
  5. Click the Upgrade button.

Review the available component updates in the dialog, and confirm to proceed.

INFO
  • Upgrading the Kubernetes version is optional. However, since service disruptions may occur regardless, we recommend including the Kubernetes upgrade to avoid multiple maintenance windows.

  • If the Alauda Container Platform GitOps is installed in the global cluster, and after the upgrading, the pods of the plugin is running abnormally.Please refer to Upgrading Alauda Container Platform GitOps.

#Install Product Docs Plugin

INFO

The Alauda Container Platform Product Docs plugin provides access to product documentation within the platform. All help links throughout the platform will direct users to this documentation. If this plugin is not installed, clicking help links in the platform will result in 404 access errors.

Starting from ACP 4.0, the built-in product documentation has been separated into the Alauda Container Platform Product Docs plugin. If you are upgrading from version 3.18, you need to install this plugin by following these steps:

  1. Navigate to Administrator.

  2. In the left sidebar, click Marketplace > Cluster Plugins and select the global cluster.

  3. Locate the Alauda Container Platform Product Docs plugin and click Install.

#Post-upgrade

#global DR procedure

#Verify data consistency

Follow your regular global DR inspection procedures to ensure that data in the standby global cluster is consistent with the primary global cluster.

If inconsistencies are detected, contact technical support before proceeding.

On both clusters, run the following command to ensure no Machine nodes are in a non-running state:

kubectl get machines.platform.tkestack.io

If any such nodes exist, contact technical support to resolve them before continuing.

#Uninstall the etcd sync plugin

Upgrading from 3.18
Upgrading from 3.16
  1. Access the Web Console of the primary global cluster via its IP or VIP.
  2. Switch to the Platform Management view.
  3. Navigate to Catalog > Cluster Plugin.
  4. Select global from the cluster dropdown.
  5. Find the EtcdSync plugin and click Uninstall. Wait for the uninstallation to complete.

#Upload images

Perform the Upload images step on both the standby cluster and the primary cluster.

See Upload images in Standard procedure for details.

#Upgrade the standby cluster

INFO

Accessing the standby cluster Web Console is required to perform the upgrade.

Before proceeding, verify that the ProductBase resource of the standby cluster is correctly configured with the cluster VIP under spec.alternativeURLs.

If not, update the configuration as follows:

apiVersion: product.alauda.io/v1alpha2
kind: ProductBase
metadata:
  name: base
spec:
  alternativeURLs:
    - https://<standby-cluster-vip>

On the standby cluster, follow the steps in the Standard procedure to complete the upgrade.

#Upgrade the primary cluster

After the standby cluster has been upgraded, proceed with the Standard procedure on the primary cluster.

#Reinstall the etcd sync plugin

Before reinstalling, verify that port 2379 is properly forwarded from both global cluster VIPs to their control plane nodes.

To reinstall:

  1. Access the Web Console of the standby global cluster via its IP or VIP.
  2. Switch to Administrator view.
  3. Go to Marketplace > Cluster Plugins.
  4. Select the global cluster.
  5. Locate Alauda Container Platform etcd Synchronizer, click Install, and provide the required parameters.

To verify installation:

kubectl get po -n cpaas-system -l app=etcd-sync  # Ensure pod is 1/1 Running

kubectl logs -n cpaas-system $(kubectl get po -n cpaas-system -l app=etcd-sync --no-headers | awk '{print $1}' | head -1) | grep -i "Start Sync update"
# Wait until the logs contain "Start Sync update"

# Recreate the pod to trigger synchronization of resources with ownerReferences
kubectl delete po -n cpaas-system $(kubectl get po -n cpaas-system -l app=etcd-sync --no-headers | awk '{print $1}' | head -1)

#Check Synchronization Status

Run the following to verify the synchronization status:

curl "$(kubectl get svc -n cpaas-system etcd-sync-monitor -ojsonpath='{.spec.clusterIP}')/check"

Explanation of output:

  • "LOCAL ETCD missed keys:" – Keys exist in the primary cluster but are missing in the standby. This often resolves after a pod restart.
  • "LOCAL ETCD surplus keys:" – Keys exist in the standby cluster but not in the primary. Review these with your operations team before deletion.