logo
Alauda Container Platform
logo
Alauda Container Platform
Navigation

Overview

Architecture
Release Notes

Install

Overview

Prepare for Installation

Prerequisites
Download
Node Preprocessing
Installing
Global Cluster Disaster Recovery

Upgrade

Overview
Pre-Upgrade Preparation
Upgrade the global cluster
Upgrade Workload Clusters

User Interface

Web Console

Overview
Accessing the Web Console
Customizing the Web Console
Customizing the Left Navigation
CLI Tools

Configure

Feature Gate

Clusters

Overview
Creating an On-Premise Cluster
Cluster Node Planning

How to

Add External Address for Built-in Registry
Choosing a Container Runtime
Updating Public Repository Credentials

Backup and Recovery

Overview
Install
Backup repository

Backup Management

Create an application backup schedule
Hooks

Recovery Management

Run an Application Restore Task
Image Registry Replacement

Networking

Introduction

Architecture

Understanding Kube-OVN
Understanding ALB
Understanding MetalLB

Concepts

Auth
Ingress-nginx Annotation Compatibility
TCP/HTTP Keepalive
ModSecurity
Comparison Among Different Ingress Method
HTTP Redirect
L4/L7 Timeout
GatewayAPI
OTel

Guides

Creating Services
Creating Ingresses
Configure Gateway
Creating a Domain Name
Creating Certificates
Creating External IP Address Pool
Creating BGP Peers
Configure Subnets
Configure Network Policies
Creating Admin Network Policies
Configure Cluster Network Policies

How To

Deploy High Available VIP for ALB
Soft Data Center LB Solution (Alpha)
Preparing Kube-OVN Underlay Physical Network
Automatic Interconnection of Underlay and Overlay Subnets
Use OAuth Proxy with ALB
Creating GatewayAPI Gateway
Configure a Load Balancer
How to properly allocate CPU and memory resources
Forwarding IPv6 Traffic to IPv4 Addresses within the Cluster
Calico Network Supports WireGuard Encryption
Kube-OVN Overlay Network Supports IPsec Encryption
ALB Monitoring

Trouble Shooting

How to Solve Inter-node Communication Issues in ARM Environments?
Find Who Cause the Error

Storage

Introduction

Concepts

Core Concepts
Persistent Volume
Access Modes and Volume Modes

Guides

Creating CephFS File Storage Type Storage Class
Creating CephRBD Block Storage Class
Create TopoLVM Local Storage Class
Creating an NFS Shared Storage Class
Deploy Volume Snapshot Component
Creating a PV
Creating PVCs
Using Volume Snapshots

How To

Setting the naming rules for subdirectories in the NFS Shared Storage Class
Generic ephemeral volumes
Using an emptyDir
How to Annotate Third-Party Storage Capabilities

Troubleshooting

Recover From PVC Expansion Failure

Scalability and Performance

Evaluating Resources for Global Cluster
Evaluating Resources for Workload Cluster
Disk Configuration

Storage

Ceph Distributed Storage

Introduction

Install

Create Standard Type Cluster
Create Stretch Type Cluster
Architecture

Concepts

Core Concepts

Guides

Accessing Storage Services
Managing Storage Pools
Node-specific Component Deployment
Adding Devices/Device Classes
Monitoring and Alerts

How To

Configure a Dedicated Cluster for Distributed Storage
Cleanup Distributed Storage

Disaster Recovery

File Storage Disaster Recovery
Block Storage Disaster Recovery
Object Storagge Disaster Recovery
Update the optimization parameters
Permissions

MinIO Object Storage

Introduction
Install
Architecture

Concepts

Core Concepts

Guides

Adding a Storage Pool
Monitoring & Alerts

How To

Data Disaster Recovery

TopoLVM Local Storage

Introduction
Install

Guides

Device Management
Monitoring and Alerting

Security

Alauda Container Security

Security and Compliance

Compliance

Install

Alauda Compliance with Kyverno

API Refiner

Introduction
Install

Users and Roles

User

Introduction

Guides

Manage User Roles
Create User
User Management

Group

Introduction

Guides

Manage User Group Roles
Create Local User Group
Manage Local User Group Membership

Role

Introduction

Guides

Create Role
Manage Custom Roles

IDP

Introduction

Guides

LDAP Management
OIDC Management

Troubleshooting

Delete User

User Policy

Introduction

Multitenancy(Project)

Introduction

Guides

Create Project
Manage Project
Manage Project Cluster
Manage Project Members

Audit

Introduction

Telemetry

Install

Certificates

cert-manager
OLM Certificates
Certificate Monitoring

Virtualization

Virtualization

Overview

Introduction
Features
Install

Images

Introduction

Guides

Adding Virtual Machine Images
Update/Delete Virtual Machine Images
Update/Delete Image Credentials

How To

Creating Windows Images Based on ISO using KubeVirt
Creating Linux Images Based on ISO Using KubeVirt
Exporting Virtual Machine Images
Permissions

Virtual Machine

Introduction

Guides

Creating Virtual Machines/Virtual Machine Groups
Batch Operations on Virtual Machines
Logging into the Virtual Machine using VNC
Managing Key Pairs
Managing Virtual Machines
Monitoring and Alerts
Quick Location of Virtual Machines

How To

Configuring USB host passthrough
Virtual Machine Hot Migration
Virtual Machine Recovery
Virtual Machine Clone
Physical GPU Passthrough Environment Preparation

Troubleshooting

Pod Migration and Recovery from Abnormal Shutdown of Virtual Machine Nodes
Hot Migration Error Messages and Solutions

Network

Introduction

Guides

Configure Network

How To

Control Virtual Machine Network Requests Through Network Policy
Configuring SR-IOV
Configuring Virtual Machines to Use Network Binding Mode for IPv6 Support

Storage

Introduction

Guides

Managing Virtual Disks

Backup and Recovery

Introduction

Guides

Using Snapshots

Developer

Overview

Introduction

Concepts

Resource Unit Description
Application Types
Workload Types
Features

Quick Start

Creating a simple application via image

Building Applications

Overview

Concepts

Understanding Parameters
Understanding Startup Commands
Understanding Environment Variables

Guides

Namespaces

Creating Namespaces
Importing Namespaces
Resource Quota
Limit Range
Pod Security Admission
Overcommit Ratio
Managing Namespace Members
Updating Namespaces
Deleting/Removing Namespaces

Pre-Application-Creation Preparation

Configuring ConfigMap
Configuring Secrets

Creating Applications

Creating applications from Image
Creating applications from Chart
Creating applications from YAML
Creating applications from Code
Creating applications from Operator Backed
Creating applications by using CLI

Post-Application-Creation Configuration

Configuring HPA
Configuring VerticalPodAutoscaler (VPA)
Configuring CronHPA

Operation and Maintenance

Status Description
Starting and Stopping Applications
Updating Applications
Exporting Applications
Updating and deleting Chart Applications
Version Management for Applications
Deleting Applications
Health Checks

Application Observability

Monitoring Dashboards
Logs
Events

Workloads

Deployments
DaemonSets
StatefulSets
CronJobs
Jobs
Working with Helm charts

Pod

Introduction
Pod Parameters
Deleting Pods

Container

Introduction
Debug Container (Alpha)
Entering the Container via EXEC

How To

Setting Scheduled Task Trigger Rules

Registry

Introduction

Install

Install Via YAML
Install Via Web UI

How To

Common CLI Command Operations
Using Alauda Container Platform Registry in Kubernetes Clusters

Source to Image

Introduction

Install

Installing Alauda Container Platform Builds
Architecture

Guides

Managing applications created from Code

How To

Creating an application from Code

Node Isolation Strategy

Introduction
Architecture

Concepts

Core Concepts

Guides

Create Node Isolation Strategy
Permissions
FAQ

GitOps

Introduction

Install

Installing Alauda Build of Argo CD
Installing Alauda Container Platform GitOps

Upgrade

Upgrading Alauda Container Platform GitOps
Architecture

Concepts

GitOps

Argo CD Concept

Introduction
Application
ApplicationSet
Tool
Helm
Kustomize
Directory
Sync
Health

Alauda Container Platform GitOps Concepts

Introduction
Alauda Container Platform GitOps Sync and Health Status

Guides

Creating GitOps Application

Creating GitOps Application
Creating GitOps ApplicationSet

GitOps Observability

Argo CD Component Monitoring
GitOps Applications Ops

How To

Integrating Code Repositories via Argo CD dashboard
Creating an Argo CD Application via Argo CD dashboard
Creating an Argo CD Application via the web console
How to Obtain Argo CD Access Information
Troubleshooting

Extend

Overview
Operator
Cluster Plugin
Upload Packages

Observability

Overview

Introduction
Features

Monitoring

Introduction
Install

Architecture

Monitoring Module Architecture
Monitoring Component Selection Guide
Concepts

Guides

Management of Metrics
Management of Alert
Management of Notification
Manage Dashboards
Management of Probe

How To

Backup and Restore of Prometheus Monitoring Data
VictoriaMetrics Backup and Recovery of Monitoring Data
Collect Network Data from Custom-Named Network Interfaces
Permissions

Distributed Tracing

Introduction
Install
Architecture
Concepts

Guides

Query Tracing
Query Trace Logs

How To

Non-Intrusive Integration of Tracing in Java Applications
Business Log Associated with the TraceID

Troubleshooting

Unable to Query the Required Tracing
Incomplete Tracing Data

Logs

Introduction
Install

Architecture

Log Module Architecture
Log Component Selection Guide
Log Component Capacity Planning
Concepts

Guides

Logs

How To

How to Archive Logs to Third-Party Storage
How to Interface with External ES Storage Clusters
Permissions

Events

Introduction
Events
Permissions

Inspection

Introduction
Architecture

Guides

Inspection
Component Health Status
Permissions

Hardware accelerators

Overview

Introduction
Features
Install

Application Development

Introduction

Guides

CUDA Driver and Runtime Compatibility
Add Custom Devices Using ConfigMap

Troubleshooting

Troubleshooting float16 is only supported on GPUs with compute capability at least xx Error in vLLM
Paddle Autogrow Memory Allocation Crash on GPU-Manager

Configuration Management

Introduction

Guides

Configure Hardware accelerator on GPU nodes

Resource Monitoring

Introduction

Guides

GPU Resource Monitoring

Alauda Service Mesh

About Alauda Service Mesh

Alauda AI

About Alauda AI

Alauda DevOps

About Alauda DevOps

Alauda Cost Management

About Alauda Cost Management

Alauda Application Services

Overview

Introduction
Architecture
Features
Install
Upgrade

Alauda Database Service for MySQL

About Alauda Database Service for MySQL-MGR
About Alauda Database Service for MySQL-PXC

Alauda Cache Service for Redis OSS

About Alauda Cache Service for Redis OSS

Alauda Streaming Service for Kafka

About Alauda Streaming Service for Kafka

Alauda Streaming Service for RabbitMQ

About Alauda Streaming Service for RabbitMQ

Alauda support for PostgreSQL

About Alauda support for PostgreSQL

Operations Management

Introduction

Parameter Template Management

Introduction

Guides

Parameter Template Management
Permissions

Backup Management

Introduction

Guides

External S3 Storage
Backup Management
Permissions

Inspection Management

Introduction

Guides

Create Inspection Task
Exec Inspection Task
Update and Delete Inspection Tasks

How To

How to set Inspection scheduling?

Inspection Optimization Recommendations

MySQL

MySQL IO Load Optimization
MySQL Memory Usage Optimization
MySQL Storage Space Optimization
MySQL Active Thread Count Optimization
MySQL Row Lock Optimization

Redis

Redis BigKey
High CPU Usage in Redis
High Memory Usage in Redis

Kafka

High CPU Utilization in Kafka
Kafka Rebalance Optimization
Kafka Memory Usage Optimization
Kafka Storage Space Optimization

RabbitMQ

RabbitMQ Mnesia Database Exception Handling
Permissions

Alert Management

Introduction

Guides

Relationship with Platform Capabilities
Permissions

Upgrade Management

Introduction

Guides

Instance Upgrade
Permissions

API Reference

Overview

Introduction
Kubernetes API Usage Guide

Advanced APIs

Alert APIs

AlertHistories [v1]
AlertHistoryMessages [v1]
AlertStatus [v2]
SilenceStatus [v2]

Event APIs

Search

Log APIs

Aggregation
Archive
Context
Search

Monitoring APIs

Indicators [monitoring.alauda.io/v1beta1]
Metrics [monitoring.alauda.io/v1beta1]
Variables [monitoring.alauda.io/v1beta1]

Kubernetes APIs

Alert APIs

AlertTemplate [alerttemplates.aiops.alauda.io/v1beta1]
PrometheusRule [prometheusrules.monitoring.coreos.com/v1]

Inspection APIs

Inspection [inspections.ait.alauda.io/v1alpha1]

Notification APIs

Notification [notifications.ait.alauda.io/v1beta1]
NotificationGroup [notificationgroups.ait.alauda.io/v1beta1]
NotificationTemplate [notificationtemplates.ait.alauda.io/v1beta1]
📝 Edit this page on GitHub
Previous PageHow to Obtain Argo CD Access Information
Next PageExtend

View full docs as PDF

#Troubleshooting

#TOC

#I've deleted/corrupted my repo and can't delete my app?

Argo CD can't delete an app if it cannot generate manifests. You need to either:

  1. Reinstate/fix your repo.
  2. Delete the app using --cascade=false and then manually deleting the resources.

#Why is my application still OutOfSync immediately after a successful Sync?

See Diffing Documentation for reasons resources can be OutOfSync, and ways to configure Argo CD to ignore fields when differences are expected.

#Why is my application stuck in Progressing state?

Argo CD provides health for several standard Kubernetes types. The Ingress, StatefulSet and SealedSecret types have known issues which might cause health check to return Progressing state instead of Healthy.

  • Ingress is considered healthy if status.loadBalancer.ingress list is non-empty, with at least one value for hostname or IP. Some ingress controllers (contour , traefik) don't update status.loadBalancer.ingress field which causes Ingress to stuck in Progressing state forever.

  • StatefulSet is considered healthy if value of status.updatedReplicas field matches to spec.replicas field. Due to Kubernetes bug kubernetes#68573 the status.updatedReplicas is not populated. So unless you run Kubernetes version which include the fix kubernetes#67570 StatefulSet might stay in Progressing state.

  • Your StatefulSet or DaemonSet is using OnDelete instead of RollingUpdate strategy.

  • For SealedSecret, see Why are resources of type SealedSecret stuck in the Progressing state?

As workaround Argo CD allows providing health check customization which overrides default behavior.

If you are using Traefik for your Ingress, you can update the Traefik config to publish the loadBalancer IP using publishedservice, which will resolve this issue.

providers:
  kubernetesIngress:
    publishedService:
      enabled: true

#How to disable admin user?

Add admin.enabled: "false" in the argocd-cm ConfigMap.

#Argo CD cannot deploy Helm Chart based applications without internet access, how can I solve it?

Argo CD might fail to generate Helm chart manifests if the chart has dependencies located in external repositories. To solve the problem you need to make sure that requirements.yaml uses only internally available Helm repositories. Even if the chart uses only dependencies from internal repos Helm might decide to refresh stable repo. As workaround override stable repo URL in argocd-cmconfig map:

data:
  repositories: |
    - type: helm
      url: http://<internal-helm-repo-host>:8080
      name: stable

#After creating my Helm application with Argo CD I cannot see it with helm ls and other Helm commands?

When deploying a Helm application Argo CD is using Helm only as a template mechanism. It runs helm template and then deploys the resulting manifests on the cluster instead of doing helm install. This means that you cannot use any Helm command to view/verify the application. It is fully managed by Argo CD. Note that Argo CD supports natively some capabilities that you might miss in Helm (such as the history and rollback commands).

This decision was made so that Argo CD is neutral to all manifest generators.

#I've configured cluster secret but it does not show up in CLI/UI, how do I fix it?

Check if the cluster secret has the label argocd.argoproj.io/secret-type: cluster. If the secret has the label but the cluster is still not visible, it may be a permission issue. Try listing the clusters using the admin user (e.g.: argocd login --username admin && argocd cluster list).

Check if cluster secret has argocd.argoproj.io/secret-type: cluster label. If secret has the label but the cluster is still not visible then make sure it might be a permission issue. Try to list clusters using admin user (e.g. argocd login --username admin && argocd cluster list).

#Why Is My App Out Of Sync Even After Syncing?

In some cases, the tool you use may conflict with Argo CD by adding the app.kubernetes.io/instance label. E.g. using Kustomize common labels feature.

Argo CD automatically sets the app.kubernetes.io/instance label and uses it to determine which resources form the app. If the tool does this too, this causes confusion. You can change this label by setting the application.instanceLabelKey value in the argocd-cm. We recommend that you use argocd.argoproj.io/instance.

INFO

When you make this change your applications will become out of sync and will need re-syncing.

#How often does Argo CD check for changes to my Git or Helm repository?

The default polling interval is 3 minutes (180 seconds) with a configurable jitter. You can change the setting by updating the timeout.reconciliation value and the timeout.reconciliation.jitter in the argocd-cm config map. If there are any Git changes, Argo CD will only update applications with the auto-sync setting enabled. If you set it to 0 then Argo CD will stop polling Git repositories automatically and you can only use alternative methods such as webhooks and/or manual syncs for creating applications.

#How Do I Fix invalid cookie, longer than max length 4093?

Argo CD uses a JWT as the auth token. You likely are part of many groups and have gone over the 4KB limit which is set for cookies. You can get the list of groups by opening "developer tools -> network":

  1. Click login to the Argo CD dashboard How to Obtain Argo CD Access Information
  2. Find the call to <argocd_instance>/auth/callback?code=<random_string>

Decode the token at jwt.io. That will provide the list of teams that you can remove yourself from.

#Why Am I Getting rpc error: code = Unavailable desc = transport is closing When Using The CLI?

Maybe you're behind a proxy that does not support HTTP 2? Try the --grpc-web flag:

argocd ... --grpc-web

#Why are resources of type SealedSecret stuck in the Progressing state?

The controller of the SealedSecret resource may expose the status condition on resource it provisioned. Since version v2.0.0 Argo CD picks up that status condition to derive a health status for the SealedSecret.

Versions before v0.15.0 of the SealedSecret controller are affected by an issue regarding this status conditions updates, which is why this feature is disabled by default in these versions. Status condition updates may be enabled by starting the SealedSecret controller with the --update-status command line parameter or by setting the SEALED_SECRETS_UPDATE_STATUS environment variable.

To disable Argo CD from checking the status condition on SealedSecret resources, add the following resource customization in your argocd-cm ConfigMap via resource.customizations.health.<group_kind> key.

resource.customizations.health.bitnami.com_SealedSecret: |
  hs = {}
  hs.status = "Healthy"
  hs.message = "Controller doesn't report resource status"
  return hs

#How to rotate Redis keys?

  • Delete argocd-redis secret in the namespace where Argo CD is installed.

    kubectl delete secret argocd-redis -n <argocd namesapce>
  • If you are running Redis in HA mode, restart Redis in HA.

    kubectl rollout restart deployment argocd-redis-ha-haproxy
    kubectl rollout restart statefulset argocd-redis-ha-server
  • If you are running Redis in non-HA mode, restart Redis.

    kubectl rollout restart deployment argocd-redis
  • Restart other components.

    kubectl rollout restart deployment argocd-server argocd-repo-server
    kubectl rollout restart statefulset argocd-application-controller

#How do I fix Manifest generation error (cached)?

Manifest generation error (cached) means that there was an error when generating manifests and that the error message has been cached to avoid runaway retries.

Doing a hard refresh (ignoring the cached error) can overcome transient issues. But if there's an ongoing reason manifest generation is failing, a hard refresh will not help.

Instead, try searching the repo-server logs for the app name in order to identify the error that is causing manifest generation to fail.