Alauda AI Platform Gateway
Agent Router
Alauda AI Platform Gateway is based on Agent Router, formerly known as Envoy AI Gateway. Agent Router is a Kubernetes-native, AI-specific gateway layer built on top of Envoy Gateway, providing intelligent traffic management, routing, and policy enforcement for AI inference workloads.
Main components and capabilities include:
- AI-Aware Routing: Routes inference requests to the appropriate backend model service based on request content, model name, and backend availability — enabling transparent multi-model serving behind a single endpoint.
- OpenAI-Compatible API: Exposes a unified, OpenAI-compatible API surface (
/v1/chat/completions,/v1/completions,/v1/models) for all downstream inference services, regardless of the underlying runtime. - Per-Model Rate Limiting & Policies: Enforces fine-grained rate limiting, token quotas, and traffic policies at the individual model level, preventing resource starvation and ensuring fair usage across tenants.
- Backend Load Balancing: Distributes inference requests across multiple replicas of the same model using configurable load-balancing strategies, with health checking and automatic failover.
- Envoy Gateway Integration: Runs as an extension of Envoy Gateway, inheriting its Kubernetes Gateway API-native control plane, TLS termination, and observability features (metrics, access logs, distributed tracing).
- Gateway API Inference Extension (GIE): Integrates with the Kubernetes SIG Gateway API Inference Extension for advanced, inference-aware scheduling and load balancing decisions based on real-time backend state.
Alauda AI Platform Gateway is a required dependency of Alauda AI Platform Inference Management for exposing inference services.
Alauda AI Platform Gateway is installed and managed by Alauda AI Platform through the envoyAIGateway component in the default AmlCluster. The component key is unchanged by the rename. For installation and component configuration, see Install Alauda AI Platform.
Guides
The following guides configure Agent Router as a multi-tenant model serving control plane:
- Authenticating Consumers — verify SSO tokens or API keys and propagate caller identity.
- Configuring Token Quotas — enforce per-user, per-department, and per-tier token budgets.
- Metering Token Usage — report consumption per tenant and feed chargeback.
- Routing to LLM Providers — front external providers with credential injection and failover.
- Charging Back Token Usage — price token usage into per-namespace bills with Cost Management.
Create the Gateway and its AIGatewayRoute in a dedicated namespace, such as maas-system, rather than in the Envoy Gateway control-plane namespace envoy-gateway-system. This keeps tenant gateways separate from the control plane, and avoids an issue on some versions where a gateway placed in the control-plane namespace does not get the AI Gateway request-processing (ext_proc) filter or SecurityPolicy rules applied to its listener — which silently breaks model routing, token quotas, and authentication. The data-plane proxy pods are created in envoy-gateway-system either way; only the Gateway resource's namespace matters here.
Documentation
Agent Router upstream documentation and related resources:
- Agent Router Documentation: https://theagentrouter.ai/ — Official documentation covering architecture, configuration, and API references.
- Agent Router GitHub: https://github.com/theagentrouter/agent-router — Source code, release notes, and issues.
- Envoy Gateway: https://gateway.envoyproxy.io/ — The underlying gateway infrastructure that Agent Router extends.
- Gateway API Inference Extension (GIE): https://gateway-api-inference-extension.sigs.k8s.io/ — Kubernetes SIG project for AI-aware routing integrated with Agent Router.
- KServe (Alauda AI Platform Inference Management): ../kserve/intro — KServe uses Alauda AI Platform Gateway as a required dependency for exposing and routing inference services.