Validated Models

Ready-to-deploy recipes for validated open-weight LLMs on Alauda AI Platform. Each model in this guide has been deployed end-to-end on a real cluster and benchmarked, so you get a known-good deployment manifest, the runtime image that serves it, and the throughput you can expect.

The models here were validated on Huawei Ascend NPUs (910B4 and 910B3) with the community vLLM-Ascend engine, deployed through Alauda AI Platform's InferNex surface — a KServe LLMInferenceService reconciled by the InferNex-Bridge into a load-aware router (hermes-router / EPP) in front of the vLLM-Ascend instances. Most were run through the same InferNex aggregation surface and the same two benchmark scenarios (the two Qwen models share an identical 2 × TP=4 topology and are directly comparable; DeepSeek-V4-Flash and MiniMax-M2.5 use larger MoE topologies, up to a 16-card DP2 × TP=8 cross-node aggregation). For the runtime model (KServe, ModelCar storage, scheduling) see Deploy.

Qwen3.8-27B and Gemma-4-31B-it (NVFP4) are validated on NVIDIA: each on a single RTX PRO 6000 Blackwell with community vLLM, deployed as a plain KServe LLMInferenceService (no InferNex) behind the MaaS gateway. Their benchmarks use a different method (Exgentic agent replay, grouped by requests in flight) and are reported on their own pages; they are not comparable with the Ascend scenarios below.

Validated models

ModelTypeParamsdtypeDeviceEnginePer-page
Qwen3-32BQwen3 dense32BBF16Ascend 910B4 ×8vLLM-Ascend v0.18.0Qwen3-32B
Qwen3.6-27B (W8A8)qwen3_5 hybrid (GDN + MTP)27.78BW8A8Ascend 910B4 ×8vLLM-Ascend nightlyQwen3.6-27B (W8A8)
DeepSeek-V4-Flash (W4A8)deepseek_v4 MoE (MLA + DSA + MTP)256-expert MoEW4A8Ascend 910B4 ×8vLLM-Ascend nightlyDeepSeek-V4-Flash (W4A8)
DeepSeek-V4-Flash (W8A8)deepseek_v4 MoE (MLA + DSA + MTP)256-expert MoEW8A8Ascend 910B3 ×16vLLM-Ascend nightlyDeepSeek-V4-Flash (W8A8)
MiniMax-M2.5 (W8A8)minimax_m2 MoE (full-attn, 256-expert)230B-A10BW8A8 (QuaRot)Ascend 910B3 ×16vLLM-Ascend v0.23.0MiniMax-M2.5 (W8A8)
Gemma-4-31B-itgemma4 dense multimodal31BBF16Ascend 910B3 ×8vLLM-Ascend nightly-main (0721)Gemma-4-31B-it
GLM-5.2 (W4A8C8)glm_moe_dsa MoE (MLA + DSA + MTP), 1M context753B (~40B activated)W4A8C8Ascend 910B3 ×64vLLM-Ascend v0.23.0GLM-5.2 (W4A8C8)
Qwen3.8-27B (NVFP4)qwen3_5 hybrid (GDN + MTP)27BNVFP4NVIDIA RTX PRO 6000 ×1vLLM v0.31.0Qwen3.8-27B (NVFP4)
Gemma-4-31B-it (NVFP4)gemma4 dense multimodal, served text-only31BNVFP4NVIDIA RTX PRO 6000 ×1vLLM v0.31.0Gemma-4-31B-it (NVFP4)

The two Qwen models and DeepSeek-V4-Flash (W4A8) were validated on Ascend 910B4 (32 GB/card), driven through KServe LLMInferenceService with load-aware routing (InferNex-Bridge + hermes-router). The two Qwen models run an 8-card aggregation — 2 instances × TP=4; DeepSeek-V4-Flash (~151 GB W4A8) fills all 8 cards as 1 instance × TP=8, and additionally validated the MaaS gateway (API-key) ingress next to the internal KServe ingress.

DeepSeek-V4-Flash (W8A8, ~280 GB) was validated on Ascend 910B3 (64 GB/card) across two nodes = 16 cards as a single DP2 × TP=8 cross-node aggregation with a mooncake cross-rank KV store, driven through both the internal KServe ingress and the product MaaS gateway.

MiniMax-M2.5 (W8A8, ~230 GB, 230B-A10B) was validated on the same Ascend 910B3 16-card DP2 × TP=8 cross-node aggregation, but as a plain agg baseline — no mooncake store and no speculative decoding. Because this model is "fast" (~10B activated

  • full attention + W8A8), both add-ons were measured to lose, so the production config keeps only the decode graph + local prefix cache. It was additionally validated as a real coding agent (OpenCode / Pi) through the product MaaS gateway.

Gemma-4-31B-it is a dense BF16 model validated as a P4+D4 composite service across two 910B3 nodes: TP4 Prefill and TP4 Decode exchange KV through Ascend Direct transport. It uses eight NPUs in total; its figures are an operating envelope for that P/D service, not a comparison with a four-card aggregated replica.

Qwen3.8-27B (NVFP4 + DFlash2 speculative decoding) runs on one NVIDIA RTX PRO 6000 Blackwell with the GPU memory budget capped at 84 GB to emulate an RTX PRO 6000D. On a real agent replay whose per-call input and output lengths are close to Alauda Hyperflux's use (Exgentic swebench, median input ~11–16k tokens), the card processes up to 8 requests at the same time with TTFT p90 ≤ 0.7 s and ~50 tok/s per request.

Gemma-4-31B-it (NVFP4 + Google's MTP assistant for speculative decoding) runs on the same single RTX PRO 6000 Blackwell with the same 84 GB budget. On the same agent replay it processes up to 8 requests at the same time with TTFT p90 ≈ 1.0 s and ~46 tok/s per request.

GLM-5.2 (W4A8C8) serves a 1M-token context as a Prefill/Decode-disaggregated service across 8 Atlas 800 A2 nodes (64 × 910B3) — 4 prefill + 4 decode nodes with a shared mooncake KV pool. It is benchmarked by replaying real coding-agent sessions (Exgentic agent traces) swept over concurrent sessions, not with the two Ascend scenarios below: it served 64 concurrent sessions at TTFT P90 9.25 s (line < 10 s); see its page.

Runtime images

EngineDeviceImage (validated tag)Used byNotes
vLLM-Ascend v0.18.0Huawei Ascend NPUquay.io/ascend/vllm-ascend:v0.18.0-openeulerQwen3-32BCommunity vLLM for Ascend, V1 engine. Serves standard Qwen3 dense models directly. Match the tag's CANN version to your host NPU driver.
vLLM-Ascend nightly (release-pinned)Huawei Ascend NPUquay.io/ascend/vllm-ascend:nightly-releases-v0.22.1rc-openeulerQwen3.6-27B (W8A8), DeepSeek-V4-Flash (W4A8)Carries the qwen3_5 Gated DeltaNet hybrid + MTP and the deepseek_v4 MoE stack — the stock v0.18.0 cannot load either. Use this release-pinned tag, not the moving nightly-main-openeuler (it drifted to a broken build whose TP workers crash).
vLLM-Ascend nightly-main (dated snapshot)Huawei Ascend NPUquay.io/ascend/vllm-ascend:nightly-main-openeuler-0709DeepSeek-V4-Flash (W8A8)A later nightly-main snapshot (vLLM 0.23.0) that carries the DP2 cross-node graph + DSA-CP prefix-cache fixes the W8A8 cross-node deployment needs. Pin a dated snapshot like this one, not the moving nightly-main tag.
vLLM-Ascend v0.23.0 (release-pinned)Huawei Ascend NPUquay.io/ascend/vllm-ascend:v0.23.0-openeulerMiniMax-M2.5 (W8A8), GLM-5.2 (W4A8C8)The release-pinned nightly-releases-v0.23.0-openeuler build (vLLM 0.23.0), carrying the minimax_m2 MoE stack + tool-call/reasoning parsers and vLLM #11505 (streaming tool-call arguments held until the closing </parameter>). Prefer this dated release tag over the moving nightly-main.
vLLM-Ascend nightly-main (dated snapshot)Huawei Ascend NPUquay.io/ascend/vllm-ascend:nightly-main-openeuler-0721Gemma-4-31B-itvLLM 0.25.1 / CANN 9.0.1 snapshot used by the validated P4+D4 service. Keep this exact image for the known-good Gemma baseline; it is not a current-version recommendation.
vLLM v0.31.0 (community)NVIDIA GPUdocker.io/vllm/vllm-openai:v0.31.0 (sha256:c1c9f6fd…)Qwen3.8-27B (NVFP4), Gemma-4-31B-it (NVFP4)CUDA 13.0.2, amd64; needs a driver with CUDA 13 support (validated on 580.126.09). Brought in by an LLMInferenceServiceConfig that overrides only the image of the platform's default template.
TIP

The Ascend CANN images are arm64. Always match the runtime image's CANN version to the host NPU driver on your nodes. Only the engines actually used in this guide are listed; other engines (MindIE, SGLang, …) were not benchmarked at this size.

Benchmark scenarios

These models were measured with aiperf against the same two scenarios, modelled on real serving patterns. Output is pinned to 128 tokens and load is closed-loop, concurrency 4 (4 in-flight requests, fixed); each scenario ran 240 requests. DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) are the exceptions — they were swept at concurrency 8 / 16 / 32 (480 requests per tier). Gemma-4-31B-it uses both scenarios and the same concurrency sweep, but is sent directly to its P4+D4 proxy because the MaaS backend was still attached to the older aggregate service at measurement time. GLM-5.2 (W4A8C8) does not use these scenarios: it is measured by replaying coding-agent sessions over a sweep of concurrent sessions — see its page.

ScenarioWhat it modelsDatasetRequest shape
① Fixed-length system-prompt reuseA reused system prompt — performance when request length is fixed and KV cache repeats.60 distinct 8k-token system prompts, each reused 4 times = 240 requests.ISL ~8k / OSL 128
② Multi-turn dialogueMultiple users in a continuing conversation, several requests per session.60 independent users × 4 rounds = 240 requests.Round 1 sends 16k and the model replies 128 tokens; round 2 adds 1k (~17k total); and so on for 4 rounds (16k / 17k / 18k / 19k), averaging ~17.5k ISL / OSL 128.
NOTE

The two Qwen models and DeepSeek-V4-Flash (W4A8) run these scenarios on a single-node 8-card deployment (Qwen models: 2 instances × TP=4; DeepSeek-V4-Flash W4A8: 1 instance × TP=8). Latency (TTFT / ITL / E2E) is the per-instance operating point under steady 2-in-flight load; total throughput (TPS) is the aggregate across the instances and scales with the instance count. TPS is the total-token (input + output) caliber; the decode-only output rate is reported separately and is much smaller under these long-input workloads.

DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) instead run a 16-card DP2 × TP=8 cross-node deployment and are swept at concurrency 8 / 16 / 32 (480 req/tier), so their numbers are an operating envelope reported on each model's own page. Both were driven through both the internal KServe ingress and the product MaaS gateway (API-key) ingress.

Deploy a validated model

Each Ascend model page links self-contained YAMLs under ascend/assets/ that hold the real InferNex deployment — a KServe LLMInferenceService (infernex.io/runtime: true) plus the two LLMInferenceServiceConfig objects (engine template + hermes-router/EPP template) that the InferNex-Bridge reconciles into the running instances.

base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/plan/validated_models/ascend/assets

# 1. Edit the file first: set the namespace and the image tag (match your CANN /
#    host driver). model.uri already points at the public ModelCar on Docker Hub
#    (oci://docker.io/alaudadockerhub/...) — repoint it only if you mirror locally.
# 2. Apply — the InferNex-Bridge reconciles the LLMInferenceService into the
#    hermes-router (EPP) + the vLLM-Ascend instances.
kubectl apply -f $base/qwen3-32b/qwen3-32b-agg-base-llmisvc.yaml

# 3. Watch it come up (first start loads weights + compiles graphs — can take minutes).
kubectl -n <your-namespace> get llminferenceservice -w

# 4. Call the OpenAI-compatible endpoint through the gateway.
curl -s http://<gateway>/<namespace>/qwen3-32b-agg-base/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-32B","messages":[{"role":"user","content":"hello"}]}'

Caveats

  • These manifests deploy through InferNex (LLMInferenceService + InferNex-Bridge
    • hermes-router). The two LLMInferenceServiceConfig objects live in the kserve namespace; the LLMInferenceService lives in your deployment namespace.
  • Resource keys are for Ascend 910B4 (huawei.com/Ascend910). Adjust the resource key, image, and version fields for your actual NPU model.
  • The NVIDIA models are the exception to the two points above: no InferNex and resource key nvidia.com/gpu; each checkpoint and its draft model are two separate ModelCars — see Qwen3.8-27B (NVFP4) and Gemma-4-31B-it (NVFP4).
  • The ModelCar images are public on Docker Hub under alaudadockerhub — the manifests pull them with no credentials. Mirror them to your own registry and repoint model.uri if you prefer; the modelcar pull secret in the manifest is only needed for a private registry.
  • The benchmark numbers were measured closed-loop (concurrency 4) on 8 cards. Treat them as the per-instance operating point under steady load, not a saturation ceiling.

Verify the ModelCar signature

The ModelCar images are signed with Cosign. Verify an image against the published public key (cosign.pub) before deploying:

curl -sO https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/plan/validated_models/cosign.pub

cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  docker.io/alaudadockerhub/modelcar-qwen3-32b:v0.1.0

The signed images and their digests:

ImageDigest
docker.io/alaudadockerhub/modelcar-qwen3-32b:v0.1.0sha256:eccffdb567038196638eb7e1c6bb9572d8bcc7829d11fcdfa17f2c43fbca0c6e
docker.io/alaudadockerhub/modelcar-qwen3-6-27b-w8a8:v0.1.0sha256:5eb8841a29a4f3d659f8cef25c7f8b16208b32265db23354dae79c4c04e8ef79
docker.io/alaudadockerhub/modelcar-deepseek-v4-flash-w4a8-mtp:v0.1.0sha256:ef57a46726bec9c6505504262beaa55ec6c45d595a1726ab90d594f42606c6be
docker.io/alaudadockerhub/modelcar-qwen3.8-27b-nvfp4:v0.1.0sha256:29e36f5bf1d731e3ee175f75ba00821292e21bc6323cfd659533fe702f927aac
docker.io/alaudadockerhub/modelcar-qwen3.8-27b-dflash2:v0.1.0sha256:ab910d1dce269083ea6b6427ce952fefd225d33e03ae60714c5ba6f391d5908a
docker.io/alaudadockerhub/modelcar-gemma-4-31b-it-nvfp4:v0.1.0sha256:1a8eca87ac19759961317474a63db4a6e570f508b1eb23a69d1a410d6baebe23
docker.io/alaudadockerhub/modelcar-gemma-4-31b-it-assistant:v0.1.0sha256:033b599c021bd740b7ad6fd90a449186b9e58a3a15f7ab6849bef53292d14031
docker.io/alaudadockerhub/modelcar-glm-5.2-w4a8c8:v0.1.0sha256:0277c991ec74d38883fdf4490e46cecb2c9eeba2644f1ff809be1ee8cb8b9e7b
NOTE

DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) are not in this Cosign table — their ModelCars are distributed as OCI Image Layout tars (not signed registry images), so there is no Cosign signature to verify. Integrity is content-addressed instead: each tar's index.json carries the image digest and every blob under blobs/sha256/ is verified against its own digest on import (skopeo copy checks this). See the DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) pages.

--insecure-ignore-tlog=true is required because these were signed with --tlog-upload=false (no public transparency-log entry); verification relies on the public key alone.