Validated Models
Ready-to-deploy recipes for validated open-weight LLMs on Alauda AI Platform. Each model in this guide has been deployed end-to-end on a real cluster and benchmarked, so you get a known-good deployment manifest, the runtime image that serves it, and the throughput you can expect.
The models here were validated on Huawei Ascend NPUs (910B4 and 910B3) with the
community vLLM-Ascend engine, deployed through Alauda AI Platform's InferNex surface — a
KServe LLMInferenceService reconciled by the InferNex-Bridge into a load-aware router
(hermes-router / EPP) in front of the vLLM-Ascend instances. Most were run through the
same InferNex aggregation surface and the same two benchmark scenarios (the two
Qwen models share an identical 2 × TP=4 topology and are directly comparable;
DeepSeek-V4-Flash and MiniMax-M2.5 use larger MoE topologies, up to a 16-card
DP2 × TP=8 cross-node aggregation). For the runtime model (KServe, ModelCar storage,
scheduling) see Deploy.
Qwen3.8-27B and Gemma-4-31B-it (NVFP4) are validated on NVIDIA: each on a single
RTX PRO 6000 Blackwell with community vLLM, deployed as a plain KServe
LLMInferenceService (no InferNex) behind the MaaS gateway. Their benchmarks use a
different method (Exgentic agent replay, grouped by requests in flight) and are reported
on their own pages; they are not comparable with the Ascend scenarios below.
TOC
Validated modelsRuntime imagesBenchmark scenariosDeploy a validated modelCaveatsVerify the ModelCar signatureValidated models
The two Qwen models and DeepSeek-V4-Flash (W4A8) were validated on Ascend 910B4 (32 GB/card),
driven through KServe LLMInferenceService with load-aware routing (InferNex-Bridge +
hermes-router). The two Qwen models run an 8-card aggregation — 2 instances × TP=4;
DeepSeek-V4-Flash (~151 GB W4A8) fills all 8 cards as 1 instance × TP=8, and
additionally validated the MaaS gateway (API-key) ingress next to the internal
KServe ingress.
DeepSeek-V4-Flash (W8A8, ~280 GB) was validated on Ascend 910B3 (64 GB/card) across two nodes = 16 cards as a single DP2 × TP=8 cross-node aggregation with a mooncake cross-rank KV store, driven through both the internal KServe ingress and the product MaaS gateway.
MiniMax-M2.5 (W8A8, ~230 GB, 230B-A10B) was validated on the same Ascend 910B3
16-card DP2 × TP=8 cross-node aggregation, but as a plain agg baseline — no
mooncake store and no speculative decoding. Because this model is "fast" (~10B activated
- full attention + W8A8), both add-ons were measured to lose, so the production config keeps only the decode graph + local prefix cache. It was additionally validated as a real coding agent (OpenCode / Pi) through the product MaaS gateway.
Gemma-4-31B-it is a dense BF16 model validated as a P4+D4 composite service across two 910B3 nodes: TP4 Prefill and TP4 Decode exchange KV through Ascend Direct transport. It uses eight NPUs in total; its figures are an operating envelope for that P/D service, not a comparison with a four-card aggregated replica.
Qwen3.8-27B (NVFP4 + DFlash2 speculative decoding) runs on one NVIDIA RTX PRO 6000
Blackwell with the GPU memory budget capped at 84 GB to emulate an RTX PRO 6000D. On a
real agent replay whose per-call input and output lengths are close to Alauda Hyperflux's
use (Exgentic swebench, median input ~11–16k tokens), the card processes up to 8
requests at the same time with TTFT p90 ≤ 0.7 s and ~50 tok/s per request.
Gemma-4-31B-it (NVFP4 + Google's MTP assistant for speculative decoding) runs on the same single RTX PRO 6000 Blackwell with the same 84 GB budget. On the same agent replay it processes up to 8 requests at the same time with TTFT p90 ≈ 1.0 s and ~46 tok/s per request.
GLM-5.2 (W4A8C8) serves a 1M-token context as a Prefill/Decode-disaggregated service across 8 Atlas 800 A2 nodes (64 × 910B3) — 4 prefill + 4 decode nodes with a shared mooncake KV pool. It is benchmarked by replaying real coding-agent sessions (Exgentic agent traces) swept over concurrent sessions, not with the two Ascend scenarios below: it served 64 concurrent sessions at TTFT P90 9.25 s (line < 10 s); see its page.
Runtime images
The Ascend CANN images are arm64. Always match the runtime image's CANN version to the host NPU driver on your nodes. Only the engines actually used in this guide are listed; other engines (MindIE, SGLang, …) were not benchmarked at this size.
Benchmark scenarios
These models were measured with aiperf against the same two scenarios, modelled on
real serving patterns. Output is pinned to 128 tokens and load is closed-loop,
concurrency 4 (4 in-flight requests, fixed); each scenario ran 240 requests.
DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) are the exceptions — they were swept at
concurrency 8 / 16 / 32 (480 requests per tier).
Gemma-4-31B-it uses both scenarios and the same concurrency sweep, but is sent directly
to its P4+D4 proxy because the MaaS backend was still attached to the older aggregate
service at measurement time.
GLM-5.2 (W4A8C8) does not use these scenarios: it is measured by replaying coding-agent
sessions over a sweep of concurrent sessions — see its page.
The two Qwen models and DeepSeek-V4-Flash (W4A8) run these scenarios on a single-node 8-card deployment (Qwen models: 2 instances × TP=4; DeepSeek-V4-Flash W4A8: 1 instance × TP=8). Latency (TTFT / ITL / E2E) is the per-instance operating point under steady 2-in-flight load; total throughput (TPS) is the aggregate across the instances and scales with the instance count. TPS is the total-token (input + output) caliber; the decode-only output rate is reported separately and is much smaller under these long-input workloads.
DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) instead run a 16-card DP2 × TP=8 cross-node deployment and are swept at concurrency 8 / 16 / 32 (480 req/tier), so their numbers are an operating envelope reported on each model's own page. Both were driven through both the internal KServe ingress and the product MaaS gateway (API-key) ingress.
Deploy a validated model
Each Ascend model page links self-contained YAMLs under
ascend/assets/
that hold the real InferNex deployment — a KServe LLMInferenceService
(infernex.io/runtime: true) plus the two LLMInferenceServiceConfig objects
(engine template + hermes-router/EPP template) that the InferNex-Bridge reconciles
into the running instances.
Caveats
- These manifests deploy through InferNex (
LLMInferenceService+ InferNex-Bridge- hermes-router). The two
LLMInferenceServiceConfigobjects live in thekservenamespace; theLLMInferenceServicelives in your deployment namespace.
- hermes-router). The two
- Resource keys are for Ascend 910B4 (
huawei.com/Ascend910). Adjust the resource key, image, and version fields for your actual NPU model. - The NVIDIA models are the exception to the two points above: no InferNex and resource
key
nvidia.com/gpu; each checkpoint and its draft model are two separate ModelCars — see Qwen3.8-27B (NVFP4) and Gemma-4-31B-it (NVFP4). - The ModelCar images are public on Docker Hub under
alaudadockerhub— the manifests pull them with no credentials. Mirror them to your own registry and repointmodel.uriif you prefer; the modelcar pull secret in the manifest is only needed for a private registry. - The benchmark numbers were measured closed-loop (concurrency 4) on 8 cards. Treat them as the per-instance operating point under steady load, not a saturation ceiling.
Verify the ModelCar signature
The ModelCar images are signed with Cosign. Verify an
image against the published public key (cosign.pub)
before deploying:
The signed images and their digests:
DeepSeek-V4-Flash (W8A8) and MiniMax-M2.5 (W8A8) are not in this Cosign table —
their ModelCars are distributed as OCI Image Layout tars (not signed registry images),
so there is no Cosign signature to verify. Integrity is content-addressed instead: each
tar's index.json carries the image digest and every blob under blobs/sha256/ is
verified against its own digest on import (skopeo copy checks this). See the
DeepSeek-V4-Flash (W8A8) and
MiniMax-M2.5 (W8A8) pages.
--insecure-ignore-tlog=true is required because these were signed with
--tlog-upload=false (no public transparency-log entry); verification relies on the
public key alone.