Install InferNex Bridge
This guide describes enabling Alauda Build of InferNex Bridge on arm64 clusters through the existing default AmlCluster. Alauda AI Platform manages installation; users do not select an OLM channel or install the operator manually. See the component matrix for the version included in your platform release.
TOC
PrerequisitesRequired DependenciesCRDs Installed by This OperatorRuntime Templates and ImagesConfigure ImageWhiteListPrepare the ModelOptional DependenciesOptional EagleEye ObservabilityEnable InferNex Bridge and NPU Serving RuntimeVerificationCommunity ExamplesUpgrading InferNex BridgeVerificationRollbackPrerequisites
Before installing Alauda Build of InferNex Bridge, ensure the target cluster has the required platform and inference dependencies.
Required Dependencies
InferNex Bridge is installed and managed by Alauda AI Platform through the infernexBridge component in the default AmlCluster; it does not require installing the InferNex main chart. For Ascend inference, enable servingRuntimeNPUOperator as well. The Alauda AI Platform NPU Serving Runtime supplies the vLLM-Ascend image, ClusterServingRuntime, and LLMInferenceServiceConfig; model files and any feature-specific dependencies must still be prepared separately.
CRDs Installed by This Operator
The Alauda Build of InferNex Bridge OLM bundle installs the two product APIs and three CRDs used by its managed elastic orchestration components:
The following CRDs are not installed by this OLM bundle. Install them separately according to the requirement shown below:
The rolebasedgroups.workloads.x-k8s.io CRD is provided by the upstream RoleBasedGroup (RBG) project. It is a conditional dependency and is not required for Deployment or LeaderWorkerSet targets. To install the upstream RBG v0.7.0 controller and CRDs, use the versioned release manifest:
The expected result is v1alpha2 served=true storage=true. The manifest uses
docker.io/rolebasedgroup/rbgs-controller:v0.7.0-888ade7, which provides an
arm64 image. For a disconnected cluster, mirror this image to a registry
accessible from the cluster and update the manifest image reference before
applying it.
See the upstream RBG releases when selecting a different version.
Enabling the managed Tidal, ElasticScaler, or ResourceScalingGroup components
deploys their controller workloads. Scaling policy resources such as Tidal
and ElasticScaler must still be created separately by the user or platform
workflow.
Runtime Templates and Images
For Ascend inference, use Alauda AI Platform NPU Serving Runtime rather than manually installing the vLLM-Ascend runtime image or creating its ClusterServingRuntime and LLMInferenceServiceConfig. Enable servingRuntimeNPUOperator as described in NPU Serving Runtime setup; the component manages these resources.
InferNex Bridge's own runtime templates and component images may still reference their upstream repositories. In restricted or disconnected environments, verify that the package-generated ImageWhiteList contains the required image references and rewrite rules, and mirror any additional InferNex Bridge images to a registry accessible by the cluster.
Configure ImageWhiteList
The platform creates an operator-managed ImageWhiteList when the InferNex Bridge package is uploaded. Do not create or delete this object manually. Operator-managed ImageWhiteList objects are reconciled by olm-registry and may be recreated after deletion.
Before creating an inference service, verify that the generated object contains the required source image references and source-to-target rewrite rules. Keep unrelated entries that are already present.
The following table shows the expected repository mappings. Each rule preserves the original image tag.
The relevant part of the final object should look like this. The complete object contains additional package images and platform-managed entries.
For BusyBox, retain both busybox:1.36.1 and
docker.io/library/busybox:1.36.1 in repoList, but do not add a custom
BusyBox rewrite rule. The platform _default rule handles the short name and
Docker Hub normalization.
You can back up and edit the operator-managed object with these commands. If
the object has a different name, locate it with
kubectl get imagewhitelist -A.
The workload specification may continue to show the community source image.
The rewrite is successful only when the actual Pod imageID points to the
target registry. For example, an InferNex Bridge component should resolve as follows:
Verify that the InferNex Bridge component images resolve to the intended registry by inspecting Pod image IDs:
Prepare the Model
Model artifacts are not included in the InferNex Bridge operator package. Use the Alauda AI Platform model catalog to prepare a ModelCar when OCI model storage is selected. See Model Storage.
The Alauda OLM bundle registers the InferNex Bridge admission webhook for the KServe LLMInferenceService API versions used by the release examples, including serving.kserve.io/v1alpha2. The webhook is used for admission-time compatibility patches when infernex.io/runtime: "true" is set on a KServe LLMInferenceService; it does not create or reconcile the LLMInferenceService resource itself.
Optional Dependencies
Optional EagleEye Observability
EagleEye is disabled by default in the Alauda OLM templates. Disabling EagleEye does not affect the inference engine, aggregate or prefill-decode deployment modes, Mooncake, cache-indexer, or PD-Orchestrator. It disables only the following optional observability workloads:
Before enabling EagleEye, verify that NATS has a ready Service and endpoint:
The EagleEye init containers connect to nats.nats:4222. If this endpoint is unavailable, the enabled EagleEye Pods remain in the init phase.
Enable EagleEye explicitly in an InferNexService or a referenced InferNexServiceConfig:
After applying the resource, verify the optional workloads in the inference service namespace:
Expected workloads include the EagleEye hardware diagnosis Deployment, hardware monitor DaemonSet, and network performance exporter DaemonSet. If EagleEye is not required, keep all three child switches set to false.
Enable InferNex Bridge and NPU Serving Runtime
Alauda AI Platform installs and manages InferNex Bridge through the infernexBridge component of the existing default AmlCluster. For Ascend NPU inference, enable the Alauda AI Platform NPU Serving Runtime through servingRuntimeNPUOperator as well. Merge the required settings into the existing AmlCluster; do not create another instance:
Save the AmlCluster and wait for reconciliation. The NPU Serving Runtime component installs the vLLM-Ascend image and provides the ClusterServingRuntime and LLMInferenceServiceConfig resources required by serving workloads. There is no need to install the InferNex Bridge or NPU Serving Runtime operators manually through OperatorHub.
Verification
Check the AmlCluster status and confirm the managed components and their workloads are ready:
The AmlCluster should report Ready / Reconciled, the InferNex Bridge controller should be running, all five InferNex Bridge CRDs should be installed, and the LLMInferenceService mutating webhook should include v1alpha2. When servingRuntimeNPUOperator is managed, confirm the NPU ClusterServingRuntime and LLMInferenceServiceConfig are present.
Community Examples
For community-maintained examples, see InferNex Bridge v26.6.0 examples.
Upgrading InferNex Bridge
Alauda AI Platform manages the InferNex Bridge lifecycle through the infernexBridge component in the default AmlCluster. Follow your platform's Alauda AI Platform component upgrade procedure; do not upgrade the operator independently through OperatorHub. Before upgrading, back up the existing custom resources:
Verification
After the AmlCluster reconciliation completes, verify the controller and CRD status:
Rollback
Use the platform's supported component rollback procedure and confirm that the target version's CRDs are compatible with the custom resources already stored in the cluster.
After rollback, verify the controller, CRDs, inference services, runtime image IDs, and one representative inference request. Do not delete the five bundled CRDs while custom resources still exist; deleting them also deletes the stored custom resources.