#Migrating from Alauda Build of OpenTelemetry to Alauda Build of OpenTelemetry v2

#Delete the v1 Instrumentation resources

1. List the existing Instrumentation resources:

   kubectl get instrumentation -A

2. Delete each Instrumentation resource. Replace <namespace> and <name> with the values from the previous step:

   kubectl -n <namespace> delete instrumentation <name>
   TIP

   Deleting Instrumentation does not affect application pods that have already been mutated by the webhook — the previously injected init container, environment variables, and JAVA_TOOL_OPTIONS remain in the running pods. The deletion only prevents the v1 Operator from injecting them into newly created pods.

#Delete the v1 OpenTelemetryCollector resources

1. List the existing OpenTelemetryCollector resources:

   kubectl get opentelemetrycollector -A

2. Delete each OpenTelemetryCollector resource:

   kubectl -n <namespace> delete opentelemetrycollector <name>

#Uninstall the v1 Operator

1. Delete the Subscription:

   kubectl delete subscription opentelemetry-operator -n opentelemetry-operator

2. Delete the RBAC and monitoring resources that the v1 deployment created in cpaas-system (skip this step if your v1 deployment did not create them). These resources are backed up in Back up v1 resources for rollback.

   kubectl -n cpaas-system delete servicemonitor otel-collector-monitoring otel-collector
   kubectl -n cpaas-system delete sa otel-collector
   kubectl delete clusterrolebinding otel-collector:cpaas-system:cluster-admin

3. Wait until no v1 Operator CSV remains in the cluster. This check is important — if any v1 CSV is still present, OLM rejects the v2 Operator installation in Step 4 due to a CRD owner conflict.

   kubectl get csv -A | grep '^opentelemetry-operator '

   The expected output is empty.

#Install the v2 Operator

Follow Installing the Alauda Build of OpenTelemetry v2 Operator. The condensed CLI flow is:

1. Confirm the available v2 Operator versions:

   kubectl get packagemanifest opentelemetry-operator2 -o json | jq -r '
     .status.channels[]
     | .name as $channel
     | .entries[]
     | [$channel, .name, .version] | @tsv
   ' | column -t

2. Create the Operator namespace:

   kubectl get namespace opentelemetry-operator2 || \
     kubectl create namespace opentelemetry-operator2

3. Create the Subscription. Replace startingCSV with the version returned in step 1.

   kubectl apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     annotations:
       cpaas.io/target-namespaces: ""
     labels:
       catalog: platform
     name: opentelemetry-operator2
     namespace: opentelemetry-operator2
   spec:
     channel: stable
     installPlanApproval: Manual
     name: opentelemetry-operator2
     source: platform
     sourceNamespace: cpaas-system
     # startingCSV example: opentelemetry-operator2.v0.147.0-r0
     startingCSV: {step-1-operator-csv-version}
   EOF

4. Approve the InstallPlan:

   kubectl -n opentelemetry-operator2 wait \
     --for=condition=InstallPlanPending subscription opentelemetry-operator2 --timeout=2m
   
   PLAN="$(kubectl -n opentelemetry-operator2 get subscription opentelemetry-operator2 \
     -o jsonpath='{.status.installPlanRef.name}')"
   kubectl -n opentelemetry-operator2 patch installplan "$PLAN" --type=json \
     -p='[{"op": "replace", "path": "/spec/approved", "value": true}]'

5. Wait for the v2 CSV to reach Succeeded:

   kubectl wait --for=jsonpath='{.status.phase}'=Succeeded csv \
     --all -n opentelemetry-operator2 --timeout=3m

#Recreate the OpenTelemetryCollector resources

#What this migration changes

When you rebuild the OpenTelemetryCollector manifests from your v1 backup, the following aspects must be adjusted before they can be applied on v2.

• Collector namespace. The Collector namespace must be different from the Operator namespace (opentelemetry-operator2). Choose the namespace based on your deployment scenario:

  • Standalone Collector: a dedicated namespace such as opentelemetry-collector.
  • Alauda Container Platform Tracing integration: keep using the same Collector namespace (typically cpaas-system) so that downstream services that reference the Collector service do not need to change.
  • Alauda Service Mesh v2 integration: keep the Collector in istio-system so the existing Istio meshConfig.extensionProviders[].opentelemetry.service remains valid.

• Component compatibility. Every component used in spec.config must be supported on v2. For the recommended Collector configuration when integrating with Alauda Distributed Tracing, see Deploying the OpenTelemetry Collector in the Alauda Distributed Tracing documentation. For other scenarios, follow Deploying the OpenTelemetry Collector and adapt the example to your environment.

• Feature gates. v1 Collectors often pass Collector feature gates via spec.args.feature-gates. Many of those gates were either stabilized (and therefore no longer toggleable) or removed entirely in newer Collector versions, so reusing the v1 list can prevent the v2 Collector pod from starting. Strip spec.args.feature-gates from the backup and reintroduce only the gates that the v2 Collector version in use explicitly documents.

• Internal metrics Prometheus endpoint. The service.telemetry.metrics.address field is no longer the supported way to expose the internal metrics Prometheus endpoint. Configure it under service.telemetry.metrics.readers[].pull.exporter.prometheus instead, as described in the OpenTelemetry Collector internal telemetry documentation. A typical v1 backup looks like:

  service:
    telemetry:
      logs:
        level: info
      metrics:
        address: 0.0.0.0:8888
        level: detailed

• Internal metrics verbosity. level: detailed enables histogram buckets and per-instance labels for the Collector's own internal metrics, which significantly inflates Prometheus cardinality and storage cost — especially in Gateway-mode deployments with many receiver/exporter instances. The default level: normal is recommended for production: it still exposes process resource usage and per-component sent/received/refused/dropped counters, which is sufficient for most SRE alerting and capacity needs. Switch back to detailed only temporarily when investigating exporter latency distributions or batch sizing.

• Server-managed metadata. Fields written by the API server (metadata.creationTimestamp, metadata.resourceVersion, metadata.uid, metadata.generation, metadata.managedFields, metadata.finalizers, the kubectl.kubernetes.io/last-applied-configuration annotation, and status) cannot be reused on create and must be stripped from the backup.

• Operator-managed RBAC and Prometheus scraping. The v2 Operator automatically creates the ServiceAccount and ClusterRoleBinding resources required by the Collector. Drop the v1 spec.serviceAccount field from the backup so the Operator can provision a fresh ServiceAccount with the correct permissions; you generally do not have to recreate the v1 RBAC resources by hand. To have the Operator also create a ServiceMonitor for the internal Prometheus endpoint, set spec.observability.metrics.enableMetrics: true and add a discovery label (such as prometheus: kube-prometheus) to metadata.labels so that your Prometheus Operator instance picks the resource up. If a Collector component requires additional cluster-level RBAC (for example, the k8sattributes processor or the k8sobjects receiver), follow Creating the Required RBAC Resources Automatically.

#Migration procedure

1. Recreate the OpenTelemetryCollector resource from the backup. The following example copies ./otel-v1-backup/collectors.yaml into a new working directory, strips server-managed metadata, removes the v1 spec.serviceAccount and spec.args.feature-gates fields, downgrades level: detailed to the default level: normal, replaces the deprecated address field with the new readers configuration, enables Operator-managed metrics scraping by setting spec.observability.metrics.enableMetrics: true and adding the prometheus: kube-prometheus label so that the kube-prometheus stack picks up the auto-created ServiceMonitor, and applies the result.

   jq does not read YAML directly, so the example uses kubectl patch --local -p='[]' -o json only as a local YAML-to-JSON decoder before passing the resources to jq.

   RESTORE_DIR=./otel-v2-restore
   
   mkdir -p "$RESTORE_DIR"
   cp ./otel-v1-backup/collectors.yaml "$RESTORE_DIR/collectors.yaml"
   
   kubectl patch --local -f "$RESTORE_DIR/collectors.yaml" --type=json -p='[]' -o json \
     | jq -s '
         {
           apiVersion: "v1",
           kind: "List",
           items: map(
             del(
               .metadata.annotations."kubectl.kubernetes.io/last-applied-configuration",
               .metadata.creationTimestamp,
               .metadata.finalizers,
               .metadata.generation,
               .metadata.managedFields,
               .metadata.resourceVersion,
               .metadata.uid,
               .status,
               .spec.serviceAccount,
               .spec.args."feature-gates"
             )
             | .spec.config.service.telemetry.metrics = (
                 (.spec.config.service.telemetry.metrics // {})
                 | del(.address)
                 | if .level == "detailed" then .level = "normal" else . end
                 | .readers = [
                     {
                       pull: {
                         exporter: {
                           prometheus: {
                             host: "0.0.0.0",
                             port: 8888,
                             without_scope_info: true,
                             without_type_suffix: true,
                             without_units: true
                           }
                         }
                       }
                     }
                   ]
               )
             | .spec.observability.metrics.enableMetrics = true
             | .metadata.labels.prometheus = "kube-prometheus"
           )
         }
       ' > "$RESTORE_DIR/collectors.json"
   
   kubectl apply -f "$RESTORE_DIR/collectors.json"

2. Wait for the Collector pods to become ready:

   kubectl wait --for=condition=Ready pod \
     -l app.kubernetes.io/managed-by=opentelemetry-operator \
     -n <collector-namespace> --timeout=3m

#Recreate the Instrumentation resources

For each Instrumentation resource that you backed up, recreate it on v2 with the new spec.java.image field set. The exporter endpoint and other environment variables follow the same shape used in v1, but Java auto-instrumentation now uses the autoinstrumentation-java 2.x image. In this version, the default OTLP exporter protocol is http/protobuf, so endpoints that previously pointed to the Collector gRPC port 4317 must be changed to the Collector HTTP port 4318 unless you explicitly configure OTEL_EXPORTER_OTLP_PROTOCOL=grpc. Update the host value as well if the Collector namespace or service name has changed.

Use the same working directory created in the previous step. The following example copies ./otel-v1-backup/instrumentations.yaml, sets spec.java.image from the JAVA_AUTO_INSTRUMENTATION_IMAGE variable, changes the backed-up OTEL_EXPORTER_OTLP_ENDPOINT value from port 4317 to 4318, and creates the Instrumentation resources:

RESTORE_DIR=./otel-v2-restore
JAVA_AUTO_INSTRUMENTATION_IMAGE="ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.26.1"

mkdir -p "$RESTORE_DIR"
cp ./otel-v1-backup/instrumentations.yaml "$RESTORE_DIR/instrumentations.yaml"

kubectl patch --local -f "$RESTORE_DIR/instrumentations.yaml" --type=json -p='[]' -o json \
  | jq -s --arg javaAutoInstrumentationImage "$JAVA_AUTO_INSTRUMENTATION_IMAGE" '
      {
        apiVersion: "v1",
        kind: "List",
        items: map(
          del(
            .metadata.annotations."kubectl.kubernetes.io/last-applied-configuration",
            .metadata.creationTimestamp,
            .metadata.finalizers,
            .metadata.generation,
            .metadata.managedFields,
            .metadata.resourceVersion,
            .metadata.uid,
            .status
          )
          | .spec.java.image = $javaAutoInstrumentationImage
          | (.spec.env[]? | select(.name == "OTEL_EXPORTER_OTLP_ENDPOINT").value) |= sub(":4317"; ":4318")
          | (.spec.java.env[]? | select(.name == "OTEL_EXPORTER_OTLP_ENDPOINT").value) |= sub(":4317"; ":4318")
        )
      }
    ' > "$RESTORE_DIR/instrumentations.json"

kubectl apply -f "$RESTORE_DIR/instrumentations.json"

#Roll out the application pods

Application pods that were previously instrumented by the v1 Operator still carry the v1 init container, agent path, and JAVA_TOOL_OPTIONS. Because the Collector backing those pods has been replaced, telemetry export from those pods is no longer functional. Roll out the affected workloads so that the v2 mutating webhook injects the new Java agent image and environment variables.

1. List the deployments that opt in to Java auto-instrumentation:

   kubectl get deploy -A -o json | jq -r '
     .items[]
     | select(.spec.template.metadata.annotations["instrumentation.opentelemetry.io/inject-java"])
     | "\(.metadata.namespace) \(.metadata.name)"'

2. Restart each instrumented deployment and wait for the rollout to complete. Pick one of the two approaches below based on how cautiously you need to validate.

   Option A — One deployment at a time. Run the rollout against each Deployment individually. For large fleets, restart deployments in waves ordered by criticality so you can pause and validate after each wave.

   kubectl -n <namespace> rollout restart deployment/<name>
   kubectl -n <namespace> rollout status deployment/<name>

   Option B — All instrumented deployments in one command. Iterate over every Deployment that opts in to Java auto-instrumentation. This is faster but offers no built-in pause point, so prefer it for small fleets or after you have validated the change on a canary.

   kubectl get deploy -A -o json | jq -r '
     .items[]
     | select(.spec.template.metadata.annotations["instrumentation.opentelemetry.io/inject-java"])
     | [.metadata.namespace, .metadata.name] | @tsv
   ' | while IFS=$'\t' read -r namespace name; do
     kubectl -n "$namespace" rollout restart deployment/"$name"
     kubectl -n "$namespace" rollout status deployment/"$name"
   done

#Verify the migration

1. Verify that only the v2 Operator CSV exists and has reached the Succeeded phase:

   kubectl get csv -A | grep -i opentelemetry

2. Verify the v2 Operator workloads are running:

   kubectl -n opentelemetry-operator2 get csv,deploy

3. Verify the OpenTelemetryCollector resources are healthy and report the v2 version:

   kubectl get opentelemetrycollector -A

4. Verify the Instrumentation resources have spec.java.image configured:

   kubectl get instrumentation -A \
     -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.spec.java.image}{"\n"}{end}'

5. Verify that an instrumented application pod uses the new Java agent init container:

   kubectl -n <namespace> get pod <pod-name> \
     -o jsonpath='{.spec.initContainers[*].image}'

   The output must include the image configured in Instrumentation.spec.java.image.

6. Verify that OpenTelemetry environment variables are present in the pod spec. This checks the Kubernetes object directly and does not require the application container to support kubectl exec or include the env command.

   kubectl -n <namespace> get pod <pod-name> -o json | jq -r '
     .spec.containers[]
     | .name as $container
     | (.env // [])
     | map(select(.name == "JAVA_TOOL_OPTIONS" or (.name | startswith("OTEL_"))))
     | select(length > 0)
     | "container=" + $container,
       (.[] | "\(.name)=\(.value // (.valueFrom | tostring))")
   '

7. Send a test request to an instrumented application and confirm that the resulting traces and metrics appear in your tracing backend (for example, Jaeger UI or the platform Tracing console) and Prometheus.

#Delete the v2 resources

kubectl delete instrumentation -A --all
kubectl delete opentelemetrycollector -A --all
kubectl delete subscription opentelemetry-operator2 -n opentelemetry-operator2
kubectl delete csv -n opentelemetry-operator2 --all

#Wait for the v2 Operator to be fully removed

kubectl get csv -A | grep -i opentelemetry-operator2

The expected output is empty.

#Reinstall the v1 Operator

Follow the v1 installation procedure documented in Installing the OpenTelemetry Operator.

#Recreate the v1 resources from the backup

Recreate the OpenTelemetryCollector, Instrumentation, ServiceAccount, ClusterRoleBinding, and ServiceMonitor resources from the YAML files captured in Back up v1 resources.

#Roll out the application pods again

Restart the workloads with kubectl rollout restart so that the v1 mutating webhook re-injects the v1 Java agent.