Configuring distributed tracing platform with Service Mesh (Deprecated)

Deprecation Notice

The Alauda build of Jaeger (Jaeger 1.60.0) and Alauda Build of OpenTelemetry integration described in this document is deprecated and will be removed in ACP 4.4. Use the new Alauda Distributed Tracing integration documented in Configuring distributed tracing platform with Service Mesh for all new deployments.

To migrate an existing installation to the new stack, follow Migrating to Jaeger v2.

Alauda Service Mesh supports distributed tracing through integration with the following components:

  • Alauda Build of Jaeger: A customized distribution based on the open source Jaeger project. It provides end-to-end visibility into requests across complex distributed systems.

  • Alauda Build of OpenTelemetry: Based on the OpenTelemetry project, this component simplifies telemetry data collection across metrics, logs, and traces by managing the OpenTelemetry Collector and workload instrumentation.

The OpenTelemetry Collector acts as an intermediary for telemetry signals. It supports multiple data formats and provides a standardized pipeline for processing and exporting telemetry to backends such as Jaeger.

Configuring distributed tracing data collection with Service Mesh

You can integrate Alauda Service Mesh with OpenTelemetry to instrument, generate, collect, and export OpenTelemetry traces, metrics, and logs to analyze and understand your software's performance and behavior.

Prerequisites

Procedure

Label the istio-system namespace with the project label:

kubectl label namespace istio-system cpaas.io/project=cpaas-system --overwrite

Install a Jaeger instance in the istio-system namespace.

Refer to the installation script in Deploying a Jaeger Instance and use the example command below to deploy an Istio-dedicated Jaeger:

INFO

The --jaeger-es-index-prefix parameter sets the index prefix in Elasticsearch where trace data is stored.

  • For a single-cluster service mesh, we recommend ending the prefix with the cluster name, for example istio-tracing-cluster-1.
  • For a multi-cluster service mesh, traces from all clusters must be stored in the same index; we recommend ending the prefix with the meshID, for example istio-tracing-mesh-1.
./install-jaeger.sh \
  --es-url='https://xxx' \
  --es-user-base64='xxx' \
  --es-pass-base64='xxx' \
  --target-namespace='istio-system' \
  --jaeger-basepath-suffix='/istio/jaeger' \
  --jaeger-es-index-prefix='istio-tracing-xxx'

After the installation completes successfully, you can access the Jaeger UI to query traces at <platform-url>/clusters/<cluster>/istio/jaeger.

Example OpenTelemetry Collector in istio-system namespace

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: istio-system
spec:
  observability:
    metrics: {}
  deploymentUpdateStrategy: {}
  config:
    processors:
      batch: {}
    exporters:
      debug: {}
      otlp:
        endpoint: 'dns:///jaeger-prod-collector-headless.istio-system:4317'
        balancer_name: round_robin
        tls:
          insecure: true
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: '0.0.0.0:4317'
    service:
      pipelines:
        traces:
          exporters:
            - debug
            - otlp
          processors:
            - batch
          receivers:
            - otlp
  1. The endpoint field is the Jaeger collector service in the istio-system namespace.

Update the Istio resource to enable tracing and define the OpenTelemetry tracing provider:

Example: Enabling tracing via meshConfig

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
  # ...
spec:
  namespace: istio-system
  # ...
  values:
    meshConfig:
      enableTracing: true
      extensionProviders:
      - name: otel
        opentelemetry:
          port: 4317
          service: otel-collector.istio-system.svc.cluster.local
  1. The service field is the OpenTelemetry collector service in the istio-system namespace.

Update the Telemetry resource to enable the tracing provider defined in the meshConfig:

Example Istio Telemetry resource

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: asm-default
  namespace: istio-system
  # ...
spec:
  # ...
  tracing:
    - providers:
        - name: otel
      randomSamplingPercentage: 100
NOTE

Once you verify that you can see traces, lower the randomSamplingPercentage value to reduce the number of requests.

Uninstalling distributed tracing

If you no longer need the Alauda Build of OpenTelemetry and Alauda build of Jaeger integration with Service Mesh, follow the steps below to remove them.

WARNING

After uninstalling, historical traces stored in Elasticsearch are no longer reachable from the Jaeger UI.

Removing the Service Mesh tracing configuration

Before deleting the underlying components, remove the mesh-side tracing configuration so that Istio stops sending spans to the legacy collector.

  1. Edit the Telemetry resource and remove the tracing providers entry that references the otel provider:

    kubectl -n istio-system edit telemetry asm-default

    Alternatively, remove the tracing configuration non-interactively with kubectl patch:

    kubectl -n istio-system patch telemetry asm-default --type=json -p='[{"op": "remove", "path": "/spec/tracing"}]'
  2. Edit the Istio resource and remove the meshConfig.extensionProviders entry named otel, or set meshConfig.enableTracing to false:

    kubectl edit istio default

    Alternatively, set meshConfig.enableTracing to false non-interactively with kubectl patch:

    kubectl patch istio default --type=merge -p='{"spec":{"values":{"meshConfig":{"enableTracing":false}}}}'

Deleting the OpenTelemetry Collector instance

Log in to the cluster where the integration is installed and delete the OpenTelemetryCollector instance:

kubectl -n istio-system delete opentelemetrycollector otel

Uninstalling the Alauda Build of OpenTelemetry Operator

Skip this step if other workloads in the cluster still rely on the Alauda Build of OpenTelemetry Operator.

You can uninstall the Operator using the Administrator view in the Web Console.

Steps

  • From MarketplaceOperatorHub → use the search box to search for Alauda build of OpenTelemetry.
  • Click on the Alauda build of OpenTelemetry title to enter its details.
  • On the Alauda build of OpenTelemetry details page, click the Uninstall button in the upper right corner.
  • In the Uninstall "opentelemetry-operator"? window, click Uninstall.

Deleting the Jaeger instance

Log in to the cluster where the integration is installed and delete the Jaeger instance and its supporting resources. The resource names below match the defaults produced by install-jaeger.sh with --target-namespace='istio-system'; adjust them if you used a different --jaeger-instance-name during installation.

kubectl -n istio-system delete ingress      jaeger-prod-query         --ignore-not-found
kubectl -n istio-system delete podmonitor   jaeger-prod-monitor       --ignore-not-found
kubectl -n istio-system delete jaeger       jaeger-prod               --ignore-not-found
kubectl -n istio-system delete rolebinding  jaeger-prod-rb            --ignore-not-found
kubectl -n istio-system delete role         jaeger-prod-role          --ignore-not-found
kubectl -n istio-system delete sa           jaeger-prod-sa            --ignore-not-found
kubectl -n istio-system delete secret       jaeger-prod-oauth2-proxy  --ignore-not-found
kubectl -n istio-system delete secret       jaeger-prod-es-basic-auth --ignore-not-found
kubectl -n istio-system delete configmap    jaeger-prod-oauth2-proxy  --ignore-not-found

Uninstalling the Alauda build of Jaeger Operator

Skip this step if other workloads in the cluster still rely on the Alauda build of Jaeger Operator.

You can uninstall the Operator using the Administrator view in the Web Console.

Steps

  • From MarketplaceOperatorHub → use the search box to search for Alauda build of Jaeger.
  • Click on the Alauda build of Jaeger title to enter its details.
  • On the Alauda build of Jaeger details page, click the Uninstall button in the upper right corner.
  • In the Uninstall "jaeger-operator"? window, click Uninstall.