Configure OpenTelemetry (OTel)

OpenTelemetry (OTel) is an open-source project designed to provide a vendor-neutral standard for distributed systems (such as microservice architectures) to collect, process, and export telemetry data. It supports developers in analyzing software performance and behavior more easily, making it easier to diagnose and troubleshoot application issues.

Terminology

Prerequisites

• Ensure an operational ALB exists: Create or use an existing ALB, with its name replaced by <otel-alb> in the document. For details on creating an ALB, refer to Create Load Balancer .

• Ensure the existence of a server address for OTel data reporting: This address will hereafter be referred to as <jaeger-server>.

Steps

Update ALB Configuration

1. On the Master node of the cluster, use the CLI tool to execute the following command to edit the ALB configuration.

   kubectl edit alb2 -n cpaas-system <otel-alb> # Replace <otel-alb> with the ALB name

2. Add the following fields under the spec.config field.

   otel:
     enable: true
     exporter:
       collector:
         address: "<jaeger-server>" # Replace <jaeger-server> with the server address for OTel data reporting
         request_timeout: 1000

   Sample configuration:

   spec:
     address: 192.168.1.1
     config:
       otel:
        enable: true
        exporter:
          collector:
            address: "http://jaeger.default.svc.cluster.local:4318"
            request_timeout: 1000
       antiAffinityKey: system
       defaultSSLCert: cpaas-system/cpaas-system
       defaultSSLStrategy: Both
       gateway:
       ...
   type: nginx

3. Execute the following command to save the update. Once updated, the ALB will enable OpenTelemetry by default and report all request Trace information to the Jaeger Server.

   :wq

Related Operations

Configure OTel in Ingress

• Enable or Disable OTel on Ingress

  By configuring whether to enable OTel on Ingress, you can better monitor and debug the request flow of applications. This helps trace the propagation path of requests between different services, identifying performance bottlenecks or errors.

  Steps

  Add the following configuration under the metadata.annotations field of the Ingress:

  nginx.ingress.kubernetes.io/enable-opentelemetry: "true"

  Parameter Explanation:

  • nginx.ingress.kubernetes.io/enable-opentelemetry: When set to true, it indicates that the Ingress controller enables OpenTelemetry when processing requests through this Ingress, meaning the Trace information of the requests will be collected and reported. When set to false or this annotation is removed, it indicates that the Trace information of the requests will not be collected or reported.

• Enable or Disable OTel Trust on Ingress

  The purpose of OTel Trust is to determine whether the Ingress trusts and uses the OTel Trace information (such as trace ID) in incoming requests.

  Steps

  Add the following configuration under the metadata.annotations field of the Ingress:

  nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true"

  Parameter Explanation:

  • nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: When set to true, the Ingress will continue to use the existing Trace information, helping to maintain cross-service tracing consistency, allowing the entire request chain to be completely traced and analyzed in a distributed tracing system. When set to false, new tracing information will be generated for the request, which may cause the request to be treated as part of a new tracing chain upon entering the Ingress, interrupting cross-service tracing continuity.

• Add Different OTel Configurations on Ingress

  This configuration allows you to customize the behavior and data export method of OTel for different Ingress resources, enabling fine-grained control for each service and helping to configure different tracing strategies or target services.

  Steps

  Add the following configuration under the metadata.annotations field of the Ingress:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      alb.ingress.cpaas.io/otel: >
       {
          "enable": true,
          "exporter": {
              "collector": {
                  "address": "<jaeger-server>", # Replace <jaeger-server> with the server address for OTel data reporting, e.g., "address": "http://128.0.0.1:4318"
                  "request_timeout": 1000
              }
          }
       }     

  Parameter Explanation:

  • exporter: Specifies how to send the collected Trace data to the OTel Collector (OTel data reporting server).
  • address: Specifies the address of the OTel Collector.
  • request_timeout: Specifies the request timeout duration.

Use OTel in Applications

The following configuration shows the complete OTel configuration structure, which can be used to define how to enable and use OTel features in applications.

You can execute the following command on the cluster’s Master node using the CLI tool to obtain the complete configuration structure for OTel.

kubectl get crd alaudaloadbalancer2.crd.alauda.io -o json|jq ".spec.versions[2].schema.openAPIV3Schema.properties.spec.properties.config.properties.otel"

Output result:

{
    "otel": {
        "enable": true
    }
    "exporter": {
        "collector": {
            "address": ""
          },
    },
    "flags": { 
        "hide_upstream_attrs": false
        "notrust_incoming_span": false
        "report_http_request_header": false
        "report_http_response_header": false
    },
    "sampler": {
        "name": "", 
        "options": {
            "fraction": ""
            "parent_name": ""
          },
      },
 }

Parameter Explanation:

Inheritance

By default, if certain OTel parameters are configured on the ALB and not on the FT, the FT will use the ALB’s parameters as its own configuration, meaning the FT inherits the ALB’s configuration. Similarly, Rules can inherit configurations from both ALB and FT.

• ALB: Configurations on the ALB are usually global and default settings. You can configure global parameters here, such as the Collector address, which will be inherited by the underlying FT and Rule.

• FT: FT can inherit the configuration from ALB. If certain OTel parameters are not configured on the FT, it will use the ALB’s configuration. However, FT can also be further customized. For example, you can selectively enable or disable OTel on FT without affecting other FTs or the global settings of ALB.

• Rule: Rules can inherit configurations from both ALB and FT. However, they can also be further customized. For instance, a specific Rule can choose not to trust incoming OTel Trace information or adjust the sampling strategy, etc.

Operational Steps

You can add relevant OTel configurations by configuring the spec.config.otel field in the YAML files of ALB, FT, and Rule.

Related Information

Sampling Strategy

Attributes

• Resource Attributes

  These attributes are reported by default.

  Parameter	Explanation
  hostname	Hostname of the ALB Pod
  service.name	Name of the ALB
  service.namespace	Namespace where ALB resides
  service.type	Default is ALB
  service.instance.id	Name of the ALB Pod

• Span Attributes

  • Attributes reported by default

    Parameter	Explanation
    http.status_code	Status code
    http.request.resend_count	Retry count
    alb.rule.rule_name	Name of the rule matched by this request
    alb.rule.source_type	Type of the rule matched by this request, currently only Ingress
    alb.rule.source_name	Name of the Ingress
    alb.rule.source_ns	Namespace where Ingress resides

  • Attributes reported by default but can be disabled by modifying the flag.hide_upstream_attrs field

    Parameter	Explanation
    alb.upstream.svc_name	Name of the forwarded Service (internal routing)
    alb.upstream.svc_ns	Namespace of the forwarded Service (internal routing)
    alb.upstream.peer	IP address and port of the forwarded Pod

  • Attributes not reported by default but can be enabled by modifying the flag.report_http_request_header field

    Parameter	Explanation
    http.request.header.	Request Header

  • Attributes not reported by default but can be enabled by modifying the flag.report_http_response_header field

    Parameter	Explanation
    http.response.header.	Response Header

Configuration Example

In the YAML configuration below, an ALB is deployed using Jaeger as the OTel server and Hotrod-proxy as the demo backend. By configuring Ingress rules, when a client requests the ALB, the traffic will be forwarded to HotROD. Meanwhile, communication between microservices within HotROD is also forwarded through the ALB.

1. Save the following YAML to a file named all.yaml.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hotrod
   spec:
     replicas: 1
     selector:
       matchLabels:
         service.cpaas.io/name: hotrod
         service_name: hotrod
     template:
       metadata:
         labels:
           service.cpaas.io/name: hotrod
           service_name: hotrod
       spec:
         containers:
           - name: hotrod
             env:
               - name: PROXY_PORT
                 value: "80"
               - name: PROXY_ADDR
                 value: "otel-alb.default.svc.cluster.local:"
               - name: OTEL_EXPORTER_OTLP_ENDPOINT
                 value: "http://jaeger.default.svc.cluster.local:4318"
             image: theseedoaa/hotrod-with-proxy:latest
             imagePullPolicy: IfNotPresent
             command: ["/bin/hotrod","all","-v"]
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hotrod-frontend
   spec:
     ingressClassName: otel-alb
     rules:
     - http:
         paths:
         - backend:
             service:
               name: hotrod
               port:
                 number: 8080
           path: /dispatch
           pathType: ImplementationSpecific
         - backend:
             service:
               name: hotrod
               port:
                 number: 8080
           path: /frontend
           pathType: ImplementationSpecific
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hotrod-customer
   spec:
     ingressClassName: otel-alb
     rules:
     - http:
         paths:
         - backend:
             service:
               name: hotrod
               port:
                 number: 8081
           path: /customer
           pathType: ImplementationSpecific
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hotrod-route
   spec:
     ingressClassName: otel-alb
     rules:
     - http:
         paths:
         - backend:
             service:
               name: hotrod
               port:
                 number: 8083
           path: /route
           pathType: ImplementationSpecific
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: hotrod
   spec:
     internalTrafficPolicy: Cluster
     ipFamilies:
       - IPv4
     ipFamilyPolicy: SingleStack
     ports:
       - name: frontend
         port: 8080
         protocol: TCP
         targetPort: 8080
       - name: customer
         port: 8081
         protocol: TCP
         targetPort: 8081
       - name: router
         port: 8083
         protocol: TCP
         targetPort: 8083
     selector:
       service_name: hotrod
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: jaeger
   spec:
     replicas: 1
     selector:
       matchLabels:
         service.cpaas.io/name: jaeger
         service_name: jaeger
     template:
       metadata:
         labels:
           service.cpaas.io/name: jaeger
           service_name: jaeger
       spec:
         containers:
           - name: jaeger
             env:
              - name: LOG_LEVEL
                value: debug
             image: jaegertracing/all-in-one:1.58.1
             imagePullPolicy: IfNotPresent
         hostNetwork: true
         tolerations:
           - operator: Exists
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger
   spec:
     internalTrafficPolicy: Cluster
     ipFamilies:
       - IPv4
     ipFamilyPolicy: SingleStack
     ports:
       - name: http
         port: 4318
         protocol: TCP
         targetPort: 4318
     selector:
       service_name: jaeger
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: crd.alauda.io/v2
   kind: ALB2
   metadata:
     name: otel-alb
   spec:
     config:
       loadbalancerName: otel-alb
       otel:
         enable: true
         exporter:
           collector:
             address: "http://jaeger.default.svc.cluster.local:4318"
             request_timeout: 1000
       projects:
       - ALL_ALL
       replicas: 1
       resources:
         alb:
           limits:
             cpu: 200m
             memory: 2Gi
           requests:
             cpu: 50m
             memory: 128Mi
         limits:
           cpu: "1"
           memory: 1Gi
         requests:
           cpu: 50m
           memory: 128Mi
     type: nginx

2. Execute the following command in the CLI tool to deploy Jaeger, ALB, HotROD, and all CRs needed for testing.

   kubectl apply -f ./all.yaml

3. Execute the following command to get the access address for Jaeger.

   export JAEGER_IP=$(kubectl get po -A -o wide | grep jaeger | awk '{print $7}'); echo "http://$JAEGER_IP:16686"

4. Execute the following command to get the access address for otel-alb.

   export ALB_IP=$(kubectl get po -A -o wide | grep otel-alb | awk '{print $7}'); echo $ALB_IP

5. Execute the following command to send a request to HotROD through the ALB. The ALB will report the Trace to Jaeger.

   curl -v "http://<$ALB_IP>:80/dispatch?customer=567&nonse=" # Replace the <$ALB_IP> part in the command with the otel-alb access address obtained in the previous step

6. Open the Jaeger access address obtained in Step 3 to view the results.