#OTel

OpenTelemetry (OTel) is an open-source project aimed at providing a vendor-neutral standard for collecting, processing, and exporting telemetry data in distributed systems, such as microservices architectures. It helps developers analyze the performance and behavior of software more easily, thus facilitating the diagnosis and resolution of application issues.

#Terminology

#Prerequisites

• Ensure that an operable ALB exists: Create or use an existing ALB, where the name of the ALB is replaced with <otel-alb> in this document. For instructions on creating an ALB, refer to Creating Load Balancer.

• Ensure that there is an OTel data reporting server address: This address will hereinafter be referred to as <jaeger-server>.

#Procedure

#Update ALB Configuration

1. On the Master node of the cluster, use the CLI tool to execute the following command to edit the ALB configuration.

   kubectl edit alb2 -n cpaas-system <otel-alb> # Replace <otel-alb> with the actual ALB name

2. Add the following fields under the spec.config section.

   otel:
     enable: true
     exporter:
       collector:
         address: "<jaeger-server>" # Replace <jaeger-server> with the actual OTel data reporting server address
         request_timeout: 1000

   Example configuration once completed:

   spec:
     address: 192.168.1.1
     config:
       otel:
        enable: true
        exporter:
          collector:
            address: "http://jaeger.default.svc.cluster.local:4318"
            request_timeout: 1000
       antiAffinityKey: system
       defaultSSLCert: cpaas-system/cpaas-system
       defaultSSLStrategy: Both
       gateway:
       ...
   type: nginx

3. Execute the following command to save the updates. After the update, the ALB will default to enabling OpenTelemetry, and all request Trace information will be reported to the Jaeger Server.

   :wq

#Related Operations

#Configuring OTel in Ingress

• Enable or Disable OTel on Ingress

  By configuring whether to enable OTel on Ingress, you can better monitor and debug the request flow of applications, identifying performance bottlenecks or errors by tracing requests as they propagate between different services.

  Procedure

  Add the following configuration under the metadata.annotations field of Ingress:

  nginx.ingress.kubernetes.io/enable-opentelemetry: "true"

  Parameter Explanation:

  • nginx.ingress.kubernetes.io/enable-opentelemetry: When set to true, it indicates that the Ingress controller enables OpenTelemetry functionality while processing requests through this Ingress, which means request Trace information will be collected and reported. When set to false or this annotation is removed, it means that request Trace information will not be collected or reported.

• Enable or Disable OTel Trust on Ingress

  OTel Trust determines whether Ingress trusts and uses the Trace information (e.g., trace ID) from incoming requests.

  Procedure

  Add the following configuration under the metadata.annotations field of Ingress:

  nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true"

  Parameter Explanation:

  • nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: When set to true, the Ingress continues to use already existing Trace information, helping maintain consistency in cross-service tracing, allowing the entire request chain to be fully traced and analyzed in the distributed tracing system. When set to false, it will generate new tracing information for the request, which may cause the request to be treated as part of a new tracing chain after entering the Ingress, interrupting cross-service trace continuity.

• Add Different OTel Configurations on Ingress

  This configuration allows you to customize OTel's behavior and data export methodology for different Ingress resources, enabling fine-grained control over each service's tracing strategy or target.

  Procedure

  Add the following configuration under the metadata.annotations field of Ingress:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      alb.ingress.cpaas.io/otel: >
        {
           "enable": true,
           "exporter": {
               "collector": {
                   "address": "<jaeger-server>", # Replace <jaeger-server> with the actual OTel data reporting server address, e.g., "address": "http://128.0.0.1:4318"
                   "request_timeout": 1000
               }
           }
        }

  Parameter Explanation:

  • exporter: Specifies how the collected Trace data is sent to the OTel Collector (the OTel data reporting server).
  • address: Specifies the address of the OTel Collector.
  • request_timeout: Specifies the request timeout.

#Using OTel in Applications

The following configuration shows the complete OTel configuration structure, which can be used to define how to enable and use OTel features in applications.

On the cluster Master node, use the CLI tool to execute the following command to get the complete OTel configuration structure.

kubectl get crd alaudaloadbalancer2.crd.alauda.io -o json|jq ".spec.versions[2].schema.openAPIV3Schema.properties.spec.properties.config.properties.otel"

Echoed Result:

{
    "otel": {
        "enable": true
    }
    "exporter": {
        "collector": {
            "address": ""
          },
    },
    "flags": {
        "hide_upstream_attrs": false
        "notrust_incoming_span": false
        "report_http_request_header": false
        "report_http_response_header": false
    },
    "sampler": {
        "name": "",
        "options": {
            "fraction": ""
            "parent_name": ""
          },
      },
 }

Parameter Explanation:

#Inheritance

By default, if the ALB configures certain OTel parameters and FT is not configured, FT will inherit the parameters from the ALB as its own configuration; that is, FT inherits the ALB configuration, while Rule can inherit configurations from both ALB and FT.

• ALB: The configuration on the ALB is typically global and default. You can configure global parameters such as Collector addresses here, which will be inherited by the lower-level FT and Rule.

• FT: FT can inherit configurations from ALB, meaning that certain OTel parameters that are not configured on FT will use the configuration from ALB. However, FT can also be refined further; for instance, you can choose to selectively enable or disable OTel on FT without affecting other FT or the global settings of ALB.

• Rule: Rule can inherit configurations from both ALB and FT. However, Rule can also be refined further; for instance, a specific Rule can choose not to trust the incoming OTel Trace information or to adjust the sampling strategies.

Procedure

By configuring the spec.config.otel field in the YAML files of ALB, FT, and Rule, you can add OTel-related configuration.

#Additional Notes

#Sampling Strategies

#Attributes

• Resource Attributes

  These attributes are reported by default.

  Parameter	Description
  hostname	The hostname of the ALB Pod
  service.name	The name of the ALB
  service.namespace	The namespace where the ALB resides
  service.type	Default is ALB
  service.instance.id	The name of the ALB Pod

• Span Attributes

  • Attributes reported by default:

    Parameter	Description
    http.status_code	Status code
    http.request.resend_count	Retry count
    alb.rule.rule_name	The name of the rule matched by this request
    alb.rule.source_type	The type of the rule matched by this request, currently only Ingress
    alb.rule.source_name	The name of the Ingress
    alb.rule.source_ns	The namespace where the Ingress resides

  • Attributes reported by default but can be excluded by modifying the flag.hide_upstream_attrs field:

    Parameter	Description
    alb.upstream.svc_name	The name of the Service (internal route) to which traffic is forwarded
    alb.upstream.svc_ns	The namespace where the Service (internal route) being forwarded resides
    alb.upstream.peer	The IP address and port of the Pod being forwarded to

  • Attributes not reported by default but can be reported by modifying the flag.report_http_request_header field:

    Parameter	Description
    **http.request.header.<header>**	Request Header

  • Attributes not reported by default but can be reported by modifying the flag.report_http_response_header field:

    Parameter	Description
    **http.response.header.<header>**	Response Header

#Configuration Example

The following YAML configuration deploys an ALB and uses Jaeger as the OTel server, with Hotrod-proxy as the demonstration backend. By configuring Ingress rules, when clients request the ALB, the traffic will be forwarded to HotROD. Additionally, the communication between internal microservices of HotROD is also routed through the ALB.

1. Save the following YAML as a file named all.yaml.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hotrod
   spec:
     replicas: 1
     selector:
       matchLabels:
         service.cpaas.io/name: hotrod
         service_name: hotrod
     template:
       metadata:
         labels:
           service.cpaas.io/name: hotrod
           service_name: hotrod
       spec:
         containers:
           - name: hotrod
             env:
               - name: PROXY_PORT
                 value: "80"
               - name: PROXY_ADDR
                 value: "otel-alb.default.svc.cluster.local:"
               - name: OTEL_EXPORTER_OTLP_ENDPOINT
                 value: "http://jaeger.default.svc.cluster.local:4318"
             image: theseedoaa/hotrod-with-proxy:latest
             imagePullPolicy: IfNotPresent
             command: ["/bin/hotrod", "all", "-v"]
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hotrod-frontend
   spec:
     ingressClassName: otel-alb
     rules:
       - http:
           paths:
             - backend:
                 service:
                   name: hotrod
                   port:
                     number: 8080
               path: /dispatch
               pathType: ImplementationSpecific
             - backend:
                 service:
                   name: hotrod
                   port:
                     number: 8080
               path: /frontend
               pathType: ImplementationSpecific
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hotrod-customer
   spec:
     ingressClassName: otel-alb
     rules:
       - http:
           paths:
             - backend:
                 service:
                   name: hotrod
                   port:
                     number: 8081
               path: /customer
               pathType: ImplementationSpecific
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hotrod-route
   spec:
     ingressClassName: otel-alb
     rules:
       - http:
           paths:
             - backend:
                 service:
                   name: hotrod
                   port:
                     number: 8083
               path: /route
               pathType: ImplementationSpecific
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: hotrod
   spec:
     internalTrafficPolicy: Cluster
     ipFamilies:
       - IPv4
     ipFamilyPolicy: SingleStack
     ports:
       - name: frontend
         port: 8080
         protocol: TCP
         targetPort: 8080
       - name: customer
         port: 8081
         protocol: TCP
         targetPort: 8081
       - name: router
         port: 8083
         protocol: TCP
         targetPort: 8083
     selector:
       service_name: hotrod
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: jaeger
   spec:
     replicas: 1
     selector:
       matchLabels:
         service.cpaas.io/name: jaeger
         service_name: jaeger
     template:
       metadata:
         labels:
           service.cpaas.io/name: jaeger
           service_name: jaeger
       spec:
         containers:
           - name: jaeger
             env:
               - name: LOG_LEVEL
                 value: debug
             image: jaegertracing/all-in-one:1.58.1
             imagePullPolicy: IfNotPresent
         hostNetwork: true
         tolerations:
           - operator: Exists
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger
   spec:
     internalTrafficPolicy: Cluster
     ipFamilies:
       - IPv4
     ipFamilyPolicy: SingleStack
     ports:
       - name: http
         port: 4318
         protocol: TCP
         targetPort: 4318
     selector:
       service_name: jaeger
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: crd.alauda.io/v2
   kind: ALB2
   metadata:
     name: otel-alb
   spec:
     config:
       loadbalancerName: otel-alb
       otel:
         enable: true
         exporter:
           collector:
             address: "http://jaeger.default.svc.cluster.local:4318"
             request_timeout: 1000
       projects:
         - ALL_ALL
       replicas: 1
       resources:
         alb:
           limits:
             cpu: 200m
             memory: 2Gi
           requests:
             cpu: 50m
             memory: 128Mi
         limits:
           cpu: "1"
           memory: 1Gi
         requests:
           cpu: 50m
           memory: 128Mi
     type: nginx

2. In the CLI tool, execute the following command to deploy Jaeger, ALB, HotROD, and all necessary CRs for testing.

   kubectl apply ./all.yaml

3. Execute the following command to get the access address of Jaeger.

   export JAEGER_IP=$(kubectl get po -A -o wide |grep jaeger | awk '{print $7}');echo "http://$JAEGER_IP:16686"

4. Execute the following command to obtain the access address of otel-alb.

   export ALB_IP=$(kubectl get po -A -o wide|grep otel-alb | awk '{print $7}');echo $ALB_IP

5. Execute the following command to send a request to HotROD via ALB. Here, ALB will report the Trace to Jaeger.

   curl -v "http://<$ALB_IP>:80/dispatch?customer=567&nonse=" # Replace <$ALB_IP> in the command with the access address of otel-alb obtained in the previous procedure

6. Open the access address of Jaeger obtained in Step 3 to view the results.