Alauda DevOps Pipelines Docs

PipelineRuns

A PipelineRun is a Kubernetes Custom Resource that instantiates a Pipeline for execution. PipelineRuns are responsible for executing the Tasks defined in a Pipeline and managing their lifecycle, providing a way to run end-to-end CI/CD workflows.

TOC

Why PipelineRuns are Needed

CI/CD Workflow Challenges

In CI/CD systems, there are several challenges related to workflow execution:

  • Workflow Orchestration: Need to coordinate multiple tasks in a specific order
  • Resource Coordination: Need to manage resources across multiple tasks
  • Data Sharing: Need to share data between tasks in a workflow
  • Execution Tracking: Need to track the progress of the entire workflow
  • Error Handling: Need to handle failures at both task and workflow levels
  • Auditability: Need to maintain a record of all workflow executions

Tekton's Solution

Tekton PipelineRuns address these challenges by:

  • Workflow Instance: Each PipelineRun represents a single execution of a Pipeline with specific inputs
  • Orchestration: PipelineRuns manage the execution order of Tasks based on dependencies
  • Resource Binding: PipelineRuns bind actual resources to Pipeline requirements
  • Status Tracking: PipelineRuns track the status of each Task's execution
  • Result Propagation: PipelineRuns enable sharing results between Tasks
  • Kubernetes Integration: PipelineRuns leverage Kubernetes for resource management and scheduling

Advantages

  • Isolation: Each PipelineRun executes independently, allowing parallel execution of the same Pipeline
  • Traceability: PipelineRuns provide detailed execution history and logs
  • Flexibility: PipelineRuns can override Pipeline parameters and workspace bindings
  • Reusability: The same Pipeline can be executed with different inputs via different PipelineRuns
  • Integration: PipelineRuns can be triggered by various systems through Kubernetes API
  • Observability: PipelineRun status and logs can be monitored through Kubernetes tools

Scenarios

PipelineRuns are useful in various scenarios, including:

  • Application CI/CD: Building, testing, and deploying applications
  • Infrastructure as Code: Provisioning and managing infrastructure
  • Release Management: Coordinating complex release processes
  • Scheduled Workflows: Running workflows on a schedule
  • Event-Driven Workflows: Triggering workflows in response to external events

Constraints and Limitations

  • PipelineRuns execute within a single Kubernetes cluster
  • Once started, PipelineRun parameters cannot be modified
  • PipelineRuns have limited retry capabilities for failed Tasks
  • PipelineRun execution time is limited by the Kubernetes pod timeouts
  • PipelineRuns cannot be paused once they start executing (except with "finally" tasks)

Principles

PipelineRun Execution Model

When a PipelineRun is created:

  1. Tekton validates the PipelineRun specification
  2. Tekton creates TaskRuns for each Task in the Pipeline
  3. TaskRuns are executed based on their dependencies
  4. Results from Tasks are collected and can be used by subsequent Tasks
  5. The PipelineRun status is updated as each TaskRun completes
  6. "Finally" Tasks are executed after all other Tasks complete or fail
  7. The PipelineRun completes when all TaskRuns complete

PipelineRun Status

The PipelineRun status provides detailed information about the execution:

  1. Overall status (Running, Succeeded, Failed, etc.)
  2. Start and completion times
  3. Individual TaskRun statuses
  4. Task results
  5. Reason for failure (if applicable)
  6. TaskRun names for accessing logs

Configuration Examples

Basic PipelineRun Example

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-app-run
spec:
  pipelineRef:
    name: build-deploy-app
  params:
    - name: image-name
      value: my-registry/my-app:latest
    - name: deployment-name
      value: my-app
  workspaces:
    - name: source
      persistentVolumeClaim:
        claimName: app-source-pvc

PipelineRun with Embedded Pipeline Definition

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: simple-pipeline-run
spec:
  pipelineSpec:
    params:
      - name: message
        type: string
    tasks:
      - name: print-message
        taskSpec:
          params:
            - name: message
              type: string
          steps:
            - name: print
              image: ubuntu
              script: |
                echo "$(params.message)"
        params:
          - name: message
            value: "$(params.message)"
  params:
    - name: message
      value: "Hello, Tekton Pipeline!"

Important Parameters

Timeout

Timeout allows setting a maximum duration for PipelineRun execution.

Use Cases

  • Preventing long-running Pipelines from consuming resources indefinitely
  • Ensuring CI/CD workflows complete within a reasonable timeframe
  • Handling hung or deadlocked Tasks
  • Setting different timeout limits for different phases of pipeline execution

Principles

Timeouts are:

  • Specified in the PipelineRun specification using the timeouts field (the legacy timeout field is deprecated)
  • Applied to the entire PipelineRun execution or specific sections (tasks, finally)
  • Measured from the start of the first Task to the completion of the last Task
  • Result in PipelineRun failure if exceeded
  • Must follow the constraint: timeouts.pipeline >= timeouts.tasks + timeouts.finally
  • Important: If any sub-field is set to "0", there is no timeout for that section. To set timeouts.tasks or timeouts.finally to "0", you must also set timeouts.pipeline to "0"

Timeout Fields

  • pipeline: Maximum duration for the entire PipelineRun
  • tasks: Maximum duration for all non-finally tasks
  • finally: Maximum duration for all finally tasks

Configuration Examples

Example 1: Standard timeout configuration

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-with-timeout
spec:
  pipelineRef:
    name: build-deploy-app
  timeouts:
    pipeline: "1h"    # Total pipeline timeout
    tasks: "45m"      # Timeout for all regular tasks
    finally: "15m"    # Timeout for finally tasks
  params:
    - name: image-name
      value: my-registry/my-app:latest

Example 2: No timeout for tasks and finally (requires pipeline timeout to be 0)

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-no-timeout
spec:
  pipelineRef:
    name: build-deploy-app
  timeouts:
    pipeline: "0"     # No overall timeout
    tasks: "0"        # No timeout for regular tasks
    finally: "0"      # No timeout for finally tasks
  params:
    - name: image-name
      value: my-registry/my-app:latest

Note:

  • Example 1 follows the constraint where pipeline (1h) >= tasks (45m) + finally (15m)
  • Example 2 shows how to disable timeouts by setting all values to "0"

ServiceAccount

ServiceAccount allows specifying the Kubernetes ServiceAccount to use for authentication.

Use Cases

  • Providing credentials for accessing private container registries
  • Authenticating with cloud providers
  • Authorizing access to Kubernetes resources
  • Providing credentials for Git operations

Principles

ServiceAccounts:

  • Can be specified at the PipelineRun level for all Tasks
  • Can be mapped to specific Tasks using taskRunSpecs
  • Provide credentials through Kubernetes secrets
  • Enable secure access to external resources

Configuration Example

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-with-service-account
spec:
  pipelineRef:
    name: build-deploy-app
  taskRunTemplate:
    serviceAccountName: pipeline-service-account
  params:
    - name: image-name
      value: my-registry/my-app:latest

Pod Template

Pod Template allows specifying a Pod template to use as the basis for the configuration of the Pod that executes each Task.

TIP

For more information about Pod Templates, please refer to Pod Templates.

Use Cases

  • Setting node selectors for specific hardware requirements
  • Configuring tolerations for dedicated nodes
  • Specifying security contexts for enhanced security
  • Setting resource limits and requests at the Pod level
  • Configuring affinity rules for better resource utilization
  • Adding custom annotations or labels to Pods

Principles

Pod Templates:

  • Are applied to all Tasks in the PipelineRun unless overridden by taskRunSpecs
  • Follow Kubernetes Pod template specifications
  • Can be overridden by Task-specific configurations in taskRunSpecs
  • Enable consistent Pod configuration across all Tasks in a PipelineRun
  • Support all standard Kubernetes Pod template fields

Configuration Example

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-with-pod-template
spec:
  pipelineRef:
    name: build-deploy-app
  taskRunTemplate:
    serviceAccountName: default
    # Global Pod template for all Tasks in the PipelineRun
    podTemplate:
      nodeSelector:
        disktype: ssd
        environment: production
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "pipeline"
        effect: "NoSchedule"
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
  params:
    - name: image-name
      value: my-registry/my-app:latest

TaskRunSpecs

TaskRunSpecs allow customizing individual TaskRun configurations within a PipelineRun by specifying a list of PipelineTaskRunSpec which enables setting ServiceAccountName, Pod template, and Metadata for each task.

Use Cases

  • Applying different ServiceAccounts to different Tasks for fine-grained access control
  • Setting Task-specific Pod templates to meet different resource or security requirements
  • Configuring Task-specific metadata for better organization and tracking
  • Overriding global PipelineRun configurations for specific Tasks
  • Applying different compute resources or node selectors to different stages of the Pipeline

Principles

TaskRunSpecs:

  • Override the Pod template set for the entire Pipeline
  • Allow fine-grained control over Task execution environments
  • Can be used to meet specific security, resource, or scheduling requirements
  • Enable different configurations for different stages of the Pipeline
  • Support all standard Kubernetes Pod template specifications
  • Can be combined with global PipelineRun configurations

Configuration Fields

Each TaskRunSpec supports the following fields:

  • pipelineTaskName: The name of the Pipeline task to apply the configuration to
  • serviceAccountName: The ServiceAccount to use for this specific Task
  • podTemplate: Pod template configuration specific to this Task
  • metadata: Metadata configuration for the TaskRun

Configuration Examples

Example 1: Basic TaskRunSpecs with ServiceAccount and Pod Template

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-with-taskrunspecs
spec:
  pipelineRef:
    name: build-deploy-app
  taskRunSpecs:
    - pipelineTaskName: build
      serviceAccountName: build-service-account
      podTemplate:
        nodeSelector:
          disktype: ssd
          environment: build
        tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "build"
    - pipelineTaskName: deploy
      serviceAccountName: deploy-service-account
      podTemplate:
        nodeSelector:
          disktype: ssd
          environment: production
        tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "deploy"
          effect: "NoSchedule"
  params:
    - name: image-name
      value: my-registry/my-app:latest

Example 2: TaskRunSpecs with Advanced Pod Configuration

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: advanced-taskrunspecs
spec:
  pipelineRef:
    name: multi-stage-pipeline
  taskRunSpecs:
    - pipelineTaskName: build
      serviceAccountName: build-sa
      podTemplate:
        nodeSelector:
          gpu: available
          disktype: nvme
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
    - pipelineTaskName: test
      serviceAccountName: test-sa
      podTemplate:
        nodeSelector:
          environment: testing
        tolerations:
        - key: "test-only"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
    - pipelineTaskName: deploy
      serviceAccountName: deploy-sa
      podTemplate:
        nodeSelector:
          environment: production
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/os
                  operator: In
                  values:
                  - linux

Example 3: TaskRunSpecs with Metadata Configuration

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: taskrunspecs-with-metadata
spec:
  pipelineRef:
    name: build-test-deploy
  taskRunSpecs:
    - pipelineTaskName: build
      serviceAccountName: build-sa
      metadata:
        labels:
          stage: build
          environment: development
        annotations:
          tekton.dev/description: "Build stage with optimized resources"
      podTemplate:
        nodeSelector:
          disktype: ssd
    - pipelineTaskName: deploy
      serviceAccountName: deploy-sa
      metadata:
        labels:
          stage: deploy
          environment: production
        annotations:
          tekton.dev/description: "Production deployment stage"
      podTemplate:
        nodeSelector:
          environment: production

Example 4: TaskRunSpecs Overriding Global Pod Template

TaskRunSpecs can override the global Pod template set at the PipelineRun level:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: override-global-template
spec:
  pipelineRef:
    name: build-deploy-app
  taskRunTemplate:
    serviceAccountName: default
    # Global Pod template for all Tasks
    podTemplate:
      nodeSelector:
        disktype: ssd
      tolerations:
      - key: "general"
        operator: "Equal"
        value: "pipeline"
        effect: "NoSchedule"
  # Task-specific overrides
  taskRunSpecs:
    - pipelineTaskName: build
      podTemplate:
        nodeSelector:
          disktype: nvme
          gpu: available
        tolerations:
        - key: "build-only"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
    - pipelineTaskName: deploy
      podTemplate:
        nodeSelector:
          environment: production
        tolerations:
        - key: "deploy-only"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
  params:
    - name: image-name
      value: my-registry/my-app:latest

Workspaces

Workspaces allow sharing data between Tasks in a Pipeline.

Use Cases

  • Sharing source code between build and test Tasks
  • Passing build artifacts to deployment Tasks
  • Sharing configuration files across multiple Tasks
  • Persisting data between Pipeline executions

Principles

Workspaces:

  • Are declared in the Pipeline and bound in the PipelineRun
  • Can be backed by various volume types (PVC, ConfigMap, Secret, etc.)
  • Enable data sharing without direct dependencies
  • Support different access modes (ReadOnly, ReadWrite)

Configuration Example

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-test-with-workspace
spec:
  pipelineRef:
    name: build-test-pipeline
  workspaces:
    - name: source-code
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
    - name: cache
      persistentVolumeClaim:
        claimName: build-cache-pvc

PipelineRun Status Management

Monitoring Execution Status

The status field of a PipelineRun provides detailed information about the execution progress:

status:
  completionTime: "2023-05-04T02:19:14Z"
  conditions:
    - lastTransitionTime: "2023-05-04T02:19:14Z"
      message: "Tasks Completed: 4, Skipped: 0"
      reason: Succeeded
      status: "True"
      type: Succeeded
  startTime: "2023-05-04T02:00:11Z"
  childReferences:
  - name: build-deploy-app-run-build
    pipelineTaskName: build
    kind: TaskRun

Cancelling a PipelineRun

To cancel a running PipelineRun, update its spec.status field:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-app-run
spec:
  # [...other fields...]
  status: "Cancelled"

This immediately terminates all running TaskRuns and does not execute pending Tasks or finally Tasks.

Gracefully Stopping a PipelineRun

To gracefully stop a PipelineRun, allowing finally Tasks to execute:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: build-deploy-app-run
spec:
  # [...other fields...]
  status: "StoppedRunFinally"

This completes currently running Tasks but does not start new ones, except for finally Tasks.

References