#CSI Workload Configuration Troubleshooting

This guide provides detailed steps for diagnosing and resolving common issues with the Connectors CSI Driver when mounting OCI registry configurations in workloads.

#Common Issues Overview

#Checking CSI Volume Configuration

Verify the CSI volume configuration in your workload YAML:

volumes:
- name: docker-config
  csi:
    readOnly: true
    driver: connectors-csi
    volumeAttributes:
      connector.name: "<connector-name>"
      configuration.names: "docker-config"

Common configuration options:

Common configuration issues:

How to verify:

# Check if the connector exists
kubectl get connector <connector-name> -n <namespace>

# Verify the connector is Ready
kubectl get connector <connector-name> -n <namespace> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

# Check CSI driver availability
kubectl get pods -n connectors-system -l app.kubernetes.io/name=connectors-csi

#Verifying Volume Mount

Check if the volume mount configuration is correct:

volumeMounts:
- name: docker-config
  mountPath: "/root/.docker"  # For docker-config

Common mount paths for different configurations:

#Examining Pod Events

Check Pod events for mount-related issues:

kubectl describe pod <pod-name> -n <namespace>

Common error messages and solutions:

Example error and resolution:

  Warning  FailedMount  3m (x5 over 5m)  kubelet  MountVolume.SetUp failed for volume "docker-config" : 
  rpc error: code = NotFound desc = connector "oci-registry" not found

Resolution: Create the connector or correct the connector name in the volume attributes.

#Finding Generated OCI Configuration Files

Locate the configuration files:

# For docker-config
kubectl exec <pod-name> -n <namespace> -- cat /root/.docker/config.json

# For dockerd
kubectl exec <pod-name> -n <namespace> -- cat /etc/docker/daemon.json

# For buildkitd
kubectl exec <pod-name> -n <namespace> -- cat /etc/buildkit/buildkitd.toml

If configuration files are not found, check:

1. Volume mount is successful
2. CSI driver is healthy
3. ServiceAccount has permissions
4. Connector is Ready
5. Mount path matches container user's expected path

#Examining Docker Configuration Content

#docker-config

Check the generated config.json file:

kubectl exec <pod-name> -n <namespace> -- cat /root/.docker/config.json

Expected configuration elements:

{
  "auths": {
    "c-<connector-name>.<namespace>.svc.cluster.local": {
      "auth": "<base64-encoded-token>"
    }
  }
}

#dockerd configuration

Check the generated daemon.json file:

kubectl exec <pod-name> -n <namespace> -- cat /etc/docker/daemon.json

Expected configuration elements:

{
  "insecure-registries": [
    "c-<connector-name>.<namespace>.svc.cluster.local"
  ]
}

#buildkitd configuration

Check the generated buildkitd.toml file:

kubectl exec <pod-name> -n <namespace> -- cat /etc/buildkit/buildkitd.toml

Expected configuration elements:

insecure-entitlements = [ "network.host", "security.insecure" ]
[registry."c-<connector-name>.<namespace>.svc.cluster.local"]
  http = true

#Insecure Registry Issues

Symptoms:

• server certificate verification failed errors
• TLS handshake failures

Troubleshooting:

1. Verify insecure registry settings are correctly configured:

   kubectl exec <pod-name> -n <namespace> -- cat /etc/docker/daemon.json

2. Check if the container runtime is using the mounted configuration:

   kubectl exec <pod-name> -n <namespace> -- docker info | grep -A 5 "Insecure Registries"

3. For containerd, verify proxy address is properly configured:

   kubectl exec -it <node-name> -n <namespace> -- cat /etc/containerd/config.toml | grep -A 10 registry

#Advanced Troubleshooting

#CSI Driver Logs

Check CSI driver logs for detailed error information:

kubectl logs -n connectors-system -l app.kubernetes.io/name=connectors-csi -c csi-driver

#Proxy Service Logs

Check the proxy service logs for authentication or access issues:

kubectl logs -n connectors-system -l app.kubernetes.io/name=connectors-proxy

#Testing with a Diagnostic Pod

Create a diagnostic pod to test OCI functionality:

apiVersion: v1
kind: Pod
metadata:
  name: oci-debug-pod
  namespace: <namespace>
spec:
  containers:
  - name: debug
    image: docker:20.10
    command: ["sleep", "3600"]
    volumeMounts:
    - name: docker-config
      mountPath: "/root/.docker"
  volumes:
  - name: docker-config
    csi:
      driver: connectors-csi
      readOnly: true
      volumeAttributes:
        connector.name: "<connector-name>"
        configuration.names: "docker-config"

#Additional Resources

• Connectors CSI Driver Concepts
• OCI Connector Quick Start
• Connector Status Troubleshooting
• Using OCI Connector in Kubernetes Workloads
• Using OCI Connector in Tekton Pipeline