CSI Workload Configuration Troubleshooting
This guide provides detailed steps for diagnosing and resolving common issues with the Connectors CSI Driver when mounting configurations in workloads.
TOC
Common Issues Overview
Checking CSI Volume Configuration
Verify the CSI volume configuration in your workload YAML:
volumes:
- name: gitconfig
csi:
readOnly: true
driver: connectors-csi
volumeAttributes:
connector.name: "<connector-name>"
configuration.names: "gitconfig"
- Set
driver exactly to connectors-csi
- Ensure connector exists in the same namespace
- Set
configuration.names to gitconfig for Git operations
- Make sure connector is in the same namespace as pod
How to verify:
# Check if the connector exists
kubectl get connector <connector-name> -n <namespace>
# Verify the connector is Ready
kubectl get connector <connector-name> -n <namespace> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Check CSI driver availability
kubectl get pods -n connectors-system -l app.kubernetes.io/name=connectors-csi
Verifying Volume Mount
Check if the volume mount configuration is correct:
volumeMounts:
- name: gitconfig
mountPath: "/path/to/user/home/"
Important considerations:
- The mount path must match the home directory of the user running Git commands
- For Git configuration, the file is always created at
<mountPath>/.gitconfig
Finding the correct home directory:
# Check the user's home directory in the container
kubectl exec <pod-name> -n <namespace> -- env | grep HOME
# Common paths depending on container image:
# - For root user: /root/
# - For git user: /home/git/
# - For non-root users: /home/<username>/
Verifying mount success:
# Check if volumes are properly mounted
kubectl describe pod <pod-name> -n <namespace> | grep -A 5 "Mounts:"
# Look for mount events in the pod description
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 "Events:"
Examining Pod Events
Check Pod events for mount-related issues:
kubectl describe pod <pod-name> -n <namespace>
Common error messages and solutions:
Example error and resolution:
Warning FailedMount 3m (x5 over 5m) kubelet MountVolume.SetUp failed for volume "gitconfig" :
rpc error: code = NotFound desc = connector "git-github" not found
Resolution: Create the connector or correct the connector name in the volume attributes.
Finding the Generated Git Configuration
Locate the Git configuration file:
# Check if the expected path has the file
kubectl exec <pod-name> -n <namespace> -- ls -la /path/to/home/.gitconfig
If no configuration file is found, check:
- Volume mount is successful
- CSI driver is healthy
- ServiceAccount has permissions
- Connector is Ready
Examining Git Configuration Content
Check the generated .gitconfig file content:
kubectl exec <pod-name> -n <namespace> -- cat /path/to/.gitconfig
Expected configuration elements:
-
HTTP header with authentication token:
[http]
extraHeader = Authorization: Basic <token>
-
URL rewriting rule:
[url "http://c-<connector-name>.<connector-namespace>.svc"]
insteadOf = <original-git-url>
Advanced Troubleshooting
CSI Driver Logs
Check CSI driver logs for detailed error information:
kubectl logs -n connectors-system -l app.kubernetes.io/name=connectors-csi -c csi-driver
Testing with a Diagnostic Pod
Create a diagnostic pod to test CSI functionality:
apiVersion: v1
kind: Pod
metadata:
name: csi-debug-pod
namespace: <namespace>
spec:
containers:
- name: debug
image: alpine
command: ["sleep", "3600"]
volumeMounts:
- name: gitconfig
mountPath: "/root/"
volumes:
- name: gitconfig
csi:
driver: connectors-csi
readOnly: true
volumeAttributes:
connector.name: "<connector-name>"
configuration.names: "gitconfig"
Additional Resources