Troubleshooting

I've deleted/corrupted my repo and can't delete my app?

Argo CD can't delete an app if it cannot generate manifests. You need to either:

  1. Reinstate/fix your repo.
  2. Delete the app using --cascade=false and then manually deleting the resources.

Why is my application stillOutOfSync immediately after a successful Sync?

See Diffing Documentation for reasons resources can be OutOfSync, and ways to configure Argo CD to ignore fields when differences are expected.

Why is my application stuck inProgressing state?

Argo CD provides health for several standard Kubernetes types. The Ingress, StatefulSet and SealedSecret types have known issues which might cause health check to return Progressing state instead of Healthy.

  • Ingress is considered healthy if status.loadBalancer.ingress list is non-empty, with at least one value for hostname or IP. Some ingress controllers (contour , traefik) don't update status.loadBalancer.ingress field which causes Ingress to stuck in Progressing state forever.

  • StatefulSet is considered healthy if value of status.updatedReplicas field matches to spec.replicas field. Due to Kubernetes bug kubernetes#68573 the status.updatedReplicas is not populated. So unless you run Kubernetes version which include the fix kubernetes#67570 StatefulSet might stay in Progressing state.

  • Your StatefulSet or DaemonSet is using OnDelete instead of RollingUpdate strategy.

  • For SealedSecret, see Why are resources of type SealedSecret stuck in the Progressing state?

As workaround Argo CD allows providing health check customization which overrides default behavior.

If you are using Traefik for your Ingress, you can update the Traefik config to publish the loadBalancer IP using publishedservice, which will resolve this issue.

providers:
  kubernetesIngress:
    publishedService:
      enabled: true

How to disable admin user?

Add admin.enabled: "false" in the argocd-cm ConfigMap.

Argo CD cannot deploy Helm Chart based applications without internet access, how can I solve it?

Argo CD might fail to generate Helm chart manifests if the chart has dependencies located in external repositories. To solve the problem you need to make sure that requirements.yaml uses only internally available Helm repositories. Even if the chart uses only dependencies from internal repos Helm might decide to refresh stable repo. As workaround override stable repo URL in argocd-cmconfig map:

data:
  repositories: |
    - type: helm
      url: http://<internal-helm-repo-host>:8080
      name: stable

After creating my Helm application with Argo CD I cannot see it with helm ls and other Helm commands?

When deploying a Helm application Argo CD is using Helm only as a template mechanism. It runs helm template and then deploys the resulting manifests on the cluster instead of doing helm install. This means that you cannot use any Helm command to view/verify the application. It is fully managed by Argo CD. Note that Argo CD supports natively some capabilities that you might miss in Helm (such as the history and rollback commands).

This decision was made so that Argo CD is neutral to all manifest generators.

I've configured cluster secret but it does not show up in CLI/UI, how do I fix it?

Check if the cluster secret has the label argocd.argoproj.io/secret-type: cluster. If the secret has the label but the cluster is still not visible, it may be a permission issue. Try listing the clusters using the admin user (e.g.: argocd login --username admin && argocd cluster list).

Check if cluster secret has argocd.argoproj.io/secret-type: cluster label. If secret has the label but the cluster is still not visible then make sure it might be a permission issue. Try to list clusters using admin user (e.g. argocd login --username admin && argocd cluster list).

Why Is My App Out Of Sync Even After Syncing?

In some cases, the tool you use may conflict with Argo CD by adding the app.kubernetes.io/instance label. E.g. using Kustomize common labels feature.

Argo CD automatically sets the app.kubernetes.io/instance label and uses it to determine which resources form the app. If the tool does this too, this causes confusion. You can change this label by setting the application.instanceLabelKey value in the argocd-cm. We recommend that you use argocd.argoproj.io/instance.

INFO

When you make this change your applications will become out of sync and will need re-syncing.

How often does Argo CD check for changes to my Git or Helm repository?

The default polling interval is 3 minutes (180 seconds) with a configurable jitter. You can change the setting by updating the timeout.reconciliation value and the timeout.reconciliation.jitter in the argocd-cm config map. If there are any Git changes, Argo CD will only update applications with the auto-sync setting enabled. If you set it to 0 then Argo CD will stop polling Git repositories automatically and you can only use alternative methods such as webhooks and/or manual syncs for creating applications.

Argo CD uses a JWT as the auth token. You likely are part of many groups and have gone over the 4KB limit which is set for cookies. You can get the list of groups by opening "developer tools -> network":

  1. Click login to the Argo CD dashboard How to Obtain Argo CD Access Information
  2. Find the call to <argocd_instance>/auth/callback?code=<random_string>

Decode the token at jwt.io. That will provide the list of teams that you can remove yourself from.

Why Am I Getting rpc error: code = Unavailable desc = transport is closing When Using The CLI?

Maybe you're behind a proxy that does not support HTTP 2? Try the --grpc-web flag:

argocd ... --grpc-web

Why are resources of type SealedSecret stuck in the Progressing state?

The controller of the SealedSecret resource may expose the status condition on resource it provisioned. Since version v2.0.0 Argo CD picks up that status condition to derive a health status for the SealedSecret.

Versions before v0.15.0 of the SealedSecret controller are affected by an issue regarding this status conditions updates, which is why this feature is disabled by default in these versions. Status condition updates may be enabled by starting the SealedSecret controller with the --update-status command line parameter or by setting the SEALED_SECRETS_UPDATE_STATUS environment variable.

To disable Argo CD from checking the status condition on SealedSecret resources, add the following resource customization in your argocd-cm ConfigMap via resource.customizations.health.<group_kind> key.

resource.customizations.health.bitnami.com_SealedSecret: | hs = hs.status = "Healthy" hs.message = "Controller doesn't report resource status" return hs

How to rotate Redis keys?

  • Delete argocd-redis secret in the namespace where Argo CD is installed.
kubectl delete secret argocd-redis -n <argocd namesapce>
  • If you are running Redis in HA mode, restart Redis in HA.
kubectl rollout restart deployment argocd-redis-ha-haproxy
kubectl rollout restart statefulset argocd-redis-ha-server
  • If you are running Redis in non-HA mode, restart Redis.
kubectl rollout restart deployment argocd-redis
  1. Restart other components.
kubectl rollout restart deployment argocd-server argocd-repo-server
kubectl rollout restart statefulset argocd-application-controller

How do I fix Manifest generation error (cached)?

Manifest generation error (cached) means that there was an error when generating manifests and that the error message has been cached to avoid runaway retries.

Doing a hard refresh (ignoring the cached error) can overcome transient issues. But if there's an ongoing reason manifest generation is failing, a hard refresh will not help.

Instead, try searching the repo-server logs for the app name in order to identify the error that is causing manifest generation to fail.