Auto-Create GitLab Tenant Group and Connector with Tekton
When a new tenant is onboarded onto Alauda Container Platform, operators
typically have to repeat the same GitLab setup steps: create a tenant
group (and any per-team subgroups), mint a Group Access Token, then
kubectl apply a tenant gitlab Connector + matching auth Secret. This
guide turns that flow into a single Tekton TaskRun, and shows how to run
the same TaskRun on a schedule to rotate the tenant token before it
expires.
The implementation mechanism is the
gitlab-connector-automatic-creation Task.
Run it once for initial provisioning, run it again later to rotate
(or — when an identity-affecting input changes — recreate) the tenant
GAT.
The concept page GitLab CLI Configuration and Auto-Creation Patterns covers the underlying model (Pattern A vs Pattern B, the three-path GAT lifecycle, and how the admin credential stays in the CSI mount). This page focuses on the operator workflow.
TOC
What This SolvesBefore You BeginChoose a Deployment PatternPrerequisites — Pattern A (Top-Level +can_create_group User)Prerequisites — Pattern B (Subgroup of Umbrella + Admin User PAT)Optional: Kubernetes Connector for Cross-Cluster ApplyServiceAccount PermissionsA dedicated, least-privilege ClusterRolePlatform-team pattern: cluster-scoped role, per-tenant bindingWiring the SA into the TaskRunStep-by-Step — Pattern A: Fresh CreationStep-by-Step — Pattern B: Fresh CreationStep-by-Step — Rotate-in-PlaceCronJob Pattern for Scheduled RotationStep-by-Step — Recreate on Identity ChangeStep-by-Step — Expired-Fall-Through (Self-Healing)Verify the ResultOperations RunbookToken already expired (missed rotation window)Token-refresh CronJob disabled or misconfiguredTenant offboarding rollbackPattern-A user ownership auditGroup-path conflict at fresh-creation timeRotation-during-use raceValidating scopes against your instanceManual Cross-Group Permissions WorkaroundTroubleshootingtenantGroup value rejectedscopes does not have a valid valuePattern A admin lacks can_create_groupPattern B admin lacks owner on the umbrellaGroup-path conflictKubernetes RBAC missing for in-cluster applyFurther ReadingWhat This Solves
- Tenant onboarding. One TaskRun creates the tenant GitLab group, any
optional subgroups, the tenant GAT, and the tenant
gitlabConnector- Secret on the cluster.
- Token rotation. Re-running the TaskRun on a schedule rotates the tenant GAT in place; the Secret is updated atomically; running pipelines re-read the Secret on next launch.
- Self-healing. If a rotation cycle is missed and the GAT expires before the next run, the Task detects the expired token and automatically falls through to a recreate. No manual intervention is required.
- Recreate on identity change. Changing
scopes,accessLevel, or thesubgroups[]set on a re-run triggers a controlled delete-and-recreate of the GAT.
Before You Begin
Make sure the environment meets these requirements:
- Tekton Pipelines is installed.
- The Connectors system and the GitLab Connector ConnectorClass are installed.
- The
gitlab-connector-automatic-creationTask is installed in the namespace where you will run the TaskRun. Apply it from thecmd/kodata/connectors-gitlab-tektoncd/1.0.0/install.yamlsource manifest (or the published catalog shape) — see the Installation section of the Task reference for theREGISTRY+TARGET_NS+envsubst | kubectl applyrecipe. The connectors-operator installs only the framework (CRDs, RBAC, ConnectorClass, ResourceInterface) — the Task itself is user-applied per Tekton convention. - The catalog
glabandkubectltool images are reachable from your internal registry. The Task uses the existingcatalog.tekton.dev/tool-image-glabandcatalog.tekton.dev/tool-image-kubectlConfigMaps inkube-public.
Security recommendation:
- Keep the admin GitLab Connector in a private management namespace
(e.g.
connectors-management). - Restrict who can trigger the TaskRun — that action mints (and rotates) Group Access Tokens.
- Pick the smallest-footprint admin identity per pattern (see Choose a Deployment Pattern below).
Choose a Deployment Pattern
The Task itself does not branch on pattern — both patterns flow through the same code path. Pick based on your GitLab tenancy shape:
Why a user PAT and not a Group Access Token (GAT) on the umbrella? A GAT belongs to a group bot user that GitLab restricts to the issuance group: it cannot mint GATs on freshly-created subgroups (HTTP 400 — "User does not have permission to create group access token"), and group bots cannot be added as direct members of other groups. The Task therefore requires a regular user identity — a dedicated automation user that you grant Owner on the umbrella — to mint per-tenant GATs on subgroups. See Pattern B prerequisites below.
See Pattern A vs Pattern B for the full comparison.
Prerequisites — Pattern A (Top-Level + can_create_group User)
-
Create or pick a dedicated GitLab user (do not use a human identity). On self-hosted GitLab, ask the GitLab instance admin to set this user's
can_create_groupflag totrue. On GitLab.com, any user can create top-level groups so no flag setup is required. -
Restrict that user's ownership. The user becomes owner of every group it creates, and its blast radius if compromised is the union of those groups. Document a rule like "this user must not own anything outside the auto-created tenant set" and audit it on a schedule (see Pattern-A user ownership audit in the runbook below).
-
Mint a PAT for that user with at least the
apiscope. Store the PAT as a Kubernetes Secret in the management namespace: -
Create the admin Connector pointing at the Secret:
Prerequisites — Pattern B (Subgroup of Umbrella + Admin User PAT)
-
Create the umbrella group (e.g.
umbrella) on GitLab if it does not already exist. This is a one-time operation — every tenant from here on lives as a subgroup under it. -
Create or pick a dedicated GitLab user to be the umbrella admin. Treat it like the Pattern A automation user: a non-human bot account, not a real person's identity. You do not need
can_create_groupfor this user (it creates subgroups, not top-level groups). -
Add that user to the umbrella as Owner so it can create subgroups under it AND mint GATs on those subgroups (GAT-creation endpoints require direct Owner membership, which group access tokens cannot satisfy — that's why a user identity is required for Pattern B).
-
Mint a PAT for that user with at least the
apiscope. Store the PAT as a Kubernetes Secret in the management namespace: -
Create the admin Connector pointing at the umbrella Secret:
Optional: Kubernetes Connector for Cross-Cluster Apply
When the management namespace and the tenant Connector live on different
clusters, the Task supports an optional kubeconfig workspace. Mount it
from a kubernetes-class Connector:
Recommended: mount a kubeconfig workspace from a kubernetes-class
Connector even on the same cluster. It keeps the Task's effective identity
explicit (the SA whose token the Connector serves) and uniform across
same-cluster and cross-cluster setups. See
Using Kubernetes Connector in Tekton Task
for the Connector setup.
When the kubeconfig workspace is unbound the Task falls back to the
TaskRun's in-cluster ServiceAccount — viable for same-cluster usage, but
the platform team usually prefers the kubernetes-Connector path so the
provisioning identity lives on a Connector, not on a TaskRun's SA.
ServiceAccount Permissions
This Task is intended for platform teams who provision GitLab groups
and gitlab Connectors on behalf of many application teams. The
permission set is small, generic across Connector types (Harbor, GitLab,
…), and best expressed as a cluster-scoped role bound per tenant
namespace.
The provisioning ServiceAccount needs write access to two resource kinds
in each tenant connector namespace (the <conn_ns> portion of the
connector parameter — e.g. connectors-management/acme-corp →
connectors-management):
The Task also calls kubectl auth can-i create secrets as a pre-flight
check; that uses SelfSubjectAccessReview, which every authenticated
user can call without explicit RBAC.
Whose permissions are these?
- Recommended path — the SA whose token is embedded in the kubeconfig
served by the kubernetes Connector mounted on the
kubeconfigworkspace. - Same-cluster fallback — the TaskRun's
serviceAccountName.
The permission shape is identical either way. Use the same SA (and the same ClusterRole + RoleBinding pair below) for the Harbor automatic creation Task too — the rules are not GitLab-specific.
A dedicated, least-privilege ClusterRole
The operator ships a broad connectors-connector-editor-role ClusterRole
that grants the full edit verb set
({create,delete,get,list,patch,update,watch}). The Task does not need
that surface. It only ever runs kubectl apply --server-side against
two object kinds — so the supported recipe is one dedicated ClusterRole
with exactly the three verbs the Task uses, named to make the apply-only
intent obvious:
Naming convention: tenant- prefix flags that the binding scope is the
tenant namespace (not cluster-wide); -writer (vs -editor) signals the
narrow apply-only verb set; task- makes clear this is the role for
Tekton-Task-driven provisioning. Does not collide with anything shipped
under cmd/kodata/*/install.yaml.
Platform-team pattern: cluster-scoped role, per-tenant binding
Define the provisioning SA once, in a platform namespace, then bind
the ClusterRole into each tenant namespace via a per-namespace
RoleBinding. This keeps the SA's reach scoped to the namespaces where
you actually want to provision Connectors (no cluster-wide secret or
connector write):
To onboard a new tenant namespace later, the platform team applies only the RoleBinding (rebound to the new namespace) — no SA or ClusterRole churn.
If you need cluster-wide reach (e.g. trust the SA in every namespace), swap the per-namespace RoleBinding for a ClusterRoleBinding. This trades scoped safety for one less manifest per tenant; pick deliberately.
Wiring the SA into the TaskRun
For the recommended kubernetes-Connector path, the TaskRun does not reference this SA directly — bind it via the kubernetes Connector instead, so the SA's token is what the kubeconfig embeds. For the same-cluster fallback, set it on the TaskRun:
A future connectors-operator release will likely ship the
connectors-tenant-task-writer ClusterRole above out of the box —
mirroring the role-shape convention already used by
connectors-connector-editor-role / connectors-connector-viewer-role.
Until then, the platform team owns it.
Step-by-Step — Pattern A: Fresh Creation
This onboarding example creates a top-level GitLab group acme-corp with
two per-team subgroups, mints an owner GAT at acme-corp with a 30-day
expiry, and lands a gitlab Connector + Secret in
connectors-management/acme-corp.
Apply it and wait for completion:
After the TaskRun succeeds:
- GitLab contains top-level group
acme-corpand subgroupsacme-corp/backend,acme-corp/frontend. - A GAT named
connector-connectors-management-acme-corp/<identity>exists atacme-corpwith scopes[api], access levelowner, expiry today + 30 days. - Cluster contains
Secret/acme-corp(typeconnectors.cpaas.io/gitlab-pat-auth) andConnector/acme-corppointing at it, both inconnectors-management.
Step-by-Step — Pattern B: Fresh Creation
Same tenant, different deployment pattern: the umbrella umbrella is
pre-existing, and the admin Connector carries a user PAT whose user
holds Owner on the umbrella. (A Group Access Token cannot mint per-tenant
GATs on subgroups — see the explanation in the pattern picker.)
Result: identical cluster outputs to Pattern A; on the GitLab side the
new group sits at umbrella/acme-corp (subgroup of the umbrella) instead
of at the top level, and the GAT is issued at umbrella/acme-corp (NOT
at the umbrella).
Step-by-Step — Rotate-in-Place
Rerunning the TaskRun with all params unchanged rotates the existing
GAT in place. The Task locates the existing GAT by its name prefix
(connector-<connector-namespace>-<connector-name>), confirms the
identity matches, and calls
POST /groups/:id/access_tokens/:token_id/rotate. GitLab atomically
invalidates the old token and returns a fresh one with expires_at
advanced by tokenDuration days; the tenant Secret is updated in place.
The TaskRun is intentionally identical in shape to the Pattern A creation
TaskRun. Cron-friendly: tokenDuration is a number of days, so every
rotation refreshes expires_at to today + N days.
CronJob Pattern for Scheduled Rotation
Wrap the rotate TaskRun in a CronJob so it fires on a schedule:
Store the rotate TaskRun manifest in the rotate-acme-corp-gat-template
ConfigMap; the CronJob kubectl creates a fresh TaskRun each tick. Pair
this with a stale-token alert (see
Token already expired
in the runbook).
Tip. Set the CronJob schedule so consecutive runs are spaced shorter than
tokenDurationminus a safety margin. FortokenDuration=30days, schedule weekly. FortokenDuration=7days, schedule daily.
Step-by-Step — Recreate on Identity Change
When you change scopes, accessLevel, or the subgroups[] set, the
Task detects the identity drift and recreates the tenant GAT instead of
rotating it. Example: a scope change from [api] to
[api, read_registry]:
The behavioural sequence:
- The Task locates the existing GAT.
- The existing GAT's identity no longer matches the desired identity.
- The Task revokes it (
DELETE /groups/:id/access_tokens/:token_id) and mints a fresh one with the requested identity. - The tenant Secret is updated with the new token.
- Annotation
connectors.cpaas.io/gat-recreated-atis set on the Secret.
Note: the subgroup set is NOT identity-affecting only when extending
— adding team-c to subgroups rotates the GAT in place and creates
the missing subgroup under <tenantGroup>/team-c. Removing or changing
the canonical subgroup set is identity-affecting and triggers recreate.
Step-by-Step — Expired-Fall-Through (Self-Healing)
If a rotation cycle is missed and the GAT's expires_at passes before
the next run, the Task self-heals. The recovery TaskRun is identical in
shape to the rotate TaskRun. On the GitLab side:
ensure-gat(step 3 of the Task) lists existing GATs atacme-corpand finds the one with theconnector-<ns>-<name>prefix.- It detects
expires_at < nowand logsWARN: existing GAT <id> is expired; falling through to recreate. - It revokes the stale GAT (best-effort; GitLab garbage-collects expired tokens too) and mints a fresh one.
- The tenant Secret is rewritten with the new token value.
The TaskRun reports Succeeded. No operator action is required for
the immediate recovery — but you should still investigate why the
rotation cron missed (see
Token already expired
in the runbook below).
Verify the Result
Check the TaskRun:
Check the tenant Connector and Secret:
Check the rotation/recreation annotations on the Secret:
Sanity-check the GitLab side with glab:
You should see the GAT named
connector-connectors-management-acme-corp/<identity> with the requested
scopes, access_level, and a forward-looking expires_at.
Operations Runbook
Each scenario is structured as: trigger → what the Task does → operator action → monitoring signal.
Token already expired (missed rotation window)
Trigger. Operator created the tenant Connector with
tokenDuration=30 but the next rotation TaskRun did not run within 30
days (CronJob disabled, controller paused, weekend on-call miss, etc.).
On day 31+, GitLab has already expired the GAT.
What the Task does. On re-run, ensure-gat reads the existing
GAT from GET /groups/<tenantGroup>/access_tokens and detects
expires_at < now. It logs
WARN: existing GAT <id> is expired; falling through to recreate and
switches from the rotate path to the delete + recreate path
automatically. GitLab's rotate endpoint would otherwise return
400 invalid_grant (token expired); the Task never calls it.
Outcome. Same end state as a normal recreate: the old GAT id is
gone (best-effort delete; GitLab also garbage-collects expired tokens),
a new GAT is issued at tenantGroup, the Secret is rewritten, and the
Connector annotation connectors.cpaas.io/gat-recreated-at is set. The
TaskRun reports Succeeded. No manual operator intervention is
required.
Operator action. None for the immediate recovery. Long-term:
investigate why the rotation cron missed and tighten the alert threshold
below the expiry window (e.g. alert when stale-token age exceeds
tokenDuration - 7d).
Monitoring signal. Recommended alert rules:
gat_age_seconds > tokenDuration_seconds - 604800(warn — 7 days before expiry).gat_age_seconds > tokenDuration_seconds(page — already expired; self-healing on next TaskRun, but tenants are reading stale credentials in the meantime).taskrun_status == "Failed"for the rotation CronJob's TaskRun (page).
Token-refresh CronJob disabled or misconfigured
Trigger. Either someone disabled the CronJob, or the schedule silently stopped firing (Tekton Trigger misconfig, paused controller, etc.).
What the Task does. Nothing — the Task only runs when invoked. The "missed rotation" failure mode above kicks in once expiry arrives.
Operator action. Re-enable the CronJob; verify the next run succeeds; review the alerting that should have fired.
Monitoring signal. Page on kube_cronjob_spec_suspend == 1 for the
rotation CronJob; page on the rotation CronJob having no successful
Job run within the last 2 × schedule interval.
Tenant offboarding rollback
Trigger. Tenant acme-corp is being decommissioned; the operator
wants to revoke GitLab access cleanly.
What the Task does. Out of scope — the Task only creates and rotates. The recommended manual sequence is:
A future "tenant teardown" Task is a candidate follow-up; tracked as a deferred ticket once tenant demand surfaces.
Pattern-A user ownership audit
Trigger. Periodic compliance check (the Pattern A user accumulates ownership of every group it creates, so its blast radius grows over time).
What the Task does. Nothing — read-only audit. Run this snippet on a schedule (or wire it into a separate audit Task as a follow-up):
Operator action. If the owned-groups list contains anything the user is not supposed to own, escalate; otherwise no-op. The audit is the visibility tool for the user's residual blast-radius risk.
Monitoring signal. None native — wire the audit output into your SIEM or a periodic CI job that diffs against an expected list of auto-created tenant groups.
Group-path conflict at fresh-creation time
Trigger. Operator chose tenantGroup=acme-corp but acme-corp
already exists at GitLab and is owned by someone else (not the admin
identity).
What the Task does. ensure-group GET /groups/acme-corp returns
200, sees the admin identity is not in the owners list, and exits with:
No mutation on either side.
Operator action. Pick a different tenantGroup (e.g.
acme-corp-prod or tenants/acme-corp if you have an umbrella) or
coordinate with the existing group's owner.
Monitoring signal. TaskRun Failed with the explicit error message
in the step log. Alert on the message itself if you want a fast signal.
Rotation-during-use race
Trigger. A rotate TaskRun runs concurrently with a long-lived tenant
pipeline that reads the same tenant Secret/<connector-name>. The
in-flight pipeline pod read the old token value at startup;
mid-flight, the rotate TaskRun calls
POST /groups/:id/access_tokens/:token_id/rotate, which invalidates the
old token atomically; the in-flight pod's subsequent push or API call
fails with 401.
What the Task does. The Task does not coordinate with tenant pipelines — rotation is unconditional on schedule.
Operator action. Two options:
- Schedule rotation when tenant pipelines are idle. Pick a CronJob
window (e.g.
0 2 * * 0— 02:00 Sunday) that minimises overlap with business pipelines. - Make tenant pipelines retry on 401. When the tenant workload
detects a
401, re-mount or re-read the tenant Secret and retry. The Secret is updated atomically by the rotate TaskRun; the next read will pick up the new token.
We recommend running both mitigations.
Monitoring signal. Tenant-side 401 rate spike on the rotation
window. Pair the rotation CronJob with a dashboard panel showing
http_request_total{code="401"} by tenant.
Validating scopes against your instance
Trigger. A re-run with new scopes (e.g. adding read_registry or
write_registry) fails with the bare GitLab error:
GitLab does not name which scope is invalid in the error body. The Task surfaces this verbatim.
Why this happens. Some scopes are gated on instance features. For
example read_registry and write_registry require the GitLab
container registry feature to be enabled on the instance; on instances
where that feature is disabled, GitLab rejects the entire scopes array.
Operator action. Pre-validate the scope set against your instance before changing it on a production tenant:
If the probe fails, drop the offending scope from your tenant TaskRun input and retry. If you need a scope that requires an instance feature (e.g. container registry), enable that feature on the instance first.
Monitoring signal. TaskRun Failed with
"scopes does not have a valid value" in the step log. Alert on the
phrase to surface scope-validation failures before they propagate to
multiple tenants.
Manual Cross-Group Permissions Workaround
The auto-creation Task issues one GAT at the tenant group with one
accessLevel. If you need different access levels for different
subgroups (e.g. developer on team-a, maintainer on team-b)
under one tenant identity, the Task does not cover that today.
A first-class option is on the deferred follow-up list. The manual workaround is:
-
Create a dedicated GitLab service-account user and a PAT for it (Pattern A's user-PAT shape).
-
Use the auto-creation TaskRun to create the tenant group + subgroups and the cluster-side Connector + Secret (with any reasonable
accessLevel— it will be overridden manually). -
Add the service-account user as a member of each subgroup with the desired
access_level: -
Replace the GAT in the tenant Secret with the service-account PAT. Subsequent rotation TaskRuns will overwrite this — so document the workaround as a deferred follow-up rather than a recommended steady state.
Track the demand for first-class per-group access-level support; it will eventually become a Task-level option.
Troubleshooting
tenantGroup value rejected
The tenantGroup value must be a GitLab group full path. Top-level
paths are simple slugs (acme-corp); subgroup paths use slashes
(umbrella/acme-corp). Slug rules follow GitLab's group-path rules
(lowercase letters, digits, -, _, .).
scopes does not have a valid value
See Validating scopes against your instance in the runbook. GitLab rejects the entire scopes array if any one scope is invalid for the instance.
Pattern A admin lacks can_create_group
The admin Connector's user PAT belongs to a user without the
can_create_group flag, and the request asks for a top-level
tenantGroup. The Task fails with:
Set the flag (instance admin) or switch to Pattern B.
Pattern B admin lacks owner on the umbrella
The admin Connector's PAT belongs to a user who is only a maintainer
on the umbrella. Subgroup creation may succeed but the Task fails on the
GAT-mint step (the GitLab API requires direct Owner membership for GAT
creation):
Promote the admin user to Owner on the umbrella (Group → Members →
edit role).
Group-path conflict
See Group-path conflict at fresh-creation time in the runbook.
Kubernetes RBAC missing for in-cluster apply
If the optional kubeconfig workspace is unbound and the in-cluster
ServiceAccount lacks RBAC to create Secrets and Connectors in the
connector namespace, the apply step fails with 403 Forbidden. Either
grant the SA the required permissions or bind a kubeconfig workspace
that carries an identity with those permissions.
Further Reading
gitlab-connector-automatic-creationTask reference — full parameter, workspace, and result reference.- GitLab CLI Configuration and Auto-Creation Patterns
— concept page covering the
gitlabconfigCSI mount, Pattern A vs Pattern B, and the three-path GAT lifecycle. - GitLab Connector — the underlying GitLab ConnectorClass and its tenant-side configurations.