简体中文

从 GitLab 14.0.12 迁移数据至 17.8.z

解决的问题

版本 3.16 和 3.18 支持的 GitLab 版本落后于官方版本。升级 GitLab 至最新官方版本需要超过 10 次升级才能完成。升级到 4.0 后，产品不提供自动升级至 17.8.z 的功能。本方案解决如何从 14.0.12 升级到 17.8.z。

鉴于版本跨度较大，采用数据迁移的方式进行升级。

流程概述

WARNING

升级所需时间因 GitLab 数据大小差异较大，可能需要数天完成，因此需提前评估维护时间窗口。

测试数据：包含 3 个项目（1 个 666MB 大项目，2 个空项目），备份文件大小 668MB，升级耗时 8 小时。

数据迁移路径
根据官方升级路径文档，数据迁移路径如下：

14.0.12 -> 14.3.6
14.3.6 -> 14.9.5
14.9.5 -> 14.10.5
14.10.5 -> 15.0.5
15.0.5 -> 15.4.6
15.4.6 -> 15.11.13
15.11.13 -> 16.3.8
16.3.8 -> 16.7.9
16.7.9 -> 16.11.10
16.11.10 -> 17.3.6
17.3.6 → 17.5.5
17.5.5 → 17.8.1

最后备份 17.8.1 的数据并导入平台部署的 17.8.z 实例，完成数据迁移。

数据迁移流程

备份平台部署的 GitLab 并恢复至 all-in-one 镜像部署的 GitLab，使用 all-in-one 镜像升级至 17.8.1，最后将数据备份恢复到平台部署的 17.8.z 实例。

备份平台部署的 GitLab 14.0.12；
将备份数据恢复到 all-in-one 部署的 14.0.12；
升级 all-in-one 部署的 GitLab 至 17.8.1；
备份 all-in-one 部署的 17.8.1；
将 all-in-one 备份数据恢复到平台部署的 17.8.z。

TIP

最新 GitLab 实例和 operator 版本请参考Release Note。

操作步骤

前置准备

在执行主机安装 kubectl、yq、base64 等工具。
配置环境变量（注意：此时 gitlab-operator 版本未升级，仍对应 GitLab 14.0.12 的 operator 版本）
```
export GITLAB_NAME=<旧 gitlab 名称>
export GITLAB_NAMESPACE=<gitlab 部署命名空间>
```

准备升级所需 PVC，需在旧 GitLab 部署的命名空间内创建

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: backup-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi # 根据实际使用情况重新评估大小
  storageClassName: nfs # 替换为集群中的存储类名称
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: upgrade-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi # 根据实际使用情况重新评估大小
  storageClassName: nfs # 替换为集群中的存储类名称
  volumeMode: Filesystem

准备升级所需镜像。下载附件中各版本镜像并推送至 GitLab 部署集群可拉取的镜像仓库。镜像下载链接：

# amd 镜像
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-0-12-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-3-6-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-9-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-10-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-0-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-4-6-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-11-13-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-3-8-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-7-9-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-11-10-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-3-6-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-5-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-8-1-amd.tgz

数据迁移步骤

备份平台部署的 GitLab 14.0.12

启用 GitLab 14 的 task-runner 以执行备份命令

$ kubectl patch gitlabofficials ${GITLAB_NAME} -n $GITLAB_NAMESPACE --type='merge' -p='{"spec":{"helmValues":{"gitlab":{"task-runner":{"enabled":"true"}}}}}'
# 执行命令后确认 ${GITLAB_NAME}-task-runner 部署存在
$ kubectl get deploy -n $GITLAB_NAMESPACE ${GITLAB_NAME}-task-runner
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
high-sc-sso-ingress-https-task-runner   1/1     1            1           178m

设置 gitlab 14 为只读模式

$ kubectl patch deploy ${GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
$ kubectl patch deploy ${GITLAB_NAME}-sidekiq-all-in-1-v1 -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
$ kubectl scale deploy ${GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --replicas 0
$ kubectl scale deploy ${GITLAB_NAME}-sidekiq-all-in-1-v1 -n $GITLAB_NAMESPACE --replicas 0

为 gitlab 14 task-runner 设置备份 PVC，PVC 名称为 backup-pvc（PVC 需提前创建，大小需根据 gitlab 数据量评估）

$ kubectl patch deploy ${GITLAB_NAME}-task-runner -n $GITLAB_NAMESPACE -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
$ kubectl patch deploy ${GITLAB_NAME}-task-runner -n $GITLAB_NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/1", "value": {"name": "task-runner-tmp", "persistentVolumeClaim": {"claimName": "backup-pvc"}}}]'

备份 gitlab 14 数据

# 获取 task-runner 名称，确保 pod 状态为 Running
$ kubectl get po -n $GITLAB_NAMESPACE -l app=task-runner,release=${GITLAB_NAME}
gitlab-task-runner-b4459444b-bklh4   1/1     Running   0          84m
$ kubectl exec -ti -n $GITLAB_NAMESPACE gitlab-task-runner-b4459444b-bklh4 bash
$ gitlab-rake gitlab:backup:create
2024-11-01 06:16:06 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2024-11-01 06:16:18 +0000 -- done
2024-11-01 06:16:18 +0000 -- Dumping repositories ...
....
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.
# 为备份文件添加权限，使用备份 pod 用户，可能与备份 pod 用户不同。
$ ls /srv/gitlab/tmp/backups
17302758xx_xxxx_xx_xx_14.0.12_gitlab_backup.tar
$ chmod 777 /srv/gitlab/tmp/backups/17302758xx_xxxx_xx_xx_14.0.12_gitlab_backup.tar
$ exit

# 备份 rails-secret
kubectl get secrets -n ${GITLAB_NAMESPACE} ${GITLAB_NAME}-rails-secret -o jsonpath="{.data['secrets\.yml']}" | base64 --decode | yq -o json > gitlab14-rails-secret.yaml

恢复备份数据至 all-in-one 部署的 14.0.12

使用 all-in-one 镜像部署 14.0.12 gitlab。

将以下 yaml 应用到 gitlab 部署命名空间，yaml 已将 backup pvc（backup-pvc）挂载到备份目录（如需在升级过程中访问 gitlab 实例，需替换 yaml 中的 IP 和 nodeport 端口）。

---
apiVersion: v1
kind: Service
metadata:
  name: gitlab-http-nodeport
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
    - appProtocol: tcp
      name: web
      port: 30855
      protocol: TCP
      targetPort: 30855
      nodePort: 30855
  selector:
    deploy: gitlab
  sessionAffinity: None
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      deploy: gitlab
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        deploy: gitlab
    spec:
      affinity: {}
      containers:
        - env:
            - name: GITLAB_OMNIBUS_CONFIG
              value: external_url 'http://192.168.132.128:30855' # 外部访问地址，需替换为集群节点的 IP 和 nodeport 端口
          image: gitlab/gitlab-ce:14.0.12-ce.0 # 可替换为可拉取的镜像
          imagePullPolicy: IfNotPresent
          name: gitlab
          ports:
            - containerPort: 30855
              name: http
              protocol: TCP
            - containerPort: 2424
              name: ssh
              protocol: TCP
          resources:
            limits:
              cpu: "4"
              memory: 8Gi
            requests:
              cpu: 2
              memory: "4Gi"
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /var/opt/gitlab/backups
              name: backup
              subPath: backups
            - mountPath: /etc/gitlab
              name: gitlab-upgrade-data
              subPath: config
            - mountPath: /var/log/gitlab
              name: gitlab-upgrade-data
              subPath: logs
            - mountPath: /var/opt/gitlab
              name: gitlab-upgrade-data
              subPath: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: backup
          persistentVolumeClaim:
            claimName: backup-pvc
        - name: gitlab-upgrade-data
          persistentVolumeClaim:
            claimName: upgrade-pvc

将备份数据恢复到 all-in-one 部署的 14.0.12 gitlab。

TIP

恢复过程中会出现数据库相关错误，must be owner 和 does not exist 是预期行为，详情见 (https://gitlab.com/gitlab-org/gitlab/-/issues/266988)。

# 使用 deployment 启动 14.0.12 gitlab
# 进入部署的 gitlab
# 获取 gitlab 14 的 gitlab-rails 文件。
$ kubectl get secrets -n ${GITLAB_NAMESPACE} ${GITLAB_NAME}-rails-secret -o jsonpath="{.data['secrets\.yml']}" | base64 --decode | yq -o json
{
  "production": {
    "secret_key_base": "abfb978b1a25ff3887458849fea585f8d14ec7295720c98d4bd4xxxxxxfbd83268730fe71841af490607ccc7deb5974acdcde9238c441b19b4aa6c568",
    "otp_key_base": "8d1dac40d1c80c8c3fd9507417c6c3ef8e7d2078722de844d78b0xxxxxx3256cde5ec2f6ed954ae131a3604905d5470fce7cfcf887c3b8f55f6f8b0aac",
    "db_key_base": "2491be299da2a44310bdd3a9999c27c685a98xxxxx9932be0e040211344aed132544efa5eb6d46670e365b24451a7d7ce61a721792c0",
    "openid_connect_signing_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAxxxxxFtr64yx4zyrDMOV5QpY03jxr8Wqm/LkjJh3Y7DJHBSQIDAQxeem9Tlx8L\nS5OBfHS8QoH1NFGxM30o6mlUmiSGWhnQoRIUi3+9vR0WmvA1Ab904NL7HmWh4p5WpC7\n3lMXsAyOaRBmLcmXQuTzsB5OD6RuTqYMCqaBaWxewMqP/TrluaO4dObeSzRRf7PV\nElvQsrqTsqWV+a38bTlG4Z0GJLT76Eq4KGn0mDnnEGx6uezvJf4MUiECgYEA0BNa\nz6xQddjDC2rm9YNkQZQPBBdn0nqq0h+Qd6Ymap16VFVSSebjI4lshqiUOpDPbw2o\nD3eJMeVbONguYFHqD+RqzgmFMTyvXAOEY1lvVMfiPaTXRzSuYz31WGr11tmeG7OF\nHdGUQ6+Zc4h3YLDO8y4hME7o/FquMxElvbKZWikCgYAKuHI8MJOy1i1/u1i6Svy3\nzJfbzLzDsPBS2LOI7bkHdBOpf+DsnGX9yHnDFemupC81Vhyvq0MYtVsJyFOkUhU4\n6i3EpiBIJRbFKi0hQhAz3mBhvrrK8t8j3K15Bj6tlh7JPltNlTKLcsrwFJTriUoj\nSvCRMgfyai29sqiOa1+ywQKBgQCZll/GwSOXCUxXRi569ORw/4/h7kDlfURP25qw\npsTel6UvUNdv02y/03V3JEJdxHxJNeRinlJ3sRuXpwL8eBp0Zp9rvF1DTc8G9VWo\nW+CwzOYzqFR7q+g5Owe5nyId1/475lQRAZ0WJSz4ubeceIYZvGglF2oks+63pSWd\nk5JcmQKBgQDuE/LftG8cMAet/bg6LkXeK2DVultT7UXD7bUNcm7hlliZL8piVNrW\nMBq4VRfuO/2kcYA4NyMUDFuRQCxQ9lsOkRKXBUv/PutyRo/jFe/d/jfvEbM7MSph\n/Puu6GQPYjW3MY9CV59uRAMKWE1vK2zjPume143wyILu6SVDQaKNTA==\n-----END RSA PRIVATE KEY-----\n",
    "ci_jwt_signing_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAw+emHJQxCFmM/9oaUqjRfEC4oha0qL/ahIsf11oFayGTlR0K\nYr+xxx+k19pvD2LbnEsDOv/a\nFp0MBLkdL87UB5u3c/v67IuVOZhObOP/iIboWmINHMxc2J73QnfzSYK0sUuSqc74AgEJyUsVzIO62dtNBMUqXbM5nMaUlZfYP\nUnUotz1xi6atvrFtoju0/TExMLV3hoChNv0ByO5t4TdGemBa7RQ0W1cMz2CY3wJz\nuvlzlz0kIHqlcn2ZkzLQ7OvYIsqkl4lUKhThE6ko9xqUrtmmHjrmELmoDqHu0D4j\nKbTDvJ3x5T5cVvlnqDBisO2zKGRTDuQgGhyX6eyBFFtmvFw9hujMGHivku951DkA\nOm7nVz8ZhkKu8FkxIUAEaXqMCmirjeoWzdRmbEAeBrL01FTOfUQY3BcsB3qahND/\nDIBCPcECgYEA+L/IhmM6+6cDWiIXzXR28ZQ5uqRPF25sqN//+cNLOCnGx/QAxXag\n1TzVxNiBhZeS+BuwpYMXLaFekzIHFV3URpqjsHjmB2399zSPwFdX5Q5DRr8cgvIY\nLbTIA5FCEsyPRgLL4fPT1ZEbAkA5quMi8UzqzyJvRlE8quuwHob1qI0CgYEAyZ2H\n5npGb66/U1NjG06PiWUdgMD9fOGSOAaeNoRmNyo2t6zYtPQ3qtGIKPUtcTUASY4P\nHtkt3w2EB0z7XIe8eRn6aCc0vJ8zEBAz/6IQ9bCHSC/te+Jj5lXG7OVO8TxhmgtY\n7UmW8JzHMYaeXlGBIxoQ973Gnyb01hOJe7lit4kCgYEAjcljl5aATGlKc9nzD11P\nXyxKK6T0oDqFHU1xLwCuo3jMobTnq6aOzn06rFVsnqVjVKET84PhdlUA/44Ik5lE\nImqK21BObfW4SWxgdBZVN28F0hGlQs6UEZl2WPI3Y1fOYu29ITJGkPmBF6tcM5f8\nluZtAVxzaPVtS0/Et+HdrRECgYEAocq49EvLmnQxNT0FmzRAG5H5SwmUYlLic/Nb\no4Q8QqitoFgkz5Hr2jire7LE9MQDpwNJPwgpt4WxHeq5DFgg903RlSNhPrzCzXEz\nSUFVOtSeu186xN+4K29KY3DhGNXLvUK96i3T4uLtNuFA1Y+ygei5FRZF/hHVCLZE\n7fSnM4ECgYAdVC85PiYuwmU1FxHmlgXC97Wv4/idBBqRT/rDfc/tdB2XcAJz9Jzw\nk5QiFgGbfFHWqlMH1XZ5MSVJqnl7PYtOHKBi5xV2VIbbbC4givWzAU3YJcmO7YuA\ndCa5xXeck9DptcpNMSXL81DaeSjk3+5027IaK9F9pcSYKP9dBCzfeQ==\n-----END RSA PRIVATE KEY-----"
  }
}

$ cat << 'EOF' > check_gitlab.sh
#!/bin/bash

gitlab_pod=$1
port=${2:-30855}

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

log "Starting monitoring script for GitLab pod: ${gitlab_pod} on port: ${port}"

while true; do
  curl_result=$(kubectl -n $GITLAB_NAMESPACE exec ${gitlab_pod} -- curl -s localhost:${port} | grep "You are being")
  if [[ -z "${curl_result}" ]]; then
      log "HTTP not ready. Retrying in 10 seconds..."
      sleep 10
      continue
  fi

  log "GitLab is ready"
  break
done
EOF

# 进入 pod 执行恢复
$ kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE
gitlab-98c4b9f4-lcjwn   1/1     Running   0          9s
$ bash check_gitlab.sh gitlab-98c4b9f4-lcjwn
2024-11-01 16:02:47 - Starting monitoring script for GitLab pod: gitlab-7ff8d674bd-m65nz on port: 30855
command terminated with exit code 7
2024-11-01 16:12:59 - HTTP not ready. Retrying in 10 seconds...
command terminated with exit code 7
2024-11-01 16:13:09 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:19 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:29 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:40 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:50 - GitLab is ready

#############
# 用 gitlab 14 rails-secret 的数据替换 /etc/gitlab/gitlab-secrets.json 中对应字段。
# 注意：不要覆盖整个 YAML，只替换以下字段：
# - secret_key_base, otp_key_base
# - db_key_base
# - openid_connect_signing_key
# - ci_jwt_signing_key
# 替换为备份中的对应字段数据。
##############
$ kubectl exec -ti gitlab-98c4b9f4-lcjwn -n $GITLAB_NAMESPACE bash
vi /etc/gitlab/gitlab-secrets.json
exit
# 重启实例
$ kubectl delete po -n $GITLAB_NAMESPACE gitlab-98c4b9f4-lcjwn
# 重新进入 pod，确认 gitlab 启动成功
$ kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE
NAME                    READY   STATUS    RESTARTS   AGE
gitlab-98c4b9f4-lcjwn   1/1     Running   0          9s
$ bash check_gitlab.sh gitlab-98c4b9f4-lcjwn
2024-11-01 16:02:47 - Starting monitoring script for GitLab pod: gitlab-7ff8d674bd-m65nz on port: 30855
command terminated with exit code 7
2024-11-01 16:12:59 - HTTP not ready. Retrying in 10 seconds...
command terminated with exit code 7
2024-11-01 16:13:09 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:19 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:29 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:40 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:50 - GitLab is ready

$ kubectl -n $GITLAB_NAMESPACE exec -ti gitlab-98c4b9f4-lcjwn bash
# 设置 gitlab 为只读
$ gitlab-ctl stop puma
$ gitlab-ctl stop sidekiq
# 确认 puma 和 sidekiq 已停止
$ gitlab-ctl status

# 执行备份
$ ls /var/opt/gitlab/backups/
17302758xx_xxxx_xx_xx_14.0.12_gitlab_backup.tar
$ gitlab-backup restore BACKUP=17302758xx_xxxx_xx_xx_14.0.12
$ exit

# 重启 gitlab
$ kubectl -n $GITLAB_NAMESPACE delete po gitlab-98c4b9f4-lcjwn
# 检查恢复状态
$ kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE
NAME                    READY   STATUS    RESTARTS   AGE
gitlab-98c4b9f4-lcjwn   1/1     Running   0          9s
$ bash check_gitlab.sh gitlab-98c4b9f4-lcjwn
$ kubectl -n $GITLAB_NAMESPACE exec -ti gitlab-98c4b9f4-lcjwn -- gitlab-rake gitlab:check SANITIZE=true

升级 all-in-one 部署的 gitlab 至 17.8.1

使用 all-in-one 模式逐版本升级 gitlab 至 17.8.1。需依次替换升级路径中的 gitlab 镜像。

升级路径：14.0.12 → 14.3.6 → 14.9.5 → 14.10.5 → 15.0.5 → 15.4.6 → 15.11.13 → 16.3.8 → 16.7.9 → 16.11.10 → 17.3.6 → 17.5.5 → 17.8.1

升级 14.0.12 至 14.3.6

# 修改 deployment 中的 gitlab 镜像，镜像仓库地址根据实际地址修改，此处以 127.0.0.1 为例
$ kubectl -n $GITLAB_NAMESPACE set image deployment/gitlab gitlab=127.0.0.1/gitlab/gitlab-ce:14.3.6-ce.0
$ cat << 'EOF' > monitor_gitlab.sh
#!/bin/bash

gitlab_pod=$1
port=${2:-30855}

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

log "Starting monitoring script for GitLab pod: ${gitlab_pod} on port: ${port}"

while true; do
  curl_result=$(kubectl -n $GITLAB_NAMESPACE exec ${gitlab_pod} -- curl -s localhost:${port} | grep "You are being")
  if [[ -z "${curl_result}" ]]; then
      log "HTTP check not ready. Retrying in 10 seconds..."
      sleep 10
      continue
  fi

  migration_result=$(kubectl -n $GITLAB_NAMESPACE exec ${gitlab_pod} -- gitlab-psql -c "SELECT job_class_name, table_name, column_name, job_arguments FROM batched_background_migrations WHERE status NOT IN(3, 6);" | grep "(0 rows)")

  if [[ -z "${migration_result}" ]]; then
      log "Database migration is running. Retrying in 10 seconds..."
      sleep 10
      continue
  fi

  log "GitLab is ready, all migrations are done"
  break
done
EOF

$ kubectl -n $GITLAB_NAMESPACE get po -l deploy=gitlab
NAME                    READY   STATUS    RESTARTS   AGE
gitlab-98c4b9f4-lcjwn   1/1     Running   0          9s
# 等待 pod 启动后，检查程序是否启动成功
$ bash monitor_gitlab.sh gitlab-98c4b9f4-lcjwn
2024-11-01 16:02:47 - Starting monitoring script for GitLab pod: gitlab-7ff8d674bd-m65nz on port: 30855
command terminated with exit code 7
2024-11-01 16:12:59 - HTTP not ready. Retrying in 10 seconds...
command terminated with exit code 7
2024-11-01 16:13:09 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:19 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:29 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:40 - HTTP not ready. Retrying in 10 seconds...
2024-11-01 16:13:50 - GitLab is ready, all migrations are done

升级 14.3.6 至 14.9.5：替换镜像标签为 14.9.5-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 14.9.5 至 14.10.5：替换镜像标签为 14.10.5-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 14.10.5 至 15.0.5：替换镜像标签为 15.0.5-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 15.0.5 至 15.4.6：替换镜像标签为 15.4.6-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 15.4.6 至 15.11.13：替换镜像标签为 15.11.13-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 15.11.13 至 16.3.8：替换镜像标签为 16.3.8-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 16.3.8 至 16.7.9：替换镜像标签为 16.7.9-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 16.7.9 至 16.11.10：替换镜像标签为 16.11.10-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 16.11.10 至 17.3.6：替换镜像标签为 17.3.6-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 17.3.6 至 17.5.5：替换镜像标签为 17.5.5-ce.0，其他操作同“14.0.12 到 14.3.6”。
升级 17.5.5 至 17.8.1：替换镜像标签为 17.8.1-ce.0，其他操作同“14.0.12 到 14.3.6”。

all-in-one 17.8.1 备份

在 17.8.1 部署 pod 中执行备份。

kubectl -n $GITLAB_NAMESPACE get po -l deploy=gitlab
NAME                    READY   STATUS    RESTARTS   AGE
gitlab-98c4b9f4-lcjwn   1/1     Running   0          9s
kubectl -n $GITLAB_NAMESPACE exec -ti gitlab-98c4b9f4-lcjwn bash
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
gitlab-ctl status
gitlab-rake gitlab:backup:create
ls /var/opt/gitlab/backups/
17302758xx_xxxx_xx_xx_14.0.12_gitlab_backup.tar
chmod 777 /var/opt/gitlab/backups/17302758xx_xxxx_xx_xx_14.0.12_gitlab_backup.tar
exit

# 备份 gitlab 17.8.1 部署的 gitlab-secrets
kubectl -n $GITLAB_NAMESPACE cp gitlab-98c4b9f4-lcjwn:/etc/gitlab/gitlab-secrets.json ./gitlab-secrets.json
yq -P '{"production": .gitlab_rails}' gitlab-secrets.json -o yaml > gitlab-secrets.yaml

恢复 all-in-one 备份数据至平台部署的 17.8.z

使用 operator 在平台部署 gitlab 17.8.z 并启用 toolbox（注意需与旧实例处于同一命名空间），然后将数据恢复到 operator 部署的 gitlab 17.8.z（将平台部署的 gitlab 名称设置为环境变量 NEW_GITLAB_NAME）。

在 gitlab 17.8.z 中添加以下值以启用 gitlab 17.8.z toolbox：

spec:
  helmValues:
    gitlab:
      toolbox:
        enabled: true

TIP

恢复过程中可能出现数据库相关错误，must be owner 和 does not exist 是预期行为，详情见 (https://gitlab.com/gitlab-org/gitlab/-/issues/266988)

export NEW_GITLAB_NAME=<平台 gitlab 实例名称>

# 用平台部署 gitlab 的 rails-secret 替换 operator 部署的 rails-secret
kubectl -n $GITLAB_NAMESPACE get secrets -l release=${NEW_GITLAB_NAME}| grep rails-secret
gitlab-rails-secret                                             Opaque               1      2d2h

kubectl -n $GITLAB_NAMESPACE delete secret ${NEW_GITLAB_NAME}-rails-secret
kubectl -n $GITLAB_NAMESPACE create secret generic ${NEW_GITLAB_NAME}-rails-secret --from-file=secrets.yml=gitlab-secrets.yaml

# 重启 pod 使修改生效
kubectl delete pod -n $GITLAB_NAMESPACE -l release=$NEW_GITLAB_NAME,app=webservice
kubectl delete pod -n $GITLAB_NAMESPACE -l release=$NEW_GITLAB_NAME,app=sidekiq
kubectl delete pod -n $GITLAB_NAMESPACE -l release=$NEW_GITLAB_NAME,app=toolbox
# 等待 pod 就绪后继续下一步
kubectl get po -n $GITLAB_NAMESPACE -l release=$NEW_GITLAB_NAME

# 关闭 17.8.z gitlab 的外部访问
kubectl patch deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl patch deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl scale deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --replicas 0
kubectl scale deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --replicas 0

# 修改 toolbox，挂载备份 pvc 到 toolbox
kubectl patch deploy ${NEW_GITLAB_NAME}-toolbox -n $GITLAB_NAMESPACE -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl patch deploy ${NEW_GITLAB_NAME}-toolbox -n $GITLAB_NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/1", "value": {"name": "toolbox-tmp", "persistentVolumeClaim": {"claimName": "backup-pvc"}}}]'

# 进入 pod 恢复数据
kubectl get po -n $GITLAB_NAMESPACE -l release=$NEW_GITLAB_NAME,app=toolbox
NAME READY STATUS RESTARTS AGE
gitlab-toolbox-58b7db895c-8k99q 1/1 Running 0 119m
kubectl exec -ti -n $GITLAB_NAMESPACE gitlab-toolbox-58b7db895c-8k99q bash
ls /srv/gitlab/tmp/backups/
17302758xx_xxxx_xx_xx_17.8.1_gitlab_backup.tar

# 注意此步骤会删除备份文件，可提前备份：cp /srv/gitlab/tmp/backups/17302758xx_xxxx_xx_xx_17.8.1_gitlab_backup.tar /srv/gitlab/tmp/backups/17302758xx_xxxx_xx_xx_17.8.1_gitlab_backup2.tar

backup-utility --restore -f file:///srv/gitlab/tmp/backups/17302758xx_xxxx_xx_xx_17.8.1_gitlab_backup.tar
exit

kubectl patch deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/annotations/skip-sync"}]'
kubectl patch deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/annotations/skip-sync"}]'
kubectl patch deploy ${NEW_GITLAB_NAME}-toolbox -n $GITLAB_NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/annotations/skip-sync"}]'

kubectl scale deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --replicas 1
kubectl scale deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --replicas 1

验证升级后数据与升级前数据是否一致。
数据验证完成后，手动清理旧实例。

常见问题

执行备份数据时，task-runner 容器被杀死，需要给 task-runner 容器添加请求和限制资源（至少 2c4g），然后重启 task-runner 并重新备份。
恢复备份到平台部署的 17.8.z 时，toolbox 容器被杀死，需要给 toolbox 容器添加请求和限制资源（至少 2c4g）。
升级备份不包含外部附件图片和头像，升级后页面无法显示附件图片和上传头像。可将 17.8.z PVC 替换为 14.0.12 的镜像存储 PVC。

从 GitLab 14.0.12 迁移数据至 17.8.z#

#目录

#流程概述

#数据迁移流程

#操作步骤

#前置准备

#数据迁移步骤

#备份平台部署的 GitLab 14.0.12

#恢复备份数据至 all-in-one 部署的 14.0.12

#升级 all-in-one 部署的 gitlab 至 17.8.1

#all-in-one 17.8.1 备份

#恢复 all-in-one 备份数据至平台部署的 17.8.z

#常见问题