多种方式为极狐 gitlab 安装 runner

多种方式为极狐 gitlab 安装 runner

1. 环境信息

1.1 主机信息

主机	用途
10.10.10.60	极狐gitlab v14.10.0

• 域名 gitlab.example.com，指向 10.10.10.60，这里使用设置本机 hosts 的方式解析，如果有条件的话使用本地 dns 服务器设置解析
• 开启了 san 证书的 https，ca.pem 为 ca 根证书

2. linux 上安装

参考官方文档：Install GitLab Runner | GitLab

系统：centos 7.9 x64

2.1 二进制包方式

runner 发布地址：https://gitlab.com/gitlab-org/gitlab-runner/-/tags

根据需求下载对应版本，这里下载 v14.8.0 版本：

wget https://gitlab.com/gitlab-org/gitlab-runner/-/releases/v14.8.0/downloads/binaries/gitlab-runner-linux-amd64
mv gitlab-runner-linux-amd64 /usr/local/bin/gitlab-runner
chmod +x /usr/local/bin/gitlab-runner

添加用户

useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

安装启动服务

gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
gitlab-runner start

注册 runner，参考官方文档：Registering runners | GitLab

$ gitlab-runner register --tls-ca-file=/etc/gitlab-runner/ca.pem 
Runtime platform                                    arch=amd64 os=linux pid=1594 revision=565b6c0b version=14.8.0
Running in system-mode.                            
                                                   
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.example.com 
Enter the registration token:
_xYaYy6WPUqsMm2wVLsV
Enter a description for the runner:
[localhost.localdomain]: docker runner
Enter tags for the runner (comma-separated):
docker
Enter optional maintenance note for the runner:
docker runner
Registering runner... succeeded                     runner=_xYaYy6W
Enter an executor: parallels, shell, docker+machine, virtualbox, docker-ssh+machine, kubernetes, custom, docker, docker-ssh, ssh:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 

# 重启 gitlab runner
systemctl restart gitlab-runner

2.2 rpm 或者 deb 包方式

方法和二进制方式类似，只需下载安装包时下载对应的 rpm 和 deb 即可，这里就不演示了。

更新方式：

# 下载新版本的 gitlab runner rpm 包
rpm -Uvh gitlab-runner_<arch>.rpm

随后重启 gitlab-runner 即可

3. docker 方式

3.1 创建配置目录

mkdir -p /srv/gitlab-runner/config
cp ca.pem /srv/gitlab-runner/config

3.2 启动 runner

docker run -d --name gitlab-runner --restart always \
     --env TZ=Asia/Shanghai \
     --add-host=gitlab.example.com:10.10.10.60 \
     -v /srv/gitlab-runner/config:/etc/gitlab-runner \
     -v /var/run/docker.sock:/var/run/docker.sock \
     gitlab/gitlab-runner:v14.7.0

• mac 本机配置了 hosts，那么 hosts 配置对 docker 容器里面的所有容器都适用，但是在 linux 的时候就不适用了，所以这里使用 --add-host 添加容器里的 hosts 解析

3.3 注册 runner

docker run --rm -it --add-host=gitlab.example.com:10.10.10.60 -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner:v14.7.0 register --tls-ca-file=/etc/gitlab-runner/ca.pem

docker restart gitlab-runner

• 根据提示填写 gitlab 地址，以及 token 等信息

查看配置文件

$ cat /srv/gitlab-runner/config/config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker runner"
  url = "https://gitlab.example.com"
  token = "CGcmbaHn1fHGFX7ViHuF"
  tls-ca-file = "/etc/gitlab-runner/ca.pem"
  executor = "docker"
  environment = ["GO111MODULE=on", "GOPROXY=https://goproxy.cn,direct"]
  pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    pull_policy = ["always", "if-not-present"]
    extra_hosts = ["gitlab.example.com:10.10.10.60"]
    tls_verify = false
    image = "busybox:1.28.4"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

注意：实际环境中需要优化配置，修改配置后重启容器生效 docker restart gitlab-runner

更新方式：

docker pull gitlab/gitlab-runner:latest

docker stop gitlab-runner && docker rm gitlab-runner

docker run -d --name gitlab-runner --restart always \
     --env TZ=Asia/Shanghai \
     --add-host=gitlab.example.com:10.10.10.60 \
     -v /srv/gitlab-runner/config:/etc/gitlab-runner \
     -v /var/run/docker.sock:/var/run/docker.sock \
     gitlab/gitlab-runner:latest

4. windows 上安装

系统：windows 10 64位

4.1 下载程序

runner 发布地址：https://gitlab.com/gitlab-org/gitlab-runner/-/tags

下载：https://gitlab.com/gitlab-org/gitlab-runner/-/releases/v14.7.0/downloads/binaries/gitlab-runner-windows-amd64.zip

4.2 注册 runner

解压程序包到 C:\，使用 PowerShell 执行程序（请勿使用 PowerShell ISE 执行，会报错）

4.3 安装 runner 为 windows 服务

首先以管理员方式允许 PowerShell

4.4 测试 windows runner

添加 .gitlab-ci.yml

stages:
  - build

variables:
  projectname: "p1"

build:
  stage: build
  retry: 2
  script:
    - pwd
    - get-host
  tags:
    - "windows"

运行 runner 失败

Running with gitlab-runner 14.7.0 (98daeee0)
  on windows runner xhyz8oiW
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
ERROR: Job failed (system failure): prepare environment: failed to start process: exec: "pwsh": executable file not found in %PATH%. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

解决方法：gitlab-runner: prepare environment failed to start process pwsh in windows - Stack Overflow

再次运行成功：

Running with gitlab-runner 14.7.0 (98daeee0)
  on windows runner xhyz8oiW
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:01
Running on P_FAFULI-NB1...
Getting source from Git repository
Fetching changes with git depth set to 20...
Initialized empty Git repository in C:/gitlab-runner-windows-amd64/builds/xhyz8oiW/0/g1/p1/.git/
Created fresh repository.
Checking out 88ac867c as main...
git-lfs/2.13.3 (GitHub; windows amd64; go 1.16.2; git a5e65851)
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
$ pwd
Path                                                  
----                                                  
C:\gitlab-runner-windows-amd64\builds\xhyz8oiW\0\g1\p1
$ get-host
Name             : ConsoleHost
Version          : 5.1.19041.1320
InstanceId       : 289c4b2c-386e-452a-80d3-dbd947609adc
UI               : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture   : zh-CN
CurrentUICulture : zh-CN
PrivateData      : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled  : True
IsRunspacePushed : False
Runspace         : System.Management.Automation.Runspaces.LocalRunspace
Cleaning up project directory and file based variables
00:00
Job succeeded

4.5 更新 runner

cd C:\gitlab-runner-windows-amd64
.\gitlab-runner-windows-amd64.exe stop

下载新版本覆盖，然后重新启动

cd C:\gitlab-runner-windows-amd64
.\gitlab-runner-windows-amd64.exe start

4.6 卸载 runner

cd C:\gitlab-runner-windows-amd64
.\gitlab-runner-windows-amd64.exe stop
.\gitlab-runner-windows-amd64.exe uninstall
# 然后删除目录 gitlab-runner-windows-amd64 即可

5. Kubernetes 上安装

官方文档：GitLab Runner Helm Chart | GitLab

5.1 安装 docker

参考：Get Docker | Docker Documentation 或者

echo "
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
sysctl -p

curl https://releases.rancher.com/install-docker/19.03.sh | sh
systemctl enable docker

设置国内源

vi /etc/docker/daemon.json
{
    "registry-mirrors": ["https://hccwwfjl.mirror.aliyuncs.com"]
}

systemctl restart docker

• 其他源：http://hub-mirror.c.163.com，建议用这个 163 的

5.2 安装 k3s

参考：k8s 测试环境搭建（k3s） - leffss - 博客园 (cnblogs.com)

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_VERSION=v1.20.7+k3s1 INSTALL_K3S_MIRROR=cn sh -s - --docker

$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-02-22 13:38:40 CST; 59min ago
     Docs: https://k3s.io
  Process: 2247 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 2245 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 2250 (k3s-server)
    Tasks: 18
   Memory: 638.6M
   CGroup: /system.slice/k3s.service
           └─2250 /usr/local/bin/k3s server

$ kubectl get node
NAME                    STATUS   ROLES                  AGE   VERSION
localhost.localdomain   Ready    control-plane,master   60m   v1.20.7+k3s1

$ kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   helm-install-traefik-g4xsp                0/1     Completed   0          59m
kube-system   metrics-server-86cbb8457f-8wghh           1/1     Running     0          59m
kube-system   local-path-provisioner-5ff76fc89d-r87zg   1/1     Running     0          59m
kube-system   svclb-traefik-24nj7                       2/2     Running     0          59m
kube-system   coredns-854c77959c-nm28h                  1/1     Running     0          59m
kube-system   traefik-6f9cbd9bd4-lm4xf                  1/1     Running     0          59m

5.3 安装 helm v3

安装 helm v3（v3.3.1 或者更新），参考官方安装文档：Helm | Installing Helm

这里使用手动下载安装方式：

# 发布地址：https://github.com/helm/helm/releases
$ curl -O https://get.helm.sh/helm-v3.8.0-linux-amd64.tar.gz
$ tar zxvf helm-v3.8.0-linux-amd64.tar.gz
$ mv linux-amd64/helm /usr/local/bin/helm
$ chmod +x /usr/local/bin/helm
$ helm version
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

5.4 添加 gitlab repo

helm repo add gitlab https://charts.gitlab.io/

# 查看完整的镜像仓库列表
helm search repo -l gitlab/gitlab-runner

5.5 安装 runner chart

首先准备配置文件 values.yaml ，官方模板文件：values.yaml · main · GitLab.org / charts / GitLab Runner · GitLab

vi values.yaml
imagePullPolicy: IfNotPresent
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
certsSecretName: gitlab.example.com.crt
# gitlab 地址
gitlabUrl: https://gitlab.example.com
# runner 注册 token
runnerRegistrationToken: "_xYaYy6WPUqsMm2wVLsV"
runners:
  # runner 的配置文件
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "busybox:1.28.4"
rbac:
  create: true

其中 certsSecretName 保存 gitlab 的 https 证书，名称格式为：.crt ，需要提前在 k8s 中创建

kubectl create secret generic gitlab.example.com.crt --from-file=gitlab.example.com.crt

开始创建

# 格式：helm install --namespace <NAMESPACE> gitlab-runner -f <CONFIG_VALUES_FILE> gitlab/gitlab-runner

helm install --version 0.37.2 gitlab-runner -f values.yaml gitlab/gitlab-runner

• --version 指定 chart 版本，0.37.2 对应 v14.7.0 的 runner

第一次运行会报错

Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

原因是：helm v3 版本不再需要 Tiller，而是直接访问 ApiServer 来与 k8s 交互，通过环境变量 KUBECONFIG 来读取存有 ApiServer 的地址与 token 的配置文件地址，默认地址为 ~/.kube/config，但是 k3s 的 KUBECONFIG 默认为 /etc/rancher/k3s/k3s.yaml

解决方法：手动配置 KUBECONFIG 环境变量

1）临时解决：

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

2）永久解决：

$ vi /etc/profile # 添加
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

$ source /etc/profile

再次运行

$ helm install --version 0.37.2 gitlab-runner -f values.yaml gitlab/gitlab-runner

NAME: gitlab-runner
LAST DEPLOYED: Thu Feb 24 16:52:34 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Your GitLab Runner should now be registered against the GitLab instance reachable at: "https://gitlab.example.com"

Runner namespace "default" was found in runners.config template.

查看 runner 的 pod 日志，会发现报错：

$ kubectl logs gitlab-runner-gitlab-runner-76fdff5f4f-mf66w
...
ERROR: Registering runner... failed                 runner=_xYaYy6W status=couldn't execute POST against https://gitlab.example.com/api/v4/runners: Post "https://gitlab.example.com/api/v4/runners": dial tcp: lookup gitlab.example.com on 10.43.0.10:53: no such host
PANIC: Failed to register the runner. You may be having network problems. 

原因是 runner 的 pod 无法解析 gitlab.example.com，解决方法如下：

在集群 coredns 中添加 hosts 解析

$ kubectl -n kube-system edit configmaps coredns
...
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          10.10.10.60 gitlab.example.com
          ttl 60    
          reload 15s 
          fallthrough
        }               
...

# 删除 coredns pod 生效
$ kubectl get pod -n kube-system
NAME                                      READY   STATUS      RESTARTS   AGE
metrics-server-86cbb8457f-hqt9p           1/1     Running     0          5h6m
local-path-provisioner-5ff76fc89d-cz9hj   1/1     Running     0          5h6m
helm-install-traefik-lmt7r                0/1     Completed   0          5h6m
coredns-854c77959c-w4t7j                  1/1     Running     0          5h6m
svclb-traefik-fp7f9                       2/2     Running     0          5h5m
traefik-6f9cbd9bd4-l7h45                  1/1     Running     0          5h5m

$ kubectl -n kube-system delete pod coredns-854c77959c-w4t7j
pod "coredns-854c77959c-nm28h" deleted

再次查看，runner 已经正常运行

$ kubectl get pod
NAME                                           READY   STATUS    RESTARTS   AGE
gitlab-runner-gitlab-runner-76fdff5f4f-mf66w   1/1     Running   1          4m50s

注册成功

测试任务也成功

Running with gitlab-runner 14.7.0 (98daeee0)
  on gitlab-runner-gitlab-runner-76fdff5f4f-mf66w c3qaY9MY
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: default
Using Kubernetes executor with image busybox:1.28.4 ...
Using attach strategy to execute scripts...
Preparing environment
00:18
Waiting for pod default/runner-c3qay9my-project-2-concurrent-0sdnb8 to be running, status is Pending
Waiting for pod default/runner-c3qay9my-project-2-concurrent-0sdnb8 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-c3qay9my-project-2-concurrent-0sdnb8 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-c3qay9my-project-2-concurrent-0sdnb8 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-c3qay9my-project-2-concurrent-0sdnb8 to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-c3qay9my-project-2-concurrent-0sdnb8 to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-c3qay9my-project-2-concurrent-0sdnb8 via gitlab-runner-gitlab-runner-76fdff5f4f-mf66w...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out 1c6e29cb as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlay               52403200   3039852  49363348   6% /
tmpfs                    65536         0     65536   0% /dev
tmpfs                  8185140         0   8185140   0% /sys/fs/cgroup
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /scripts-2-6
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /logs-2-6
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /builds
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /dev/termination-log
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /etc/resolv.conf
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /etc/hostname
/dev/mapper/centos-root
                      52403200   3039852  49363348   6% /etc/hosts
shm                      65536         0     65536   0% /dev/shm
tmpfs                  8185140        12   8185128   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                  8185140         0   8185140   0% /proc/acpi
tmpfs                    65536         0     65536   0% /proc/kcore
tmpfs                    65536         0     65536   0% /proc/keys
tmpfs                    65536         0     65536   0% /proc/timer_list
tmpfs                    65536         0     65536   0% /proc/sched_debug
tmpfs                  8185140         0   8185140   0% /proc/scsi
tmpfs                  8185140         0   8185140   0% /sys/firmware
$ echo ${projectname}
p1
$ date
Thu Feb 24 09:00:31 UTC 2022
Cleaning up project directory and file based variables
00:01
Job succeeded