GitLab CI/CD实践记录

 

 

1. GitLab Runner

参考：《Configuring GitLab Runners》

In GitLab CI/CD, Runners run the code defined in .gitlab-ci.yml. A GitLab Runner is a lightweight, highly-scalable agent that picks up a CI job through the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance.

GitLab Runner的有多种形式：Repositories、Binaries、Docker service。参考：《Create a group Runner》

1.1 安装Git Lab Runner

1.1.1 Ubuntu中安装gitlab-runner

参考：《Install GitLab Runner manually on GNU/Linux》

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash sudo apt-get install gitlab-runner

或者下载deb安装包：https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb。

sudo dpkg -i gitlab-runner_amd64.deb

 

安装好Runner之后，还需要将Runner注册到Gitlab。

Ubuntu下注册Runner为：GNU/Linux。

sudo gitlab-runner register

一行命令注册：

docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --executor "docker" \
  --docker-image alpine:latest \
  --url "https://gitlab.com/" \
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --description "docker-runner" \
  --tag-list "docker,aws" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

1.1.3 docker中安装gitlab-runner

参考：《Run GitLab Runner in a container》

1.2 Git Lab Runner使用

docker中对Runner进行注册有两种方法：交互式注册和一行命令注册。

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --executor "docker" \
  --docker-image alpine:latest \
  --description "docker-runner" \
  --tag-list "docker,aws" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

这句话的意思是：注册一个executor为docker，docker image为alpine:latest的gitlab-runner，到https://gitlab.com/，其token为PROJECT_REGISTRATION_TOKEN。

并且此gitlab-runner的tag为docker或aws，并设置interactive、locked、access等属性。

上述参数url和registration-token从Settings->CI/CD->Runners中获取：

另外--docker-image为指定的Runner的Docker image文件；--tag-list为Runner的tag列表。

查看所有注册的GitLab Runner：

sudo gitlab-runner list

去注册选定的GitLab Runner：

sudo gitlab-runner unregister --token g_L6R5T1RZDxbm1ttbx8 --url http://192.168.70.8/

 运行所有注册的Runner：

sudo gitlab-runner run

1.3 Git Lab Runner的config.toml配置

config.toml默认位置在/etc/gitlab-runner/config.toml，详细配置参考《Advanced configuration》。

Pipelines简要介绍

Pipelines组成

参考：《CI/CD pipelines》

Pipelines are the top-level component of continuous integration, delivery, and deployment.

Pipelines comprise:

Jobs, which define what to do. For example, jobs that compile or test code.

Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

Pipelines即流水线是持续集成、持续发布、持续部署的最顶层组件，由Stages和Jobs组成。

Jobs是真正执行单元，定义了做什么。Stags是对Jobs串接，定义了什么时候执行Jobs。

Pipelines调度

参考：《Pipeline schedules》

Pipelines的调度按触发模式可以分为：自动触发和手动触发，自动触发又分为定时触发和Merge请求触发Pipelines for Merge Requests。

默认通过commit触发Pipeline，还可以被Merge Requests触发。

比如下面build job 1可以被Merge Requests或者master上的分支触发。

build job 1:
  stage: build
  tags:
  - mytag
  script:
  only:
    - merge_requests
    - master

还可以通过except关键字，排除触发条件。

另外还可以打开Settings->CI/CD->Pipeline triggers，在其他pipeline中触发。

输入Decsription，然后Add trigger，即可增加Pipeline trigger。

可以命令行触发，也可以在.gitlab-ci.yml中其他job中触发。

trigger_build:
  stage: deploy
  script:
    - "curl -X POST -F token=<Pipeline trigger token> -F ref=<branch or tag> http://<gitlab server>/api/v4/projects/<project id>/trigger/pipeline"

在Pipeline之间传递artifacts：

build_submodule:
  image: debian
  stage: test
  script:
- curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"
    - unzip artifacts.zip
  only:
    - tags

 

如果要在curl中传递参数可以通过：

curl --request POST \
  --form token=TOKEN \
  --form ref=master \
  --form "variables[UPLOAD_TO_S3]=true" \
  https://gitlab.example.com/api/v4/projects/9/trigger/pipeline

gitlab-ci.yml配置

stages配置 stage配置 .pre和.post

参考：《stages》《stage》《.pre and .post》

stages is used to define stages that contain jobs and is defined globally for the pipeline.

Jobs of the same stage are run in parallel.Jobs of the next stage are run after the jobs from the previous stage complete successfully.

stage is defined per-job and relies on stages which is defined globally. It allows to group jobs into different stages, and jobs of the same stage are executed in parallel (subject to certain conditions).

 

The following stages are available to every pipeline:

.pre, which is guaranteed to always be the first stage in a pipeline.

.post, which is guaranteed to always be the last stage in a pipeline.

User-defined stages are executed after .pre and before .post.

The order of .pre and .post can’t be changed, even if defined out of order in .gitlab-ci.yml.

.pre和.post在所有stage的最前和最后执行，而不受pipelines中定义的顺序。

一个示例如下：

stages:
  - build
  - test
  - deploy

job 0:
  stage: .pre
  script: make something useful before build stage

job 1:
  stage: build
  script: make build dependencies

job 2:
  stage: build
  script: make build artifacts

job 3:
  stage: test
  script: make test

job 4:
  stage: deploy
  script: make deploy

job 5:
  stage: .post
  script: make something useful at the end of pipeline

这个Pilelines中定义了5个stages，分别是.pre、build、test、deploy、.post；定义了6个jobs：分别是job 0、job 1、job 2、job 3、job 4、job 5。

所以此Pipellines的执行顺序是.pre-[job 0]->build-[job 1/job 2]->test-[job 3]->deploy-[job 4]->.post[job 5]，其中job1和job2并行执行。

include配置

参考：《include》

include requires the external YAML file to have the extensions .yml or .yaml, otherwise the external file won’t be included.

include包含如下类型：

Method	Description
local	Include a file from the local project repository.
file	Include a file from a different project repository.
remote	Include a file from a remote URL. Must be publicly accessible.
template	Include templates which are provided by GitLab.

local示例如下：

include:
  - local: '/templates/.gitlab-ci-template.yml'

include: '.gitlab-ci-production.yml'

file示例如下：

include:
  - project: 'my-group/my-project'
    ref: master
    file: '/templates/.gitlab-ci-template.yml'

  - project: 'my-group/my-project'
    ref: v1.0.0
    file: '/templates/.gitlab-ci-template.yml'

  - project: 'my-group/my-project'
    ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA
    file: '/templates/.gitlab-ci-template.yml'

remote示例如下：

include:
  - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml'

template示例如下：

include:
  - template: Android-Fastlane.gitlab-ci.yml
  - template: Auto-DevOps.gitlab-ci.yml

script before_script和after_script

参考：《script》《before_scrip and after_script》

script is the only required keyword that a job needs. It’s a shell script which is executed by the Runner. 

before_script is used to define a command that should be run before each job, including deploy jobs, but after the restoration of any artifacts. This must be an array.

Scripts specified in before_script are concatenated with any scripts specified in the main script, and executed together in a single shell.

after_script is used to define the command that will be run after each job, including failed ones. This must be an array.

Scripts specified in after_script are executed in a new shell, separate from any before_script or script scripts.

before_script是在script运行之前，恢复artifacts之后运行，但是和script在一个shell中运行。

after_script单独在一个shell中运行，并且在scirpt之后。

image services

参考：《image》《service》

image用于指定Docker image，service用于指定连接到image指定Docker image的service Docker image。

The image keyword is the name of the Docker image the Docker executor will run to perform the CI tasks.
The services keyword defines just another Docker image that is run during your job and is linked to the Docker image that the image keyword defines. This allows you to access the service image during build time.

image可配置选项：

Setting	Required	GitLab version	Description
name	yes, when used with any other option	9.4	Full name of the image that should be used. It should contain the Registry part if needed.
entrypoint	no	9.4	Command or script that should be executed as the container’s entrypoint. It will be translated to Docker’s --entrypoint option while creating the container. The syntax is similar to Dockerfile’s ENTRYPOINT directive, where each shell token is a separate string in the array.

service可配置选项：

Setting	Required	GitLab version	Description
name	yes, when used with any other option	9.4	Full name of the image that should be used. It should contain the Registry part if needed.
entrypoint	no	9.4	Command or script that should be executed as the container’s entrypoint. It will be translated to Docker’s --entrypoint option while creating the container. The syntax is similar to Dockerfile’s ENTRYPOINT directive, where each shell token is a separate string in the array.
command	no	9.4	Command or script that should be used as the container’s command. It will be translated to arguments passed to Docker after the image’s name. The syntax is similar to Dockerfile’s CMD directive, where each shell token is a separate string in the array.
alias	no	9.4	Additional alias that can be used to access the service from the job’s container. Read Accessing the services for more information.

示例：

default:
  image:
    name: ruby:2.6
    entrypoint: ["/bin/bash"]

  services:
    - name: my-postgres:11.7
      alias: db-postgres
      entrypoint: ["/usr/local/bin/db-postgres"]
      command: ["start"]

  before_script:
    - bundle install

test:
  script:
    - bundle exec rake spec

only/except

参考：《only/except (basic)》

only and except are two parameters that set a job policy to limit when jobs are created:

1. only defines the names of branches and tags for which the job will run.
2. except defines the names of branches and tags for which the job will not run.

Value	Description
branches	When the Git reference for a pipeline is a branch.
tags	When the Git reference for a pipeline is a tag.
api	For pipelines triggered by the pipelines API.
external	When using CI services other than GitLab.
pipelines	For multi-project pipelines created by using the API with CI_JOB_TOKEN.
pushes	For pipelines triggered by a git push event, including for branches and tags.
schedules	For scheduled pipelines.
triggers	For pipelines created by using a trigger token.
web	For pipelines created by using Run pipeline button in the GitLab UI, from the project’s CI/CD > Pipelines section.
merge_requests	For pipelines created when a merge request is created or updated. Enables merge request pipelines, merged results pipelines, and merge trains.
external_pull_requests	When an external pull request on GitHub is created or updated (See Pipelines for external pull requests).
chat	For pipelines created by using a GitLab ChatOps command.

示例1：

job:
  # use regexp
  only:
    - /^issue-.*$/---仅执行issues-开头的分支、tags等。
  # use special keyword
  except:
    - branches----排除所有的branches

示例2：

job:
  only:
    - branches@gitlab-org/gitlab-----------在gitlab-org/gitlab的所有分支上运行。
  except:
    - master@gitlab-org/gitlab------------但是要排除master分支。
    - /^release/.*$/@gitlab-org/gitlab----以及release/xxx开头。

参考：《only/except(advanced)》

If you use multiple keys under only or except, the keys will be evaluated as a single conjoined expression. That is:

• only: means “include this job if all of the conditions match”.
• except: means “exclude this job if any of the conditions match”.

With only, individual keys are logically joined by an AND。

示例1：

test:
  script: npm run test
  only:----下面refs、variables、kubernetes是与的关系，也即必须都满足才会执行job；refs内部是或的而关系。
    refs:
      - master
      - schedules
    variables:
      - $CI_COMMIT_MESSAGE =~ /run-end-to-end-tests/
    kubernetes: active

示例2：

test:
  script: npm run test
  except:--------下面refs、changes是或的关系，也即只要满足一个条件都不会执行job。
    refs:
      - master
    changes:
      - "README.md"

rules

参考：《rules》

The rules keyword can be used to include or exclude jobs in pipelines.

Rules are evaluated in order until the first match.

When matched, the job is either included or excluded from the pipeline, depending on the configuration.

Clause	Description
if	Add or exclude jobs from a pipeline by evaluating an if statement.
changes	Add or exclude jobs from a pipeline based on what files are changed.
exists	Add or exclude jobs from a pipeline based on the presence of specific files.

Rules are evaluated in order until a match is found.

If a match is found, the attributes are checked to see if the job should be added to the pipeline.

If no attributes are defined, the defaults are:

• when: on_success
• allow_failure: false

The job is added to the pipeline:

• If a rule matches and has when: on_success, when: delayed or when: always.
• If no rules match, but the last clause is when: on_success, when: delayed or when: always (with no rule).

The job is not added to the pipeline:

• If no rules match, and there is no standalone when: on_success, when: delayed or when: always.
• If a rule matches, and has when: never as the attribute.

参考：《when》

when is used to implement jobs that are run in case of failure or despite the failure.

when can be set to one of the following values:

1. on_success - execute job only when all jobs from prior stages succeed (or are considered succeeding because they are marked allow_failure). This is the default.
2. on_failure - execute job only when at least one job from prior stages fails.
3. always - execute job regardless of the status of jobs from prior stages.
4. manual - execute job manually (added in GitLab 8.10).
5. delayed - execute job after a certain period (added in GitLab 11.14).

参考：《allow_failure》

allow_failure allows a job to fail without impacting the rest of the CI suite.

The default value is false, except for manual jobs using the when: manual syntax, unless using rules: syntax, where all jobs default to false, including when: manual jobs.

When enabled and the job fails, the job will show an orange warning in the UI.

However, the logical flow of the pipeline will consider the job a success/passed, and is not blocked.

job:
  script: "echo Hello, Rules!"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never-----------所有Merge Request触发Pipeline，不运行此job。
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: never-----------所有Schedule触发Pipeline，不运行此job。
    - when: on_success------所有其他情况，在前面所有job都成功情况下，运行此job。

参考：《rules:if》

$CI_PIPELINE_SOURCE的取值可能情况包括：

Value	Description
push	For pipelines triggered by a git push event, including for branches and tags.
web	For pipelines created by using Run pipeline button in the GitLab UI, from the project’s CI/CD > Pipelines section.
trigger	For pipelines created by using a trigger token.
schedule	For scheduled pipelines.
api	For pipelines triggered by the pipelines API.
external	When using CI services other than GitLab.
pipeline	For multi-project pipelines created by using the API with CI_JOB_TOKEN.
chat	For pipelines created by using a GitLab ChatOps command.
webide	For pipelines created by using the WebIDE.
merge_request_event	For pipelines created when a merge request is created or updated. Required to enable merge request pipelines, merged results pipelines, and merge trains.
external_pull_request_event	When an external pull request on GitHub is created or updated. See Pipelines for external pull requests.
parent_pipeline	For pipelines triggered by a parent/child pipeline with rules, use this in the child pipeline configuration so that it can be triggered by the parent pipeline.

Other commonly used variables for if clauses:

• if: $CI_COMMIT_TAG: If changes are pushed for a tag.
• if: $CI_COMMIT_BRANCH: If changes are pushed to any branch.
• if: '$CI_COMMIT_BRANCH == "master"': If changes are pushed to master.
• if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH': If changes are pushed to the default branch (usually master). Useful if reusing the same configuration in multiple projects with potentially different default branches.
• if: '$CI_COMMIT_BRANCH =~ /regex-expression/': If the commit branch matches a regular expression.
• if: '$CUSTOM_VARIABLE !~ /regex-expression/': If the custom variable CUSTOM_VARIABLE does not match a regular expression.
• if: '$CUSTOM_VARIABLE == "value1"': If the custom variable CUSTOM_VARIABLE is exactly value1.

还有一些其他常用的变量：

参考：《Predefined environment variables reference》

CI_MERGE_REQUEST_SOURCE_BRANCH_NAME: The URL of the source project of the merge request if the pipelines are for merge requests. Available only if only: [merge_requests] or rules syntax is used and the merge request is created.
CI_MERGE_REQUEST_TARGET_BRANCH_NAME: The target branch name of the merge request if the pipelines are for merge requests. Available only if only: [merge_requests] or rules syntax is used and the merge request is created.

variables

参考：《variables》

GitLab CI/CD allows you to define variables inside .gitlab-ci.yml that are then passed in the job environment.

They can be set globally and per-job.

When the variables keyword is used on a job level, it will override the global YAML variables and predefined ones of the same name.

build b1:
  stage: .post
  only:
    - merge_requests
  variables:
    TRIGGER_SOURCE: "repo-a1"
  trigger:
    project: DevOps-testfarm/repo-b1
    branch: master

Git strategy

参考：《Git strategy》

GIT_STRATEGY is used for getting recent application code, either globally or per-job in the variables section.

There are three possible values: clone, fetch, and none.

clone is the slowest option. It clones the repository from scratch for every job, ensuring that the local working copy is always pristine.
fetch is faster as it re-uses the local working copy (falling back to clone if it does not exist).
none also re-uses the local working copy, but skips all Git operations (including GitLab Runner’s pre-clone script, if present).

variables:
  GIT_STRATEGY: fetch

Git submodule strategy

 参考：《Git submodule strategy》

The GIT_SUBMODULE_STRATEGY variable is used to control if / how Git submodules are included when fetching the code before a build. You can set them globally or per-job in the variables section.

There are three possible values: none, normal, and recursive:

none means that submodules won’t be included when fetching the project code. This is the default, which matches the pre-v1.10 behavior.

normal means that only the top-level submodules will be included.

git submodule sync
git submodule update --init

recursive means that all submodules (including submodules of submodules) will be included.

git submodule sync --recursive
git submodule update --init --recursive

Package Registry

参考《GitLab Package Registry》

Conan

apt-get install python3-pip
pip install conan