Go-Kratos 项目 SonarQube、Jenkins 与 K8s CI/CD 实践

环境规划

服务器名称	操作系统	服务名称	IP	功能描述
devops	ubuntu-22.04.5	GitLab	10.1.1.252:30080	代码仓库
devops	ubuntu-22.04.5	Harbor	10.1.1.252:18090	镜像仓库
devops	ubuntu-22.04.5	Jenkins	10.1.1.252:8080	CI/CD
devops	ubuntu-22.04.5	SonarQube	10.1.1.252:9000	代码质量检测
master	ubuntu-22.04.5	K8S-master	10.1.1.2	K8S控制节点
node1	ubuntu-22.04.5	k8s-node1	10.1.1.3	k8s工作节点1
node2	ubuntu-22.04.5	k8s-node2	10.1.1.4	k8s工作节点2

默认上述服务都已经搭建完毕

CI/CD解决方案架构图

Harbor配置

Docker配置

由于需要通过docker构建镜像并且推送到镜像仓库中, 需要建立docker与镜像仓库建立认证关系

# 添加认证地址
vim /etc/docker/daemon.json
{
"insecure-registries": ["10.1.1.252:18090"]
}

# 添加认证地址之后,重启docker
systemctl daemon-reload&& systemctl restart docker

# 进行登录认证
docker login  -u admin -p Harbor12345 http://10.1.1.252:18090

K8S配置

由于需要从镜像仓库拉取镜像创建Pod, 需要建立K8S与镜像仓库认证关系

# 创建命名空间
kubectl create ns kube-devops

# 创建secret
kubectl create secret docker-registry harbor-secret --docker-server=10.1.1.252:18090 --docker-username=admin --docker-password=Harbor12345 -n kube-devops

# 查看创建的secret
kubectl get secret -n kube-devops

jenkins配置

镜像构建

由于后期需要用到go,node和sonar-scanner,重新制作jenkins

FROM jenkins/jenkins:latest
USER root
RUN sed -i 's@deb.debian.org@repo.huaweicloud.com@g' /etc/apt/sources.list.d/debian.sources

RUN apt-get update -y && apt-get install -y iputils-ping curl ca-certificates vim wget unzip \
    && curl -fsSL https://deb.nodesource.com/setup_18.x | bash - \
    && apt-get install -y nodejs \
    && node -v && npm -v \
    && rm -rf /var/lib/apt/lists/*

# Copy local Go and SonarScanner archives into the image
COPY go1.24.0.linux-arm64.tar.gz /tmp/go.tar.gz
COPY sonar-scanner-cli-7.2.0.5079-linux-aarch64.zip /tmp/sonar-scanner.zip

# Install Go
RUN tar -C /usr/local -xzf /tmp/go.tar.gz \
    && rm -f /tmp/go.tar.gz

# Install SonarScanner
RUN unzip -q /tmp/sonar-scanner.zip -d /opt \
    && rm -f /tmp/sonar-scanner.zip \
    && mv /opt/sonar-scanner-* /usr/local/sonar-scanner

# Environment variables for Go and SonarScanner
ENV GOROOT=/usr/local/go
ENV GOPATH=/var/jenkins_home/go
ENV SONAR_SCANNER_HOME=/usr/local/sonar-scanner
ENV PATH=$SONAR_SCANNER_HOME/bin:$GOROOT/bin:$GOPATH/bin:/usr/local/bin:$PATH
ENV GOPROXY=https://goproxy.cn,direct
ENV GO111MODULE=on

插件下载

Build Authorization Token Root
Gitlab
SonarQube Scanner
Node and Label parameter
Kubernetes
docker
Config File Provider
Git Paramete
Blue Ocean
Git
Pipeline Stage View

插件配置

SonarQube Scanner

登录sonarqube之后点击我的账号-->安全---->生成令牌
在jenkins中,创建认证凭证
点击全局凭证, 并且添加凭证

进入jenkins系统配置,搜索sonarqube server

点击Add SonarQube添加凭证

gitlab

点击全局凭证, 并且添加凭证

kubernetes

点击cloud---new cloud

选择kubernetes类型

查看K8S地址

kubectl cluster-info

配置域名解析

由于我这边对外提供的是域名, 需要再jenkins服务器中填写域名解析
```
vim /etc/hosts
10.1.1.2  lb.kubesphere.local
```

配置认证凭证

配置rbac权限

#创建serviceaccounts
kubectl create sa jenkins
#对jenkins做cluster-admin绑定
kubectl create clusterrolebinding jenkins --clusterrole cluster-admin --serviceaccount=default:jenkins

获取token, kubernetes-plugin与k8s连接时，并不是直接使用serviceaccount，而是通过token。因此我们需要获取serviceaccount:jenkins对应的token。

# 查看sa命名空间
kubectl get sa -n default

# 查看secret
kubectl describe sa jenkins -n default

# 获取token
kubectl describe secrets jenkins-token-82fv9 -n default

配置凭证

获取K8S证书key

yq e '.clusters[0].cluster.certificate-authority-data' /root/.kube/config | base64 -d

配置K8S连接

点击测试

Managed files

点击创建新的配置文件

content中填写config文件值

cat /root/.kube/config

CICD配置

jenkins

创建项目

点击高级按钮

生成token
进入gitlab中选择对应的项目,配置webhook

点击测试选择推送事件,出现200则说明配置成功了

流水线配置

在流水线中选择SCM并且连接gitlab中项目仓库, 认证选择连接gitlab的认证, 脚本路径选择Jenkinsfile

jenkinsfile编写

pipeline {
    agent {
        node {
            label 'master' // 在标记为master的Jenkins节点上执行
        }
    }

    // 定义全局环境变量
    environment {
        // Docker镜像仓库地址
        DOCKER_HARBOR_REGISTRY = '10.1.1.252:18090'
        // Harbor私有仓库的凭证ID（在Jenkins中配置的用户名密码凭证）
        DOCKER_HARBOR_CREDENTIAL_ID = 'harbor-user-password'
        // Git代码仓库地址
        GIT_REPO_URL = 'http://10.1.1.252:30080/root/guliedu.git'
        // Git仓库的凭证ID（在Jenkins中配置的用户名密码凭证）
        GIT_CREDENTIAL_ID = 'gitlab-username-password'
        // SonarQube凭证
        SONARQUBE_CREDENTIAL_ID = 'sonarqube-token'
        // SonarQubeURL
        SONAR_HOST_URL = 'http://10.1.1.252:9000'
        // Kubernetes集群配置文件的凭证ID
        KUBECONFIG_CREDENTIAL_ID = '93e074ab-55e4-45c8-b745-29131928f583'
        // Docker Hub命名空间（用于推送镜像）
        DOCKERHUB_NAMESPACE = 'guliedu'
        // 应用程序名称
        PROJECT_NAME = 'kratos-guliedu-backend'

        // 全局应用环境变量
        IS_DEV_MODE = "${BRANCH_NAME == 'main' ? 'false' : 'true'}"  // 根据分支动态设置开发模式
        CONSUL_HOST = '10.1.1.254:8500'         // Consul服务地址
        FLUENTD_HOST = '10.1.1.254:24224'       // Fluentd日志收集地址
    }

    // 定义流水线参数，用户可以在触发构建时指定
    parameters {
        // 分支名称参数，使用下拉选择，默认值为dev
        choice(
            name: 'BRANCH_NAME',
            choices: ['dev', 'main', 'test'],
            description: '请选择要发布的分支名称'
        )

        choice(
            name: 'POD_COUNT',
            choices: [3, 5, 7],
            description: '请选择副本数量'
        )

        choice(
            name: 'NAMESPACE',
            choices: ['guliedu-dev', 'guliedu-main', 'guliedu-test'],
            description: '请选择命名空间'
        )

        // 手动指定要构建的服务（下拉选择）。选择 auto 表示自动探测改动服务
        choice(
            name: 'TARGET_SERVICES',
            choices: ['auto', 'edu', 'file', 'learning', 'trade', 'user', 'video'],
            description: '请选择要构建的服务（auto = 自动探测改动服务）'
        )

        // 标签名称参数，用于版本标记，必须以v开头
        string(name: 'TAG_NAME', defaultValue: 'snapshot', description: '标签名称，必须以 v 开头，例如：v1、v1.0.0')
    }

    // 填写流水线步骤
    stages {
        // 克隆代码
        stage('Clone Code') {
            steps {
                checkout([
                    $class: 'GitSCM',
                    branches: [[name: "${BRANCH_NAME}"]],
                    extensions: [],
                    userRemoteConfigs: [[
                        credentialsId: "${GIT_CREDENTIAL_ID}",
                        url: "${GIT_REPO_URL}"
                    ]]
                ])
            }
        }





        // 单元测试
        stage('Unit Test') {
            steps {
                // 显示当前环境变量（用于调试）
                sh 'echo "=== Environment Variables ==="'
                sh 'echo "IS_DEV_MODE: $IS_DEV_MODE"'
                sh 'echo "CONSUL_HOST: $CONSUL_HOST"'
                sh 'echo "FLUENTD_HOST: $FLUENTD_HOST"'
                sh 'echo "BRANCH_NAME: $BRANCH_NAME"'
                sh 'echo "========================="'
                // 运行单元测试
                sh 'go test ./...'
            }
        }


        // 代码质量分析
        stage('SonarQube Analysis') {
            steps {
                // 设置SonarQube环境变量
                script {
                    // SonarQube服务器地址
                    env.SONAR_HOST_URL = "${SONAR_HOST_URL}"
                    // 项目唯一标识符，格式：项目名-分支名
                    env.SONAR_PROJECT_KEY = "${PROJECT_NAME}-${BRANCH_NAME}"
                    // 项目显示名称
                    env.SONAR_PROJECT_NAME = "${PROJECT_NAME}-${BRANCH_NAME}"
                    // 项目版本号，使用Jenkins构建号
                    env.SONAR_PROJECT_VERSION = "${BUILD_NUMBER}"
                    // 源代码根目录
                    env.SONAR_SOURCES = '.'
                    // 排除的文件和目录（不进行代码质量分析）
                    // **/third_party/** - 第三方库文件
                    // **/doc/** - 文档目录
                    // **/api/** - API定义文件（protobuf生成的接口定义）
                    // **/deploy/** - 部署配置文件
                    // **/httpclient/** - HTTP客户端测试文件
                    // **/*_test.go - Go测试文件
                    // **/*_pb.go - protobuf生成的Go文件
                    // **/*_validate.go - protobuf验证文件
                    env.SONAR_EXCLUSIONS = '**/third_party/**,**/doc/**,**/api/**,**/migrations/**,**/deploy/**,**/httpclient/**,**/*_test.go,**/*pb.go,**/*validate.go,**/*gen.go,**/*_pb.go,**/*_validate.go,**/*_gen.go,**/*.js,**/*.jsx,**/*.ts,**/*.yaml,**/*.yml,**/*.tsx,**/package*.json,**/webpack*.js,**/vite*.config.*'
                    // 排除覆盖率统计的文件和目录
                    // **/*_test.go - 测试文件不计算覆盖率
                    // **/third_party/** - 第三方库不计算覆盖率
                    env.SONAR_COVERAGE_EXCLUSIONS = '**/*_test.go,**/*_pb.go,**/*_validate.go,**/*pb.go,**/*.yaml,**/*.yml,**/*validate.go,**/third_party/**,**/api/**'
                    // 禁用 JS/TS 分析，避免 Node 依赖
                    env.SONAR_JS_EXCLUSIONS = '**/*'
                    env.SONAR_TS_EXCLUSIONS = '**/*'
                }

                // 打印SonarQube配置信息
                sh 'echo "=== SonarQube Configuration ==="'
                sh 'echo "SONAR_HOST_URL: $SONAR_HOST_URL"'
                sh 'echo "SONAR_PROJECT_KEY: $SONAR_PROJECT_KEY"'
                sh 'echo "SONAR_PROJECT_NAME: $SONAR_PROJECT_NAME"'
                sh 'echo "SONAR_PROJECT_VERSION: $SONAR_PROJECT_VERSION"'
                sh 'echo "SONAR_SOURCES: $SONAR_SOURCES"'
                sh 'echo "==============================="'

                withCredentials([string(credentialsId: "${SONARQUBE_CREDENTIAL_ID}", variable: 'SONAR_TOKEN')]) {
                    withSonarQubeEnv('sonarqube') {
                        sh '''
                              sonar-scanner \
                                -Dsonar.host.url=$SONAR_HOST_URL \
                                -Dsonar.token=$SONAR_TOKEN \
                                -Dsonar.projectKey=$SONAR_PROJECT_KEY \
                                -Dsonar.projectName=$SONAR_PROJECT_NAME \
                                -Dsonar.projectVersion=$SONAR_PROJECT_VERSION \
                                -Dsonar.sources=$SONAR_SOURCES \
                                -Dsonar.exclusions=$SONAR_EXCLUSIONS \
                                -Dsonar.go.coverage.reportPaths=coverage.out \
                                -Dsonar.sourceEncoding=UTF-8 \
                                -Dsonar.javascript.exclusions=$SONAR_JS_EXCLUSIONS \
                                -Dsonar.typescript.exclusions=$SONAR_TS_EXCLUSIONS \
                        '''
                    }
                    timeout(unit: 'MINUTES', activity: true, time: 5) {
                        waitForQualityGate 'true'
                    }
                }
            }
        }

        // 构建并且推送镜像
        stage('Build & Push Image') {
            steps {
                withCredentials([usernamePassword(credentialsId: "${DOCKER_HARBOR_CREDENTIAL_ID}", passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
                    script {
                        // 获取用户选择的服务
                        def selected = params.TARGET_SERVICES?.trim()
                        echo "TARGET_SERVICES selected: ${selected}"
                        def services = []

                        // 根据用户选择决定构建策略
                        if (selected == 'auto') {
                            // 自动检测模式：检测代码变更的服务

                            // 1) 优先用上次成功提交区间
                            def range = env.GIT_PREVIOUS_SUCCESSFUL_COMMIT ? "${env.GIT_PREVIOUS_SUCCESSFUL_COMMIT}..HEAD" : ""
                            def changed = ""
                            if (range) {
                                changed = sh(returnStdout: true, script: "git diff --name-only ${range}").trim()
                                echo "Changed by range ${range}:\n${changed}"
                            }

                            // 2) 退回 merge-base
                            if (!changed) {
                                sh 'git fetch --no-tags --all || true'
                                def base = sh(returnStdout: true, script: 'git merge-base HEAD origin/$BRANCH_NAME').trim()
                                echo "merge-base: ${base}"
                                changed = sh(returnStdout: true, script: "git diff --name-only ${base} HEAD").trim()
                                echo "Changed by merge-base:\n${changed}"
                            }

                            // 3) 最后兜底：上一提交到当前
                            if (!changed) {
                                changed = sh(returnStdout: true, script: "git diff --name-only HEAD^..HEAD || true").trim()
                                echo "Changed by HEAD^..HEAD:\n${changed}"
                            }

                            // 从变更文件中提取服务名称
                            if (changed) {
                                services = changed
                                    .split('\n')
                                    .findAll { it.startsWith('app/') && it.split('/').size() > 1 }
                                    .collect { it.split('/')[1] }
                                    .unique()
                            }
                            echo "Auto-detected services: ${services}"
                        } else {
                            // 手动选择模式：构建指定服务
                            services = [selected]
                            echo "Manual selection: ${services}"
                        }

                        // 将构建服务列表保存到环境变量，供后续部署阶段复用
                        env.SERVICES = (services && !services.isEmpty()) ? services.unique().join(',') : ''
                        echo "Services for build: ${env.SERVICES}"

                        if (!services || services.isEmpty()) {
                            echo 'No service selected or detected under app/. Skip build, continue pipeline.'
                        } else {
                            // 登录 Docker Registry
                            sh 'echo "$DOCKER_PASSWORD" | docker login $DOCKER_HARBOR_REGISTRY -u "$DOCKER_USERNAME" --password-stdin'

                            // 逐个服务构建并推送
                            for (svc in services) {
                                // 根据分支确定镜像标签
                                def tag = (env.BRANCH_NAME == 'main') ? 'latest' : "snapshot-${env.BRANCH_NAME}-${svc}-${env.BUILD_NUMBER}"
                                sh """
                                  echo Building service: ${svc} with tag: ${tag}
                                  docker build -f app/${svc}/service/Dockerfile \\
                                    --build-arg APP_RELATIVE_PATH=${svc}/service \\
                                    --build-arg APP_VERSION=${BUILD_NUMBER} \\
                                    --build-arg IS_DEV_MODE=${IS_DEV_MODE} \\
                                    --build-arg CONSUL_HOST=${CONSUL_HOST} \\
                                    --build-arg FLUENTD_HOST=${FLUENTD_HOST} \\
                                    -t ${DOCKER_HARBOR_REGISTRY}/${DOCKERHUB_NAMESPACE}/${svc}:${tag} .
                                  docker push ${DOCKER_HARBOR_REGISTRY}/${DOCKERHUB_NAMESPACE}/${svc}:${tag}
                                  echo 开始删除镜像
                                  docker rmi -f ${DOCKER_HARBOR_REGISTRY}/${DOCKERHUB_NAMESPACE}/${svc}:${tag} || true
                                  docker image prune -f || true
                                """
                            }
                        }
                    }
                }
            }
        }

        // K8S 部署
        stage('K8S Deploy') {
            steps {
                configFileProvider([configFile('93e074ab-55e4-45c8-b745-29131928f583', 'admin-kubeconfig')]) {
                    sh 'mkdir -p ~/.kube/'
                    sh 'cp "$ADMIN_KUBECONFIG" ~/.kube/config'
                    sh 'kubectl get nodes -A'
                    script {
                        def selected = params.TARGET_SERVICES?.trim()
                        def services = []
                        def namespace = env.GULIEDU_NAMESPACE
                        def pod_count = env.POD_COUNT

                        if (selected == 'auto') {
                            // 与构建阶段一致：自动检测变更服务
                            def range = env.GIT_PREVIOUS_SUCCESSFUL_COMMIT ? "${env.GIT_PREVIOUS_SUCCESSFUL_COMMIT}..HEAD" : ""
                            def changed = ""
                            if (range) {
                                changed = sh(returnStdout: true, script: "git diff --name-only ${range}").trim()
                                echo "Changed by range ${range}:\n${changed}"
                            }

                            if (!changed) {
                                sh 'git fetch --no-tags --all || true'
                                def base = sh(returnStdout: true, script: 'git merge-base HEAD origin/$BRANCH_NAME').trim()
                                echo "merge-base: ${base}"
                                changed = sh(returnStdout: true, script: "git diff --name-only ${base} HEAD").trim()
                                echo "Changed by merge-base:\n${changed}"
                            }

                            if (!changed) {
                                changed = sh(returnStdout: true, script: "git diff --name-only HEAD^..HEAD || true").trim()
                                echo "Changed by HEAD^..HEAD:\n${changed}"
                            }

                            if (changed) {
                                services = changed
                                    .split('\n')
                                    .findAll { it.startsWith('app/') && it.split('/').size() > 1 }
                                    .collect { it.split('/')[1] }
                                    .unique()
                            }
                            echo "Auto-detected services for deploy: ${services}"
                        } else {
                            services = [selected]
                            echo "Manual selection for deploy: ${services}"
                        }

                        for (svc in services) {
                            def tag = (env.BRANCH_NAME == 'main') ? 'latest' : "snapshot-${env.BRANCH_NAME}-${svc}-${env.BUILD_NUMBER}"
                            def image = "${DOCKER_HARBOR_REGISTRY}/${DOCKERHUB_NAMESPACE}/${svc}:${tag}"

                            sh """
                              echo Deploying ${svc} with IMAGE=${image}
                              cp app/${svc}/service/deploy.yaml app/${svc}/service/deploy.rendered.yaml
                              sed -i "s#\\\${IMAGE}#${image}#g" app/${svc}/service/deploy.rendered.yaml
                              sed -i "s#\\\${GULIEDU_NAMESPACE}#${namespace}#g" app/${svc}/service/deploy.rendered.yaml
                              sed -i "s#\\\${POD_COUNT}#${pod_count}#g" app/${svc}/service/deploy.rendered.yaml
                              kubectl apply -f app/${svc}/service/deploy.rendered.yaml
                            """
                        }
                    }
                }
            }
        }
    }
}

posted @ 2025-09-09 18:23 SR丶阅读(26) 评论(0) 收藏举报

刷新页面返回顶部

星辰大海(｀・ω・´)

Go-Kratos 项目 SonarQube、Jenkins 与 K8s CI/CD 实践

Go-Kratos 项目 SonarQube、Jenkins 与 K8s CI/CD 实践

环境规划

CI/CD解决方案架构图

Harbor配置

Docker配置

K8S配置

jenkins配置

镜像构建

插件下载

插件配置

SonarQube Scanner

gitlab

kubernetes

Managed files

CICD配置

jenkins

创建项目

流水线配置

jenkinsfile编写

公告