# k8s-jenkins在kubernetes中持续部署

k8s-jenkins在kubernetes中持续部署

1. k8s-jenkins在kubernetes中持续部署

  • Kubernetes Continuous Deploy插件:用于将资源配置部署到Kubernetes。
  • 插件介绍:https://plugins.jenkins.io/kubernetes-cd
  • 支持以下资源类型:
    • Deployment
    • Replica Set
    • Daemon Set
    • Pod
    • Job
    • Service
    • Ingress
    • Secret

2. 在jenkins安装插件

  • 登陆jenkins
    image

  • 系统管理
    image

  • 点击插件管理
    image

  • 安装插件 ”Kubernetes Continuous Deploy“
    image

  • 直接安装
    image

  • 安装成功后会重启

3. 在jenkins中kubernetes持续部署

3.1 在pipeline中添加kubernetes的key

  • 点击jenkins的pipeline项目
    image

  • 点击配置
    image

  • 找到pipeline语法
    image

    点击流水线语法

  • 在流水线配置kubernetesDeploy选项
    image
    image

    点击添加key

  • k8s使用的kubeadm安装的,我们可以在/root/.kube/config查看里面内容, 就是我们的认证信息

    [root@k8s-master jenkins-master]# cd /root/.kube/
    [root@k8s-master .kube]# cat config 
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1ERXdOREV6TlRNd05sb1hEVE14TURFd01qRXpOVE13Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUpyCktndlNnUllJKzJJZ2RaODNqTldINEptaXViYVN1Qkh1Rm9LK0tyRzExS2EzTm5DR3puUGlDejR3U1NGTC9icjcKOEpJQkxCQ0dKL1BqUUlLTEF2bzQ0UHN6TlkzU1VJRi9wa3VDamU1VXZYaFdpWlI0dFB5eUFGNEhzdmNLaHcvWgpqZkhLWGFCNnNRbC95L2FXS2dzdzFGQUJzWjVwMThOcDI1SC9ZV0NiK01wWE1mb3YyRUNGc2wybEhQNmlWN1JLCjZTTHhHVVI0K3gySmdhUjZEWFVTMlVKL1VVV0k3TnBtWkxIL2plRC9HNmdXbTVoWFVGRnBWbE4rbURRSXZoOG8Kb3hvWEE1cThqNnhUUWRINGs1WHI4VkhtKzI5dGVOaW1jYW92ZnRUWkJKM09WcFJOekFhZXNlaUpSMTE3ZHAzawpYb3VZdThJOTRMODc3ZEpHZE44Q0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZPWXpsZy9xY3JIMFBrN0gxMDdSRkxKVmJQQXhNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFBYnRWZFk0aU1YOVZHWGdsVFBLQXk2ckpVSkp2L2hFRDV6R3pqZXMvZVNmOGV6Wi9haAoyV1piRG5rcUpuSTNHV3Bzb25ja1luS1Nlb0l0RmJWNlN6VUJxUDNRY2ZxNmRzclVoRHJsQmJ4T0JkQ0twWlEzCkwwM0Vvek1CWXgyeC8vczRiZWR1SWtCR1RZSW9rc2xEc1hDZm5zSC9qY1V4cU01czJmaDFjNzZyejZwbGEwbGIKNyttYTRFYllXSERDQ3lFWVN1eHRwMEpSakZsQTNTSHlOL3VWa0hFemU5UWJjb2hCeHFPTVlPYk9SSW9wRmpuZAplb1dXdXhKTzljN3RRTU1VK0RXMVZoYmdraGxpellBejR4MkNPbEhzRGdKWkpBS29oMHl5NDdmNG9qY1lwN1UyCnhDRFMwVldiZlVReTNKVTBKMVlwa21GK1ZYZlhlUFJPUHB0NAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
        server: https://10.100.24.81:6443
      name: kubernetes
    contexts:
    - context:
        cluster: kubernetes
        user: kubernetes-admin
      name: kubernetes-admin@kubernetes
    current-context: kubernetes-admin@kubernetes
    kind: Config
    preferences: {}
    users:
    - name: kubernetes-admin
      user:
        client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lJR2ZmTlRDendJVVF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRBeE1EUXhNelV6TURaYUZ3MHlNakF4TURReE16VXpNRGxhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW9ST1YxTFdnek1NR1UvSXUKSnVRa2s5YXhoZXVPR2lka0RWekdoYzVNOEdQZGgxaXpWNlJELzRSQ1pOZ252bnM4d2lqNVZoN2JlaWJja09EMQpHVkNJa2Vsd2w2UzNKc1Q4UHdmN0VrMXVDU1ZXTXZpRXoxK2lpL28wNlo2bmVHam1NcVh0R3RpNitDZkNmNXBICmJ3RG9MK1h6Q2VsZEFpTTJRMjdBcFpLK3BUUjBFcStYcG5GblVxOWZPUmljQTBJYzhrV3ZFYVhSYng0RENBUlAKajJGNjJ3Z3B1QlI5cCtKQ1QzQWJOZ0Zyd2tERDl3bERicWRLR0w1N01lWlduMlR6SS9xN3pRUTFCbjk1enpJdApSVFZnMWNWTFN6ejFhUjU5cXNHaVI1YytSQTV4ZkxCclhWYTgrWFJTdExJUnJQYWVKVS9QeG9tRzc2cUVBWGN6Ckk0VXhlUUlEQVFBQm8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0h3WURWUjBqQkJnd0ZvQVU1ak9XRCtweXNmUStUc2ZYVHRFVXNsVnM4REV3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFKb3dENk1QcFNRRjJmQlZmNHBuczM3dTBSVDJkdmVYUWxYUjZUYTNiZ294Nmo1cUNZcnhjcDJ0Ckt0WmU1Ykdnc0ZTMzZSaXN0QkxUY1FKV01ocmtCL2dTM0wwQk5ZOTdYRWh6T1FlZmhZbzF0bGtlaDJ1TERKdTgKRktPditGbDNxeExUQlFCZGdtRnpWa0llN0cvOVYveUlmU0RWT1g5eDFUUnpJQURhSlIwTUZZNlZVQkE0R1UvSwpGbU5mU202QUhSY0R3amhlWHhuSGZXZzdBM2NZOW1FVlJUZ056QWNpTVJUblVKakdXUlY3aDFFUlh2eERwNDMwClB0NTd0QnhOUnZFSU9RblE3VG05SEtzTWczeWpkYnVSUG9XbEh3OExJdW1OQU02eDZQZ2dtMTE4Q1FLNzllbnEKK3JBUWFZaFUxbWgyS0ZlL3BiNU1NcTM2UVFYVitxVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
        client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBb1JPVjFMV2d6TU1HVS9JdUp1UWtrOWF4aGV1T0dpZGtEVnpHaGM1TThHUGRoMWl6ClY2UkQvNFJDWk5nbnZuczh3aWo1Vmg3YmVpYmNrT0QxR1ZDSWtlbHdsNlMzSnNUOFB3ZjdFazF1Q1NWV012aUUKejEraWkvbzA2WjZuZUdqbU1xWHRHdGk2K0NmQ2Y1cEhid0RvTCtYekNlbGRBaU0yUTI3QXBaSytwVFIwRXErWApwbkZuVXE5Zk9SaWNBMEljOGtXdkVhWFJieDREQ0FSUGoyRjYyd2dwdUJSOXArSkNUM0FiTmdGcndrREQ5d2xECmJxZEtHTDU3TWVaV24yVHpJL3E3elFRMUJuOTV6ekl0UlRWZzFjVkxTenoxYVI1OXFzR2lSNWMrUkE1eGZMQnIKWFZhOCtYUlN0TElSclBhZUpVL1B4b21HNzZxRUFYY3pJNFV4ZVFJREFRQUJBb0lCQVFDTUcrc2lZUEtENmNnaApVUEZRSnZhaUdnZkV3NGFSdlYzTU9iejlPdHUxNWtnVnFOUW5FZDJMN3RHSnduT2Z2cmEwOU1zZFhDM3c3TWpDCnhNcXR3RjFxL2pGUDFwanhnanNmNEtORjRsQXFhRzI0WEcrNXFpaEx0ZEk4dkNKVUhIclA1NEI4NzdTQ2txTWUKTjJhWGFhU050V2xZWDhMbFR4ZXVSRElTVTdpOUhhZXEyWEUwQ2huWkRSdFRjVGIvTDFBdXA1NXlMZnlCVkhCSQo4RFdkQzI0RzZabVI4ZDh0c0ZCUDBiY2dacklaYkhDVDVFMm02ZC81d2JjOEM4Y3ZsMkRncGNlOWJHdWZkaURjCkdSWkVHNmlPZmlsVlFjc3lrQUs1eUlNSVh3UUsrdEhWZWRYa2Z4b2kySE5EbWhhWmJ3TVIxVnhuZjI0L0g0eEcKdEpoQkl2VUJBb0dCQU1ON3pQa2Voek9jVzBmMCs1UUpEMnNLM2tvVC8wM2RPanFFZFdYTUsxQ2lBVGM2enI1Rwp4MXQ1clFINXFFYW1CRGU2SkZXdXVqTnlWWFY0OFRSVEtjTmJ5QlM4OGcrVzNHL0VvVDhkbHlNM01FRlFZNWNjCkJEWUVRNGFrUjMwL2Z1bTBTV0NsM2YxMW1TcHRIVnZqRnBGbkZaYXloOGNDYzZtaFJURy9KR1JKQW9HQkFOTHgKQUMrb0VCUFpnVGlsMjFSV0dyd3dHZU5hUkE0NGRPWTFwb0ZOVERFWGtwdUlTNWJwSFVnajNYMnhnMGl4eTFGaAphd2lnTzRVZjBKRFVSbGJaOTBRd0doMEZRYVVPRGNyQnJDU0R0TFdlNlVsTldTTWx1T2ZSamF0VVE5b0RiTlRICnJTNXFyTis3WXRsNXdTaWcyWEkwRW1nbnhxQWx4WkpxTGNudXR3T3hBb0dBUWMvOFFpSzFBZnN5QUJmS1d4eHgKeEdmMFRweE01N29EVUpBbnRnMzJ4ejdEaWdTU2RoeitJUjNsMFRSc0l0emhycUZNZWt4SkEra0tQSEpyUW1KOQpQUGc1ZEVIb2ZMY2RuOE5qUmd5QVJhcmU1QjVjYVJOVmw1bjM0ZXlNNU8zcm5HTEdFWFBHT1A3T2dpb2FLRVZiCjQ3Z0EzS3RlYXZHNFh2T2NpQk1CL1lFQ2dZRUF6UExCUm94WXlxQnFiOXFkeGt3K2Y1ZVFuNWZsUlpmTzVKazkKNUtWd29NWVJlTEl0Rnp2NWFrMEdkNGhRVVFJRjFGVDloSTBDSjI3bzJyNlVkUEFqMFltVzVabGsxWFJaTENXTwpGcEFzdE1Yb1RZTXNWdnpnc2NPbGpOQTdlSWdaWm9kM0dRRW1ZOTZSeFZFSnZtckJ1cmNqSnpWMlE4VlBOOW1ZCkVYbXhNWkVDZ1lBS1h0elpUMi9xTnB1ZDlITCtMamNtSUk4OHdmUlJIaWV5aUJWQUZxNFU4SjA3ZnFObnlsb20KTVdrUkwxVnhFV3FxTm5haUpyVzJITHdHc3pNa0NQY21jbVVlNWpqVVZ2ZXJyVllaK0J2UmZmR0FXYUlGNi9uOQpxRUQybktFOGl6MHVLekhHU0ViSkplSXU0VWR5RXIrT0hXcUg2YlFMbUVEVEFQKzMzOE1LOGc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
    
    
  • 配置key
    image![image]

  • 在凭据里查看key
    image

    a16f5236-7cdc-4216-a4ec-c52346751f56
    

3.2 部署案例

3.2.1 pipeline示例脚本
// 公共
def registry = "harbor.scajy.org"           // harbor的连接地址
// 项目
def project = "k8s"                      // harbor的项目目录
def app_name = "java-demo"                       // 项目名称
def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"      // 推送harbor的全路径,BUILD_NUMBER是取得当前jenkins构建的号,为版本号
def git_address = "git@10.100.24.87:k8s/tomcat_java_demo.git"             // 根据项目要更换git地址
def base_image = "${registry}/library/tomcat:latest"                      // 构建项目需要的基础镜像
// 认证
def secret_name = "registry-pull-secret"
def docker_registry_auth = "b6085682-39e7-458f-96c0-f139fe17f2e1"       // harbor仓库key,这个key要去jenkins查找
def git_auth = "d8bf675c-5dd4-4bf4-a4f7-4e8c1be1c002"                   // git代码仓库的key,这个key要去jenkins查找
def k8s_auth = "a16f5236-7cdc-4216-a4ec-c52346751f56"                   // k8s连接的key

podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
    containerTemplate(
        name: 'jnlp',          // 模板
        image: "${registry}/library/jenkins-slave:1.8"       // jenkins-slave从服务器镜像
    ),
  ],
  volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
    hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
  ],
) 
{
  node("jenkins-slave"){
      // 第一步
      stage('拉取代码'){
         checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
      }
      // 第二步
      stage('代码编译'){
          sh "mvn clean package -Dmaven.test.skip=true"
      }
      // 第三步
      stage('构建镜像'){
          withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
            sh """
              echo '
                FROM  ${base_image}
                RUN rm -rf /usr/local/tomcat/webapps/*
                ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 
              ' > Dockerfile
              ls 
              ls target
              docker build -t ${image_name} .
              docker login -u ${username} -p '${password}' ${registry}
              docker push ${image_name}
            """
            }
      }
      // 第四步
      stage('部署到K8S平台'){
          sh """
          sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yml
          sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yml
          """
          kubernetesDeploy configs: 'deploy.yml', kubeconfigId: "${k8s_auth}"
      }

  }
}

3.2.2 上传在k8s部署的脚本deploy.yaml到gitlab项目里面
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3 
  selector:
    matchLabels:
      app: java-demo
  template:
    metadata:
      labels:
        app: java-demo 
    spec:
      imagePullSecrets:
      - name: $SECRET_NAME 
      containers:
      - name: tomcat 
        image: $IMAGE_NAME 
        ports:
        - containerPort: 8080
          name: web
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12


---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: NodePort
  selector:
    app: java-demo 
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web 
spec:
  rules:
  - host: java.scajy.org
    http:
      paths:
      - path: /
        backend:
          serviceName: web 
          servicePort: 80

image
image

  • gitlab项目验证deploy部署配置文件是否上传
    image
3.2.3 在jenkins里面开始部署
  • 登陆jenkins
    image

  • 找到项目
    image

  • 点击项目配置
    image

  • 配置项目
    image
    image

    全部代码

    // 公共
    def registry = "harbor.scajy.org"           // harbor的连接地址
    // 项目
    def project = "k8s"                      // harbor的项目目录
    def app_name = "java-demo"                       // 项目名称
    def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"      // 推送harbor的全路径,BUILD_NUMBER是取得当前jenkins构建的号,为版本号
    def git_address = "git@10.100.24.87:k8s/tomcat_java_demo.git"             // 根据项目要更换git地址
    def base_image = "${registry}/library/tomcat:latest"                      // 构建项目需要的基础镜像
    // 认证
    def secret_name = "registry-pull-secret"
    def docker_registry_auth = "b6085682-39e7-458f-96c0-f139fe17f2e1"       // harbor仓库key,这个key要去jenkins查找
    def git_auth = "d8bf675c-5dd4-4bf4-a4f7-4e8c1be1c002"                   // git代码仓库的key,这个key要去jenkins查找
    def k8s_auth = "a16f5236-7cdc-4216-a4ec-c52346751f56"                   // k8s连接的key
    
    podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
        containerTemplate(
            name: 'jnlp',          // 模板
            image: "${registry}/library/jenkins-slave:1.8"       // jenkins-slave从服务器镜像
        ),
      ],
      volumes: [
        hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
        hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
      ],
    ) 
    {
      node("jenkins-slave"){
          // 第一步
          stage('拉取代码'){
             checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
          }
          // 第二步
          stage('代码编译'){
              sh "mvn clean package -Dmaven.test.skip=true"
          }
          // 第三步
          stage('构建镜像'){
              withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                sh """
                  echo '
                    FROM  ${base_image}
                    RUN rm -rf /usr/local/tomcat/webapps/*
                    ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 
                  ' > Dockerfile
                  ls 
                  ls target
                  docker build -t ${image_name} .
                  docker login -u ${username} -p '${password}' ${registry}
                  docker push ${image_name}
                """
                }
          }
          // 第四步
          stage('部署到K8S平台'){
              sh """
              sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yml
              sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yml
              """
              kubernetesDeploy configs: 'deploy.yml', kubeconfigId: "${k8s_auth}"
          }
    
      }
    }
    
    

    经过前面的CI部署,我们这里增加了第四步,部署镜像到k8s集群

3.2.4 k8s创建harbor认证凭据
  • 创建凭据命令

    [root@k8s-master test]# kubectl create secret docker-registry registry-pull-secret --docker-server=10.100.24.85 --docker-username=admin --docker-password=Aa123456--docker-email=admin@admin.com
    
3.2.5 构建执行

image
image
image

3.3 验证k8s集群是否部署

  • 验证是否部署

    [root@k8s-master test]# kubectl get pods
    NAME                                      READY   STATUS    RESTARTS   AGE
    jenkins-0                                 1/1     Running   0          26h
    nfs-client-provisioner-5d99f8d7b9-562ff   1/1     Running   1          18d
    nginx-6799fc88d8-9fchc                    1/1     Running   0          17d
    web-56d4ff7c69-gnk5b                      0/1     Running   0          52s
    web-56d4ff7c69-szgq8                      1/1     Running   0          116s
    web-7cb8dccfb6-dpwtp                      1/1     Running   0          7m58s
    web-7cb8dccfb6-hzvm8                      1/1     Running   0          5m44s
    
    
  • 验证service对外服务是否提供

    [root@k8s-master test]# kubectl get service
    NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
    jenkins      NodePort    10.105.137.255   <none>        80:30006/TCP,50000:31876/TCP   26h
    kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP                        19d
    nginx        NodePort    10.99.236.10     <none>        80:30939/TCP                   17d
    web          NodePort    10.103.228.1     <none>        80:31224/TCP                   16m
    
    [root@k8s-master test]# kubectl get ep
    NAME             ENDPOINTS                                                   AGE
    fuseim.pri-ifs   <none>                                                      18d
    jenkins          10.244.36.79:8080,10.244.36.79:50000                        26h
    kubernetes       10.100.24.81:6443                                           19d
    nginx            10.244.169.131:80                                           17d
    web              10.244.107.216:8080,10.244.169.136:8080,10.244.36.81:8080   16m
    
    
  • 验证ingress是否创建

    [root@k8s-master test]# kubectl get ingress
    Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
    NAME      CLASS    HOSTS               ADDRESS   PORTS   AGE
    jenkins   <none>   jenkins.scajy.org             80      26h
    web       <none>   java.scajy.org                80      17m
    

3.4 浏览器访问测试

  • 绑定ingress的域名hosts
    image

  • 浏览器访问java.scajy.org域名测试
    image

posted @ 2022-05-20 14:20  七月流星雨  阅读(164)  评论(0编辑  收藏  举报