Stay Hungry,Stay Foolish!

Using JobDSL and Jenkinsfiles to fully automate Jenkins job management

Using JobDSL and Jenkinsfiles to fully automate Jenkins job management

非常贴切的一个比喻

JobDSL 做Job管理的框架， 负责Job基本参数配置

Pipeline 做Job逻辑内容的容器，决定Job执行逻辑。

By using JobDSL and Jenkinsfiles we can fully automate all aspects of Jenkins job management. The two components play the following roles

  • JobDSL: The skeletal framework of a job that “registers” it in the Jenkins UI but does not contain any core job logic
  • Jenkinsfiles: Contains all the project specific steps to accomplish the project goals such as building or deploying code

Sandwhich DSL goodness

It does not have to be JobDSL vs. Jenkinsfiles but, rather, JobDSL and Jenkinsfiles.

 

JobDSLPermalink

The code statements in JobDSL map directly to what is in the Jenkins UI as shown below

Yummy JobDSL FRED_AI

JenkinsfilesPermalink

The “meat” of the job, that is, the core logic and all logic specific to the project in question is contained in the Jenkinsfile. Quite often this is just called “Jenkinsfile” or “Jenkinsfile.groovy” (if you want to clue in your editor/IDE) but it can be called anything.

Below are some Jenkinsfile snippets showing, roughly, how the correspond to what you might see in the Jenkins UI.

Yummy Jenkinsfiles

 

Jenkinsfile

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/

Jenkins创造了Pipeline类型的Job，

并给其配备了代码的方式来描述Job的逻辑， Jenkinsfile

使得 Job本身 可以被源码控制， 存到git库中，部署的时候再拿出来，部署到Jenkins服务器上。

 

As discussed in the Defining a Pipeline in SCM, a Jenkinsfile is a text file that contains the definition of a Jenkins Pipeline and is checked into source control. Consider the following Pipeline which implements a basic three-stage continuous delivery pipeline.

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

 

 

Job DSL

https://plugins.jenkins.io/job-dsl/

jobDSL，跟jenkinfile类似，也是使用编程的方式来管理Job

Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs. Unfortunately, as the number of jobs grows, maintaining them becomes tedious, and the paradigm of using a UI falls apart. Additionally, the common pattern in this situation is to copy jobs to create new ones, these "children" have a habit of diverging from their original "template" and consequently it becomes difficult to maintain consistency between these jobs.

The Job DSL plugin attempts to solve this problem by allowing jobs to be defined in a programmatic form in a human readable file. Writing such a file is feasible without being a Jenkins expert as the configuration from the web UI translates intuitively into code.

 

Jenkinsfile更加适合创建复杂的自动化过程。

JobDSL可以创建任意类型的Job，包括Pipeline类型的Job，但是实现负责的逻辑，并不擅长。

The Pipeline plugins support implementing and integrating continuous delivery pipelines via the Pipeline DSL. While it is possible to use Job DSL to create complex pipelines using freestyle jobs in combination with many plugins from the Jenkins ecosystem, creating and maintaining these pipeline, including generating jobs for individual SCM branches and possibly running steps in parallel to improve build performance, poses a significant challenge. Jenkins Pipeline is often the better choice for creating complex automated processes. Job DSL can be used to create Pipeline and Multibranch Pipeline jobs. Do not confuse Job DSL with Pipeline DSL, both have their own syntax and scope of application.

 

JobDSL 教程

https://www.digitalocean.com/community/tutorials/how-to-automate-jenkins-job-configuration-using-job-dsl

 

JobDSL API参考

https://dev.astrotech.io/jenkins/index.html#job-dsl

https://jenkinsci.github.io/job-dsl-plugin/#

 

Pipeline call jobDsl to deploy all kinds of jobs

https://www.jenkins.io/doc/pipeline/steps/job-dsl/

jobDsl: Process Job DSLs

  • additionalClasspath : String (optional)
    Newline separated list of additional classpath entries for the Job DSL scripts. All entries must be relative to the workspace root, e.g. build/classes/main. Supports Ant-style patterns like lib/*.jar.
  • additionalParameters (optional)
    • Type: java.util.Map<java.lang.String, java.lang.Object>
  • failOnMissingPlugin : boolean (optional)
    If checked, the build will be marked as failed if a plugin must be installed or updated to support all features used in the DSL scripts. If not checked, the build will be marked as unstable instead.
  • failOnSeedCollision : boolean (optional)
    Fail build if generated item(s) have the same name as existing items already managed by another seed job. By default, this plugin will always regenerate all jobs and views, thus updating previously generated jobs and views even if managed by another seed job. Check this box if you wish to fail the job if a generated item name collision is detected.
  • ignoreExisting : boolean (optional)
    Ignore previously generated jobs and views. By default, this plugin will always regenerate all jobs and views, thus updating previously generated jobs and views. Check this box if you wish to leave previous jobs and views as is.
  • ignoreMissingFiles : boolean (optional)
    Ignore missing DSL scripts. If not checked, the build step will fail if a configured script is missing or if a wildcard does not match any files.
  • lookupStrategy (optional)
    Determines how relative job names in DSL scripts are interpreted. You will only see a difference when the seed job is located in a folder.
    • Jenkins Root When this option is selected relative job names are always interpreted relative to the Jenkins root.
    • Seed Job If you choose this option relative job names in DSL scripts will be interpreted relative to the folder in which the seed job is located. Suppose you have a seed job which is located in a folder named seedJobFolder and a DSL script which creates a job named subfolder2/job. The job that is created by the seed job will be at the location /seedJobFolder/subfolder2/job.
    • Values: JENKINS_ROOT, SEED_JOB
  • removedConfigFilesAction (optional)
    Specifies what to do when previously generated config files are not referenced anymore.

    Note: when using multiple Job DSL build steps in a single job, set this to "Delete" only for the last Job DSL build step. Otherwise config files may be deleted and re-created. See JENKINS-44142 for details.

    • Values: IGNORE, DELETE
  • removedJobAction (optional)
    Specifies what to do when a previously generated job is not referenced anymore.

    Note: when using multiple Job DSL build steps in a single job, set this to "Delete" or "Disable" only for the last Job DSL build step. Otherwise jobs will be deleted and re-created or disabled and re-enabled and you may lose the job history of generated jobs. See JENKINS-44142 for details.

    • Values: IGNORE, DISABLE, DELETE
  • removedViewAction (optional)
    Specifies what to do when a previously generated view is not referenced anymore.

    Note: when using multiple Job DSL build steps in a single job, set this to "Delete" only for the last Job DSL build step. Otherwise views may be deleted and re-created. See JENKINS-44142 for details.

    • Values: IGNORE, DELETE
  • sandbox : boolean (optional)
    If checked, runs the DSL scripts in a sandbox with limited abilities. You will also need to configure this job to run with the identity of a particular user. If unchecked, and you are not a Jenkins administrator, you will need to wait for an administrator to approve the scripts.
  • scriptText : String (optional)
    DSL Script, which is groovy code. Look at documentation for details on the syntax.
  • targets : String (optional)
    Newline separated list of DSL scripts, located in the Workspace. Can use wildcards like 'jobs/**/*.groovy'. See the @includes of Ant fileset for the exact format.

    Scripts are executed in the same order as specified. The execution order of expanded wildcards is unspecified.

  • unstableOnDeprecation : boolean (optional)
    If checked, marks the build as unstable when using deprecated features. If not checked, a warning will be printed to the build log only.
  • useScriptText : boolean (optional)

https://github.com/jenkinsci/job-dsl-plugin/blob/master/docs/User-Power-Moves.md#use-job-dsl-in-pipeline-scripts

Starting with version 1.48, the Job DSL build step can be used in Pipeline scripts (e.g. in Jenkinsfile). In version 1.49 Pipeline support has been improved by enabling a more concise syntax when using Pipeline: Groovy 2.10 or later.

Pipeline syntax with version 1.49 and Pipeline: Groovy 2.10 or later:

node {
    jobDsl scriptText: 'job("example-2")'

    jobDsl targets: ['jobs/projectA/*.groovy', 'jobs/common.groovy'].join('\n'),
           removedJobAction: 'DELETE',
           removedViewAction: 'DELETE',
           lookupStrategy: 'SEED_JOB',
           additionalClasspath: ['libA.jar', 'libB.jar'].join('\n'),
           additionalParameters: [message: 'Hello from pipeline', credentials: 'SECRET']
}

 

Options:

  • targets: optional, specifies Job DSL script files to execute, newline separated list of file names relative to the workspace
  • scriptText: optional, specifies an inline Job DSL script
  • ignoreMissingFiles: optional, defaults to false, set to true to ignore missing files or empty wildcards in targets
  • ignoreExisting: optional, defaults to false, set to true to not update existing jobs and views
  • removedJobAction: optional, set to 'DELETE' or 'DISABLE' to delete or disable jobs that have been removed from DSL scripts, defaults to 'IGNORE'
  • removedViewAction: optional, set to 'DELETE' to delete views that have been removed from Job DSL scripts, defaults to 'IGNORE'
  • removedConfigFilesAction: optional, set to 'DELETE' to delete config files that have been removed from Job DSL scripts, defaults to 'IGNORE'
  • lookupStrategy: optional, when set to 'SEED_JOB' job names will be interpreted as relative to the pipeline job, defaults to 'JENKINS_ROOT which will treat all job names as absolute
  • additionalClasspath: optional, newline separated list of additional classpath entries for Job DSL scripts, file names must be relative to the workspace; this option will be ignored when script security for Job DSL is enabled on the "Configure Global Security" page
  • additionalParameters: optional map which defines global variables accessible in Job DSL scripts
  • sandbox: optional, defaults to false, if false the DSL script needs to be approved by an administrator; set to true to run the DSL scripts in a sandbox with limited abilities (see [[Script Security]]); this option will be ignored when script security for Job DSL is disabled on the "Configure Global Security" page

multibranch

https://github.com/jenkinsci/job-dsl-plugin/wiki/Tutorial---Using-the-Jenkins-Job-DSL

To show some more of the power of the DSL Plugin, let's create a bunch more Jobs.

  • Go back to the 'tutorial-job-dsl-1' Seed Job
  • Click the "Configure" link/button and navigate back down the the "Process Job DSLs" build step.
  • Add the following into the text box, below the script which we added at the beginning.

(Note: The practicality of this block is questionable, but it could be used to shard your tests into different jobs.)

def project = 'quidryan/aws-sdk-test'
def branchApi = new URL("https://api.github.com/repos/${project}/branches")
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
    def branchName = it.name
    def jobName = "${project}-${branchName}".replaceAll('/','-')
    job(jobName) {
        scm {
            git("git://github.com/${project}.git", branchName)
        }
        steps {
            maven("test -Dproject.name=${project}/${branchName}")
        }
    }
}

 

Jenkinsfile弊端一点

https://www.jenkins.io/doc/book/pipeline/syntax/#parameters

Jenkinsfile也是面向Job的全部内容， 包括Job的一些参数配置，

例如 用户输入界面的参数。

 

此Jenkinsfile内容更新后， 并发发布到Jenkins对应的Job后，

参数的变更并不会立刻生效，需要Job运行一次后，才能生效。

但是Job一旦运行， 其对应的 业务逻辑代码也 执行的一次，

这不是部署工作希望看到的。

 

可以使用一些tricky的方法规避此问题， 例如部署的build带特定的参数， 在stage中使用when语句做条件判断。

解法实在ugly。

 

pipeline {
    agent any
    parameters {
        string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?')

        text(name: 'BIOGRAPHY', defaultValue: '', description: 'Enter some information about the person')

        booleanParam(name: 'TOGGLE', defaultValue: true, description: 'Toggle this value')

        choice(name: 'CHOICE', choices: ['One', 'Two', 'Three'], description: 'Pick something')

        password(name: 'PASSWORD', defaultValue: 'SECRET', description: 'Enter a password')
    }
    stages {
        stage('Example') {
            steps {
                echo "Hello ${params.PERSON}"

                echo "Biography: ${params.BIOGRAPHY}"

                echo "Toggle: ${params.TOGGLE}"

                echo "Choice: ${params.CHOICE}"

                echo "Password: ${params.PASSWORD}"
            }
        }
    }
}

 

 

JobDSL+PipelineJob 解法

https://github.com/matschaffer/apm-pipeline-library/blob/0a5e76259ac31b7df048c9920fdc1261556748f5/.ci/fleet-ci-jobdsl-generator.groovy

构建一个pipeline， 调用job dsl语句，

运行job dsl定义文件， 生成pipeline job。

这样运行此pipeline，可以做配置修改，并更新pipeline脚本， 但是并不运行pipeline脚本。

pipeline {
  agent { label 'master' }
  environment {
    REPO = 'apm-pipeline-library'
    BASE_DIR = "src/github.com/elastic/${env.REPO}"
    NOTIFY_TO = credentials('notify-to')
    PIPELINE_LOG_LEVEL = 'DEBUG'
    LANG = "C.UTF-8"
    LC_ALL = "C.UTF-8"
    SLACK_CHANNEL = '#observablt-bots'
    GITHUB_CHECK = 'true'
    BRANCH_NAME = "${params.branch_specifier}"
  }
  options {
    timeout(time: 1, unit: 'HOURS')
    buildDiscarder(logRotator(numToKeepStr: '5', artifactNumToKeepStr: '5'))
    timestamps()
    ansiColor('xterm')
    disableResume()
    durabilityHint('PERFORMANCE_OPTIMIZED')
  }
  parameters {
    string(name: 'branch_specifier', defaultValue: 'main', description: 'the Git branch specifier to scan.')
  }
  stages {
    stage('Checkout jobs'){
      steps {
        deleteDir()
        gitCheckout(basedir: "${BASE_DIR}")
      }
    }
    stage('Unit test'){
      when {
        expression {
          return false
        }
      }
      steps {
        dir("${BASE_DIR}/.ci/jobDSL"){
          sh(label: 'Run tests', script: './gradlew clean test --stacktrace')
        }
      }
      post {
        always {
          junit(allowEmptyResults: true,
            testDataPublishers: [
              [$class: 'AttachmentPublisher']
            ],
            testResults:"${BASE_DIR}/.ci/jobDSL/build/test-results/test/TEST-*.xml"
          )
        }
      }
    }
    stage('Generate Jobs') {
      steps {
        jobDsl(
          ignoreExisting: true,
          failOnMissingPlugin: true,
          failOnSeedCollision: true,
          sandbox: true,
          targets: "${BASE_DIR}/.ci/jobDSL/jobs/fleet-ci/folder.groovy",
          unstableOnDeprecation: true
        )
        jobDsl(
          ignoreExisting: true,
          failOnMissingPlugin: true,
          failOnSeedCollision: true,
          removedConfigFilesAction: 'DELETE',
          removedJobAction: 'DELETE',
          removedViewAction: 'DELETE',
          sandbox: true,
          targets: "${BASE_DIR}/.ci/jobDSL/jobs/fleet-ci/**/*.groovy",
          unstableOnDeprecation: true
        )
      }
    }
  }
}

 

调用如下job DSL文件，生成一个pipeline job，做好参数配置，更新pipeline脚本

https://github.com/matschaffer/apm-pipeline-library/blob/0a5e76259ac31b7df048c9920fdc1261556748f5/.ci/jobDSL/jobs/fleet-ci/fleet-shared/fleet_schedule_daily.groovy

pipelineJob("fleet-shared/fleet-schedule-daily") {
  displayName('Jobs scheduled daily')
  description('Jobs scheduled daily from Monday to Friday.')
  disabled(false)
  quietPeriod(10)
  logRotator {
    numToKeep(10)
    daysToKeep(7)
    artifactNumToKeep(10)
    artifactDaysToKeep(-1)
  }
  parameters {
    stringParam("branch_specifier", "main", "the Git branch specifier to build.")
  }
  definition {
    cpsScm {
      scm {
        git {
          remote {
            github("elastic/apm-pipeline-library", "ssh")
            credentials("f6c7695a-671e-4f4f-a331-acdce44ff9ba")
          }
          branch('${branch_specifier}')
          extensions {
            wipeOutWorkspace()
          }
        }
      }
      lightweight(false)
      scriptPath(".ci/fleet-ci-schedule-daily.groovy")
    }
  }
}

 

pipeline脚本来自：

https://github.com/matschaffer/apm-pipeline-library/blob/0a5e76259ac31b7df048c9920fdc1261556748f5/.ci/fleet-ci-schedule-daily.groovy

@Library('apm@current') _

pipeline {
  agent none
  environment {
    NOTIFY_TO = credentials('notify-to')
    PIPELINE_LOG_LEVEL = 'INFO'
  }
  options {
    timeout(time: 1, unit: 'HOURS')
    buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '20'))
    timestamps()
    ansiColor('xterm')
    disableResume()
    durabilityHint('PERFORMANCE_OPTIMIZED')
  }
  triggers {
    cron('H H(4-5) * * 1-5')
  }
  stages {
    stage('Run Tasks'){
      steps {
        build(job: 'fleet-shared/gather-ci-metrics-pipeline',
          parameters: [
            string(name: 'MTIME_FILTER', value: '1'),
            string(name: 'AGENT_PREFIX', value: 'fleet-ci')
          ],
          propagate: false,
          wait: false
        )
      }
    }
  }
  post {
    cleanup {
      notifyBuildResult()
    }
  }
}

 

上面例子使用pipeline做jobDSL生成job

实际上如jobDSL中教程所说， 可以手动创建一个seed JOB

更有应用在jenkins启动过程中，调用jobDSL插件接口自动生成pipelineJOB

参考：

https://github.com/andreschaffer/one-click-microservice/tree/master

https://github.com/thbkrkr/jks

 

Job DSL 和 Jenkinsfile写在一起

https://souravatta.medium.com/create-jenkins-seedjob-using-job-dsl-script-ac337afd9986

pipelineJob("greetingJob") {
  
  parameters {
         stringParam('name', "", 'name of the person')
        }  definition {
           cps {
             script('''
                 pipeline {
                    agent any                    stages {
                        stage('Greet') {
                            steps {
                                echo "Hello!! ${name}"
                            }
                         }
                      }
                   }
              '''.stripIndent())       sandbox()
          }
      }
  }

 

posted @ 2023-08-09 17:21  lightsong  阅读(77)  评论(0)  编辑  收藏  举报
Life Is Short, We Need Ship To Travel