「CICD实践」基于Jenkins-job自由风格的python-Flask应用自动构建Docker镜像、运行容器

1、环境准备
2、创建Flask工程
3、创建Jenkins-Job流程

1、环境准备

1、Mac环境: Monterey 12.6
2、Docker: version 20.10.21
3、GitHub
4、Jenkins:  2.387.2: http://localhost:8082/
5、Python: 3.7
6、Flask
7、Pycharm
8、Jenkins插件:docker-build-step、version Number

2、创建Flask工程

https://github.com/kenwucode4fs/TechPractiseByPy37

这里涉及到一个知识点, 就是Github的使用. 其中因为后续我们要使用Jenkins部署. 所有有两个概念我们要区分清楚, 一个是 github的SSH Key, 另一个是 Deploy keys. 这里SSH key相当于最高级别的钥匙, 而Deploy keys相当于针对工程的钥匙. 所以我们这里需要设置一下.

2.1、配置GIT username 和email

(base)  ✘ kenwu@KenMBP-2  ~  git config --global user.name "kenwucode4fs"
(base)  kenwu@KenMBP-2  ~  git config --global user.email "370483689@qq.com"

2.2、生成SSH KEY

因为我已经存在. 所以这里覆盖重写.

(base)  kenwu@KenMBP-2  ~/.ssh  ssh-keygen -t rsa -C "370483689@qq.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/kenwu/.ssh/id_rsa):
/Users/kenwu/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/kenwu/.ssh/id_rsa
Your public key has been saved in /Users/kenwu/.ssh/id_rsa.pub
The key fingerprint is:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 370483689@qq.com
The key's randomart image is:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
(base)  kenwu@KenMBP-2  ~/.ssh 

2.3、生成Deploy KEY

(base)  ✘ kenwu@KenMBP-2  ~/.ssh  ssh-keygen -t rsa -C "370483689@qq.com" -f ~/.ssh/TechPractiseByPy37_id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/kenwu/.ssh/TechPractiseByPy37_id_rsa
Your public key has been saved in /Users/kenwu/.ssh/TechPractiseByPy37_id_rsa.pub
The key fingerprint is:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The key's randomart image is:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
(base)  kenwu@KenMBP-2  ~/.ssh 

2.4、在~/.ssh/目录下添加config文件

(base)  kenwu@KenMBP-2  ~/.ssh  pwd
/Users/kenwu/.ssh
(base)  kenwu@KenMBP-2  ~/.ssh  tree
.
├── TechPractiseByPy37_id_rsa
├── TechPractiseByPy37_id_rsa.pub
├── config
├── id_rsa
├── id_rsa.pub
├── known_hosts
└── known_hosts.old

(base)  kenwu@KenMBP-2  ~/.ssh  cat config
Host github.com
    HostName github.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa

Host TechPractiseByPy37.github.com
    HostName github.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/TechPractiseByPy37_id_rsa
(base)  kenwu@KenMBP-2  ~/.ssh 

2.5、配置SSH KEY

id_rsa.pub(公钥) 中的内容添加至远端github 仓库中

2.6、配置 Deploy Key

TechPractiseByPy37_id_rsa.pub(公钥) 中的内容添加至远端github 仓库中

2.7、测试配置是否正确

(base)  kenwu@KenMBP-2  ~/.ssh  ssh -T git@github.com
Hi kenwucode4fs! You've successfully authenticated, but GitHub does not provide shell access.

(base)  kenwu@KenMBP-2  ~/.ssh  ssh -T git@TechPractiseByPy37.github.com
Hi kenwucode4fs/TechPractiseByPy37! You've successfully authenticated, but GitHub does not provide shell access.

2.8、创建一个最小的Flask 应用

使用Pycharm打开该目录, 同时创建一个Project及虚拟环境, 这里关于环境的创建, Python虚拟环境就暂不叙述, 如果有需要的同学, 可以在Flask官网看一下. 官网传送门:https://flask.palletsprojects.com/en/2.2.x/installation/#virtual-environments

# 创建虚拟环境
(base)  ✘ kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  python3 -m venv venv 
# 激活虚拟环境
(base)  ✘ kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  . venv/bin/activate
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  pip list
Package    Version
---------- -------
pip        20.1.1
setuptools 47.1.0

# 安装flask包
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  pip install Flask
Collecting Flask
  Downloading Flask-2.2.3-py3-none-any.whl (101 kB)
     |████████████████████████████████| 101 kB 158 kB/s 
Collecting Jinja2>=3.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 22 kB/s 
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting importlib-metadata>=3.6.0; python_version < "3.10"
  Downloading importlib_metadata-6.1.0-py3-none-any.whl (21 kB)
Collecting click>=8.0
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 26 kB/s 
Collecting Werkzeug>=2.2.2
  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
     |████████████████████████████████| 233 kB 22 kB/s 
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.2-cp37-cp37m-macosx_10_9_x86_64.whl (13 kB)
Collecting zipp>=0.5
  Downloading zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Installing collected packages: MarkupSafe, Jinja2, itsdangerous, zipp, typing-extensions, importlib-metadata, click, Werkzeug, Flask
Successfully installed Flask-2.2.3 Jinja2-3.1.2 MarkupSafe-2.1.2 Werkzeug-2.2.3 click-8.1.3 importlib-metadata-6.1.0 itsdangerous-2.1.2 typing-extensions-4.5.0 zipp-3.15.0
WARNING: You are using pip version 20.1.1; however, version 23.0.1 is available.
You should consider upgrading via the '/Users/kenwu/Documents/TechSet/TechPractiseByPy37/venv/bin/python3 -m pip install --upgrade pip' command.

# 查看已经安装的包
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  pip list
Package            Version
------------------ -------
click              8.1.3
Flask              2.2.3
importlib-metadata 6.1.0
itsdangerous       2.1.2
Jinja2             3.1.2
MarkupSafe         2.1.2
pip                20.1.1
setuptools         47.1.0
typing-extensions  4.5.0
Werkzeug           2.2.3
zipp               3.15.0

# 这里留一个扩展. 其实在以后的python项目开发中, 我可能会尽量使用conda + pip来管理环境. 便于整体的一个环境包迁移. 

# 生成包迁移文件
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  pip freeze >requirements.txt
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  ll
total 16
-rw-r--r--  1 kenwu  staff   118B Apr  3 14:45 README.md
-rw-r--r--  1 kenwu  staff   158B Apr  3 15:23 requirements.txt
drwxr-xr-x  6 kenwu  staff   192B Apr  3 15:17 venv
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main  cat requirements.txt 
click==8.1.3
Flask==2.2.3
importlib-metadata==6.1.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.2
typing-extensions==4.5.0
Werkzeug==2.2.3
zipp==3.15.0

添加techpractiserun.py文件到Git中:

# -*- coding: utf-8 -*-

"""
@File:         techpractiserun
@Description:  
@Author:       kenwu
@Date:         2023/4/3
@Email: 370483689@qq.com
"""
from flask import Flask

app = Flask(__name__)


@app.route('/hello')
def hello():
    """
    健康检查，hello接口
    :return:
    """
    return "<p>Hello, This is TechPractiseByPy37!</p>"



if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8082, debug=app.config['DEBUG'])

2.9、开发者模式运行Flask服务

(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main ●✚  python techpractiserun.py
 * Serving Flask app 'techpractiserun'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8082
 * Running on http://10.74.28.31:8082
Press CTRL+C to quit
127.0.0.1 - - [03/Apr/2023 15:38:14] "GET /hello HTTP/1.1" 200 -

2.10、生产环境运行Flask

因为我们最终的目的是要在生产环境中直接使用Docker运行这个Flask工程镜像, 所以在这里我们直接采用gunicorn. 对于gunicorn, 实际上这个就是一个Python WSGI UNIX HTTP服务器, 详细的资料可以查阅传送门: https://docs.gunicorn.org/en/stable/

接下来我们使用gunicorn在本地运行一下这个flask工程.

# 安装gunicorn
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main ●✚  pip install gunicorn
# 安装gevent库
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main ●✚  pip install gevent
# 生成requirements.txt文件
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main ●✚  pip freeze >requirements.txt                    
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main ●✚  pip list
Package            Version
------------------ -------
click              8.1.3
Flask              2.2.3
gevent             22.10.2
greenlet           2.0.2
gunicorn           20.1.0
importlib-metadata 6.1.0
itsdangerous       2.1.2
Jinja2             3.1.2
MarkupSafe         2.1.2
pip                20.1.1
setuptools         47.1.0
typing-extensions  4.5.0
Werkzeug           2.2.3
zipp               3.15.0
zope.event         4.6
zope.interface     6.0
WARNING: You are using pip version 20.1.1; however, version 23.0.1 is available.
You should consider upgrading via the '/Users/kenwu/Documents/TechSet/TechPractiseByPy37/venv/bin/python3 -m pip install --upgrade pip' command.

#gunicorn.conf.py
# -*- coding: utf-8 -*-

"""
@File:         gunicorn.conf
@Description:  
@Author:       kenwu
@Date:         2023/4/3
@Email: 370483689@qq.com
"""
import multiprocessing

bind = "0.0.0.0:8002"
workers = 2  # workers是工作进程数，一般设置成：服务器CPU个数 + 1
worker_class = "gevent" # 采用gevent库，支持异步处理请求，提高吞吐量
daemon = False
loglevel = 'info'
access_log_format = '%(t)s %(p)s %(h)s "%(r)s" %(s)s %(L)s %(b)s %(f)s" "%(a)s"'
errorlog = 'logs/gunicorn.error.log'
accesslog = 'logs/gunicorn.access.log'
debug = False

启动命令:

gunicorn -c gunicorn.conf.py techpractiserun:app

那么到目前为止, 实际上我们已经能够正常的通过Gunicorn运行一个Flask应用.

3、创建Jenkins-Job流程

在创建Jenkins的Job的时候, 我首先考虑到的点是:
代码、配置、镜像的分离.
如果按照一个持续集成系统的角度来考虑,那么我考虑的点是:
1、镜像实际上就应该只包含基础信息.
2、代码及配置应该在后续映射到镜像中.
这样才能够实现代码、配置、镜像的完全分离及「版本化管理」. 所以我觉得很有必要对上面的构建镜像启动服务的过程进行一个改造

3.1、设计部署服务器与容器之间的目录映射——[目前暂未按照该模式操作]

1、配置下载并放入指定位置映射容器地址
部署服务器地址:/data/datakw/config_center/项目名称/conf_服务模块名称/应用配置文件
容器地址:/kenwu/kenwuapp/conf

2、应用启动脚本下载并放入指定位置映射容器地址
部署服务器地址:/data/datakw/config_center/项目名称/服务模块名称/start.sh
容器地址:/app/bin/start.sh

3、包/源码下载放入指定位置映射容器地址
部署服务器地址:/data/datakw/apps/项目名称/服务模块名称/包名或者源代码
容器地址:/kenwu/kenwuapp/app

4、日志目录映射容器目录
部署服务器地址: /data/datakw/applog/项目名称/服务模块名称/logs
容器地址:/applog

5、NAS目录映射
宿主机地址:/data/datakw/nasfiles/
容器地址:/kenwu/kenwuapp/appfile/

3.2、Jenkins Job拉取Flask工程代码到本地

3.2.1、jenkins配置部署TechPractiseByPy37所需Deploy key

此处密码使用之前生成的私钥: cat ~/.ssh/TechPractiseByPy37_id_rsa

注意内容要包含:这两个标签-----BEGIN OPENSSH PRIVATE KEY-----与-----END OPENSSH PRIVATE KEY-----

3.2.2、创建一个自由风格的Job

注意在这里, 修改了Jenkins的工作目录/.jenkins/config.xml文件:

 <workspaceDir>/Users/kenwu/Documents/TechSet/data-jenkins/localbrewjenkins/workspace/${ITEM_FULL_NAME}</workspaceDir>

3.2.3、立即构建

3.3、Jenkins Job编译容器镜像代码到本地

1、从GitHub中拉取TechPractiseByPy37 Dockerfile 以及python requirements.txt
2、根据Dockerfile build一个镜像
3、拷贝下载的python requirements.txt
4、生成容器并运行

3.3.1、「测试版」创建Dockerfile文件

FROM python:3.7.9
WORKDIR /var/app

COPY requirements.txt ./

RUN pip install -r requirements.txt

CMD ["gunicorn", "-c", "gunicorn.conf.py", "techpractiserun:app"]
EXPOSE 8002

# 由于我们需要运行容器的时候，直接映射目录，所以不需要把app代码复制到容器中

3.3.2、本地测试构建镜像脚本

# 构建镜像
(venv) (base)  kenwu@KenMBP-2  ~/Documents/TechSet/TechPractiseByPy37   main ●✚  docker build -t techpractise/py37:v1 . 
Sending build context to Docker daemon  27.88MB
Step 1/6 : FROM python:3.7.9
 ---> 65d5b6c539fd
Step 2/6 : WORKDIR /var/app
 ---> Running in 6a67fcb9851d
Removing intermediate container 6a67fcb9851d
 ---> 1e481370faed
Step 3/6 : COPY requirements.txt ./
 ---> 07891ac585e1
Step 4/6 : RUN pip install -r requirements.txt
 ---> Running in a115d35dd521
Collecting click==8.1.3
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
Collecting Flask==2.2.3
  Downloading Flask-2.2.3-py3-none-any.whl (101 kB)
Collecting gevent==22.10.2
  Downloading gevent-22.10.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.0 MB)
  ...
  
# 查看镜像
(base)  kenwu@KenMBP-2  ~/Documents/TechSet  docker images -a
REPOSITORY                    TAG          IMAGE ID       CREATED          SIZE
<none>                        <none>       f05b035ec488   13 seconds ago   920MB
techpractise/py37             v1           0be74153aa3b   13 seconds ago   920MB
<none>                        <none>       2aa5b9a7e45c   13 seconds ago   920MB
<none>                        <none>       07891ac585e1   4 minutes ago    877MB
<none>                        <none>       1e481370faed   4 minutes ago    877MB
jenkins/jenkins               latest       2fef055370ae   5 days ago       471MB

# 启动容器(映射代码路径)
(base)  kenwu@KenMBP-2  ~/Documents/TechSet  docker run -itd --name techpractisebypy37 -v /Users/kenwu/Documents/TechSet/TechPractiseByPy37:/var/app -p 8002:8002 techpractise/py37:v1
af4e3665fe6d191e761877d23c65c01ce749d045926fd64b67867ece87192138

# 查看启动的容器服务
(base)  kenwu@KenMBP-2  ~/Documents/TechSet  docker ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS                     PORTS                                              NAMES
af4e3665fe6d   techpractise/py37:v1   "gunicorn -c gunicor…"   4 seconds ago   Up 3 seconds               0.0.0.0:8002->8002/tcp                             techpractisebypy37

检查服务:

3.3.3、Jenkins 安装插件docker-build-step

3.3.4、Jenkins 安装插件 version Number

3.3.5、Jenkins系统配置: Docker Builder.

因为我们docker在Jenkins服务器所以填写:unix:///var/run/docker.sock, 如果使用另一台docker服务器作为镜像构建服务器, 需要填写docker api接口信息.

3.3.6、「测试版」修改之前的job:TechPractiseByPy37_GetCodeFromGitHub

最终效果:

3.3.7、「最终版」修改Dockerfile及gunicorn.conf.py

考虑到我们在运行服务的时候需要使用非root用户. 同时日志目录也需要做映射. 所以针对Dockerfile及gunicorn.conf.py做了对应的修改

FROM python:3.7.9

# 创建目录
WORKDIR /kenwu/app
COPY requirements.txt ./

# 安装第三方包
RUN pip install --upgrade pip \
    && pip install -r requirements.txt \
    && groupadd -r kenwu \
    && useradd -r -g kenwu kenwu \
    && mkdir -p /logs/app \
    && touch /logs/app/gunicorn.error.log \
    && touch /logs/app/gunicorn.access.log \
    && chown -R kenwu:kenwu /logs \
    && chmod -R 755 /logs

# 暴露端口
EXPOSE 8002

# 使用用户kenwu
USER kenwu
# 启动容器时，执行脚本
ENTRYPOINT ["gunicorn", "-c", "./gunicorn.conf.py", "hello.techpractiserun:app"]

# -*- coding: utf-8 -*-

"""
@File:         gunicorn.conf_hello
@Description:  
@Author:       kenwu
@Date:         2023/4/3
@Email: 370483689@qq.com
"""
import os

project_prefix = os.path.dirname(os.path.realpath(__file__))
bind = "0.0.0.0:8002"
workers = 2  # workers是工作进程数，一般设置成：服务器CPU个数 + 1
worker_class = "gevent"  # 采用gevent库，支持异步处理请求，提高吞吐量
daemon = False
loglevel = 'info'
access_log_format = '%(t)s %(p)s %(h)s "%(r)s" %(s)s %(L)s %(b)s %(f)s" "%(a)s"'
errorlog = '/logs/app/gunicorn.error.log'
accesslog = '/logs/app/gunicorn.access.log'

debug = False

截止当前我们的代码目录结构如下:

3.3.8、「最终版」修改之前的job:TechPractiseByPy37_GetCodeFromGitHub

其中构建后执行命令:

docker run --user=kenwu:kenwu --name $PACKAGE_NAME -v $HOST_CODE_PATH:/kenwu/app  -v $HOST_LOGS_PATH:/logs/app -p $HOST_PORT:8002  --privileged=true -itd $PACKAGE_NAME:$TEST_TAG.$BUILD_NUMBER

3.3.9、「效果」立即构建

问题: jenkins运行docker命令不存在
解决方案: 
1、找到jenkins运行的机器
2、执行echo $PATH
3、Jenkins添加全局path, 内容来自步骤二中的$PATH内容

立即构建:

posted @ 2023-05-06 11:06 kenwu_code4fs 阅读(383) 评论(0) 收藏举报

刷新页面返回顶部

柔水终成雕刀

Less is more

「CICD实践」基于Jenkins-job自由风格的python-Flask应用自动构建Docker镜像、运行容器

1、环境准备

2、创建Flask工程

2.1、配置GIT username 和email

2.2、生成SSH KEY

2.3、生成Deploy KEY

2.4、在~/.ssh/目录下添加config文件

2.5、配置SSH KEY

2.6、配置 Deploy Key

2.7、测试配置是否正确

2.8、创建一个最小的Flask 应用

2.9、开发者模式运行Flask服务

2.10、生产环境运行Flask

3、创建Jenkins-Job流程

3.1、设计部署服务器与容器之间的目录映射——[目前暂未按照该模式操作]

3.2、Jenkins Job拉取Flask工程代码到本地

3.2.1、jenkins配置部署TechPractiseByPy37所需Deploy key

3.2.2、创建一个自由风格的Job

3.2.3、立即构建

3.3、Jenkins Job编译容器镜像代码到本地

3.3.1、「测试版」创建Dockerfile文件

3.3.2、本地测试构建镜像脚本

3.3.3、Jenkins 安装插件docker-build-step

3.3.4、Jenkins 安装插件 version Number

3.3.5、Jenkins系统配置: Docker Builder.

3.3.6、「测试版」修改之前的job:TechPractiseByPy37_GetCodeFromGitHub

3.3.7、「最终版」修改Dockerfile及gunicorn.conf.py

3.3.8、「最终版」修改之前的job:TechPractiseByPy37_GetCodeFromGitHub

3.3.9、「效果」立即构建

公告

柔水终成雕刀

Less is more

「CICD实践」基于Jenkins-job自由风格的python-Flask应用自动构建Docker镜像、运行容器

1、 环境准备

2、 创建Flask工程

2.1、配置GIT username 和email

2.2、生成SSH KEY

2.3、生成Deploy KEY

2.4、在~/.ssh/目录下添加config文件

2.5、配置SSH KEY

2.6、配置 Deploy Key

2.7、测试配置是否正确

2.8、创建一个最小的Flask 应用

2.9、开发者模式运行Flask服务

2.10、生产环境运行Flask

3、 创建Jenkins-Job流程

3.1、设计部署服务器与容器之间的目录映射——[目前暂未按照该模式操作]

3.2、Jenkins Job拉取Flask工程代码到本地

3.2.1、jenkins配置部署TechPractiseByPy37所需Deploy key

3.2.2、创建一个自由风格的Job

3.2.3、立即构建

3.3、Jenkins Job编译容器镜像代码到本地

3.3.1、「测试版」创建Dockerfile文件

3.3.2、本地测试构建镜像脚本

3.3.3、Jenkins 安装插件docker-build-step

3.3.4、Jenkins 安装插件 version Number

3.3.5、Jenkins系统配置: Docker Builder.

3.3.6、「测试版」修改之前的job:TechPractiseByPy37_GetCodeFromGitHub

3.3.7、「最终版」修改Dockerfile及gunicorn.conf.py

3.3.8、「最终版」修改之前的job:TechPractiseByPy37_GetCodeFromGitHub

3.3.9、「效果」立即构建

公告

1、环境准备

2、创建Flask工程

3、创建Jenkins-Job流程