交付dubbo 微服务到k8s 集群
dubbo 微服务结构简介
上图中，蓝色的表示与业务有交互，绿色的表示只对Dubbo内部交互。上述图所描述的调用流程如下：
- 服务提供方发布服务到服务注册中心；
- 服务消费方从服务注册中心订阅服务；
- 服务消费方调用已经注册的可用服务
1. 部署maven 并改成国内源
在 7.200 上： tar xf apache-maven-3.6.0-bin.tar.gz -C /data/nfs-volume/jenkins_home/maven-3.6.3
修改maven 仓库为阿里云仓库
<mirrors>
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>*</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
</mirrors>
#修改成阿里云镜像源
2. 准备 dubbo 底层镜像
docker pull docker.io/stanleyws/jre8:8u112 #这个是jdk8的原始镜像
docker tag docker.io/stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
docker push harbor.od.com/public/jre:8u112
cat <<EOF >/data/dockerfile/jre8/Dockerfile
FROM harbor.od.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo 'Asia/Shanghai' >/etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
EOF
cat <<EOF>config.yml
---
rules:
- pattern: '.*'
EOF
#下载java-agent 用来给普罗米修斯监控
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar
vim entrypoint.sh
#!/bin/sh
M_OPTS="-Duser.timezone=Asia/Shanghai-javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname-i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
chmod +x entrypoint.sh
#登录harbor 创建base 仓库，权限是public
]# docker build . -t harbor.od.com/base/jre8:8u112 && docker push harbor.od.com/base/jre8:8u112
3.交付dubbo server 到k8s 集群
在 jenkins 中新建 一个pipline的项目，名字自定义，选择参数构建
#然后到Jenkins 新建pipline 工程 ，在general ， 选择 this project isparameterized。
选择string parameter 。勾选 trim is string ,分别是：
app_name image_name git_repo git_ver add_tag target_dir (./target) mvn_dir (./)
mvn_cmd (mvn clean package -Dmaven.test.skip=true)
# Choice parameter
base_image (base/jre8:8u112)
maven (3.6.3) #执行编译的软件包版本,要根据实际mvn 写。
以此填入 自定义的参数，我这里放个填好后的图，就是 app_name image_name 那一列：
然后在jenkins pipeline 下面填入pipeline 脚本
pipeline {
agent any
stages {
stage('pull') { //get project code from repo
steps {
sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
}
}
stage('build') { //exec mvn cmd
steps {
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
}
}
stage('package') { //move jar file into project_dir
steps {
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
}
}
stage('image') { //build image and push to registry
steps {
writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""#这一步是把 FROM harbor.od.com/${params.base_image}
#ADD ${params.target_dir}/project_dir /opt/project_dir 具体内容写到jenkins 里面dockerfile 里面 然后在jenkins 里面构建有了新的jar 包的镜像 并推送到
#habor 上，镜像的名字后缀是刚才的add_tag 也就是时间
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
}
}
}
}
保存，之后再项目项目里面选择参数构建：
具体的参数如下：
git_repo 要填dubbo-server 的git 仓库地址 ，git_ver 就是代码分支，这里拉取的 apolo 分支的代码。
add_tag 建议用时间日期 base_image 就是上一步构建的dockerfile 基础镜像 maven 就是之前部署的，名字要和 7.200 /data/nfs-volume/jenkins_home/ 里面 maven 的目录名字一致，要不jenkins 构建时候会找不到，类似如下报错
+ /var/jenkins_home/maven-3.6.3/bin/mvn clean package -Dmaven.test.skip=true /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: 1: /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: /var/jenkins_home/maven-3.6.3/bin/mvn: not found
然后点击构建，成功后结果如下，这时候登陆harbor 就会发现多了 一个
app/dubbo-demo-service:apollo_20200517_1600 的镜像了！
4. 准备dubbo server 的yaml 文件
[root@hdss7-200 dubbo-server]# cat dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: app
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
image: harbor.od.com/app/dubbo-demo-service:apollo_20200517_1600 #就是刚才新构建的镜像
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.od.com #这里结合apolo 配置中心来做的
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
5 交付dubbo-monitor 到k8s 集群
#先下载dubbo monitor 包
https://github.com/Jeromefromcn/dubbo-monitor#dubbo-monitor ，下载后解压，
修改配置
cat /opt/src/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties dubbo.container=log4j,spring,registry,jetty dubbo.application.name=simple-monitor dubbo.application.owner= dubbo.registry.address=zookeeper://10.4.7.11:2181 # 改成实际zk 地址 dubbo.protocol.port=20880 dubbo.jetty.port=8080 dubbo.jetty.directory=/dubbo-monitor-simple/monitor dubbo.statistics.directory=/dubbo-monitor-simple/statistics dubbo.charts.directory=/dubbo-monitor-simple/charts dubbo.log4j.file=logs/dubbo-monitor.log
准备dockerfile
vim /data/dockerfile/dubbo-monitor/Dockerfile FROM jeromefromcn/docker-alpine-java-bash MAINTAINER Jerome Jiang COPY dubbo-monitor-simple/ /dubbo-monitor-simple/ #dubbo-monitor-simple/ 就是dubbo-monitor 包里面的文件夹 CMD /dubbo-monitor-simple/bin/start.sh
docker build . -t harbor.od.com/infra/dubbo-monitor:latest #构建镜像
因为github 上 start.sh 脚本在容器环境中启动有问题 需要修改，下面是修改后的。
[root@hdss7-200 dubbo-monitor]# cat dubbo-monitor-simple/bin/start.sh
#!/bin/bash
sed -e "s/{ZOOKEEPER_ADDRESS}/$ZOOKEEPER_ADDRESS/g" /dubbo-monitor-simple/conf/dubbo_origin.properties > /dubbo-monitor-simple/conf/dubbo.properties
cd `dirname $0`
BIN_DIR=`pwd`
cd ..
DEPLOY_DIR=`pwd`
CONF_DIR=$DEPLOY_DIR/conf
SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`
SERVER_PROTOCOL=`sed '/dubbo.protocol.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`
SERVER_PORT=`sed '/dubbo.protocol.port/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`
LOGS_FILE=`sed '/dubbo.log4j.file/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`
if [ -z "$SERVER_NAME" ]; then
SERVER_NAME=`hostname`
fi
PIDS=`ps -f | grep java | grep "$CONF_DIR" |awk '{print $2}'`
if [ -n "$PIDS" ]; then
echo "ERROR: The $SERVER_NAME already started!"
echo "PID: $PIDS"
exit 1
fi
if [ -n "$SERVER_PORT" ]; then
SERVER_PORT_COUNT=`netstat -tln | grep $SERVER_PORT | wc -l`
if [ $SERVER_PORT_COUNT -gt 0 ]; then
echo "ERROR: The $SERVER_NAME port $SERVER_PORT already used!"
exit 1
fi
fi
LOGS_DIR=""
if [ -n "$LOGS_FILE" ]; then
LOGS_DIR=`dirname $LOGS_FILE`
else
LOGS_DIR=$DEPLOY_DIR/logs
fi
if [ ! -d $LOGS_DIR ]; then
mkdir $LOGS_DIR
fi
STDOUT_FILE=$LOGS_DIR/stdout.log
LIB_DIR=$DEPLOY_DIR/lib
LIB_JARS=`ls $LIB_DIR|grep .jar|awk '{print "'$LIB_DIR'/"$0}'|tr "\n" ":"`
JAVA_OPTS=" -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true "
JAVA_DEBUG_OPTS=""
if [ "$1" = "debug" ]; then
JAVA_DEBUG_OPTS=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n "
fi
JAVA_JMX_OPTS=""
if [ "$1" = "jmx" ]; then
JAVA_JMX_OPTS=" -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false "
fi
JAVA_MEM_OPTS=""
BITS=`java -version 2>&1 | grep -i 64-bit`
if [ -n "$BITS" ]; then
JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=32m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi
echo -e "Starting the $SERVER_NAME ...\c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1
6. 准备dobbo monitor 文件
cat deployment.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
image: harbor.od.com/infra/dubbo-monitor:latest
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
[root@hdss7-200 dubbo-monitor]# cat cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dubbo-monitor-cm
namespace: infra
data:
dubbo.properties: |
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=simple-monitor
dubbo.application.owner=simplemonitor
dubbo.registry.address=zookeeper://10.4.7.11:2181 #zk 的地址一定要和代码里面保持一致！
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
dubbo.log4j.level=WARN
[root@hdss7-200 dubbo-monitor]# cat ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
spec:
rules:
- host: dubbo-monitor.od.com
http:
paths:
- path: /
backend:
serviceName: dubbo-monitor
servicePort: 8080
[root@hdss7-200 dubbo-monitor]# cat svc.yaml
kind: Service
apiVersion: v1
metadata:
name: dubbo-monitor
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: dubbo-monitor
添加A 记录 dubbo-monitor.od.com IN A 10.4.7.10 测试访问 http://dubbo-monitor.od.com
7.交付 dubbo服务消费者到K8S 集群
jenkins 上新构建一个pipeline 项目，添加的参数和dubbo-serve 一样,，可以直接把dubbo-server 复制过来，然后填入参数 开始构建，参数如下图：
看下 console output
Step 1/2 : FROM harbor.od.com/base/jre8:8u112 ---> 621a01f7f85a Step 2/2 : ADD ./dubbo-client/target/project_dir /opt/project_dir ---> 7d5577992ecd Successfully built 7d5577992ecd Successfully tagged harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 + docker push harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 The push refers to repository [harbor.od.com/app/dubbo-demo-consumer] 会看到harbor 上多了 app/dubbo-demo-consumer:apollo_20200517_1650 的镜像
8. 准备consumer 的yaml 文件
# dubbo-consumer 资源配置清单
dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 #就是刚才新构建的consumer 镜像
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-client.jar
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
cat svc.yaml
kind: Service
apiVersion: v1
metadata:
name: dubbo-demo-consumer
namespace: app
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: dubbo-demo-consumer
cat ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
spec:
rules:
- host: demo.od.com
http:
paths:
- path: /
backend:
serviceName: dubbo-demo-consumer
servicePort: 8080
kubectl apply -f 这几个文件，然后 访问 http://demo.od.com/hello?name=hello 看是否有返回结果，有返回就是正常的。
登录monitor 查看： 可以看到 server 和 consumer 都有了。
浙公网安备 33010602011771号