二进制部署高可用的etcd集群
此文档已过时,仅作参考使用
使用cfssl生成证书
下载cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl_1.6.2_linux_amd64 wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssljson_1.6.2_linux_amd64 wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl-certinfo_1.6.2_linux_amd64
356 mv cfssl_1.6.2_linux_amd64 /usr/local/bin/cfssl 357 mv cfssl-certinfo_1.6.2_linux_amd64 /usr/sbin/certinfo 358 mv cfssljson_1.6.2_linux_amd64 /usr/sbin/cfssljson
可以用以下命令生成默认格式
$ cfssl print-defaults config > ca-config.json $ cfssl print-defaults csr > ca-csr.json
创建ca
- 制作证书模板配置文件
- 证书颁发机构,颁发证书时,需要参考不同模板的要求进行证书创建
[root@controller CA]# cat ca-config.json { "signing": { "default": { "expiry": "87600h" # 默认过期时间 }, "profiles": { # 设置了多个场景,创建时可以-profile=www/client指定 "www": { "expiry": "87600h", # 过期时间 "usages": [ "signing", "key encipherment", "server auth" # server auth指定此证书只能用在服务器端 ] }, "client": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "client auth" # 只能用在客户端 ] }, "all": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "client auth", # 既可以客户端,也可以服务器 "server auth" ] } } } }
证书申请文件
- 申请证书配置文件
- 证书申请者申请证书时,要填写的基本信息
- 我们去办理身份证时向机关提供的基本信息
cat ca-csr.json
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] }
生成CA证书与私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ca:生成文件名称前缀
制作kubernetes证书
gencert: 生成新的key(密钥)和签名证书 -initca:初始化一个新ca -ca:指明ca的证书 -ca-key:指明ca的私钥文件 -config:指明请求证书的json文件 -profile:与-config中的profile对应,是指根据config中的profile段来生成证书的相关信息
创建Etcd证书签名请求文件
cat etcd-csr.json { "CN":"etcd", "hosts": [ "127.0.0.1", "192.168.50.220", "192.168.50.221", "192.168.50.222", "*.kubernetes.master", "localhost", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo":"rsa", "size": 2048 }, "names": [ { "C":"CN", "ST":"BeiJing", "L":"BeiJing", "O":"k8s", "OU":"System" } ] }
知识点:
-
这个证书目前专属于 apiserver,加了一个
*.kubernetes.master
域名以便内部私有 DNS 解析使用(可删除);kubernetes 这几个能不能删掉,答案是不可以的;因为当集群创建好后,default namespace
下会创建一个叫kubenretes
的svc
,有一些组件会直接连接这个 svc 来跟 api 通信的,证书如果不包含可能会出现无法连接的情况;其他几个 kubernetes 开头的域名作用相同 -
hosts
包含的是授权范围,不在此范围的的节点或者服务使用此证书就会报证书不匹配错误。10.254.0.1
是指kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP。
签发证书
384 cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=ca-config.json -profile=all etcd-csr.json | cfssljson -bare etcd
[root@controller etcd]# ll total 16 -rw-r--r-- 1 root root 1281 Jan 16 09:50 etcd.csr -rw-r--r-- 1 root root 595 Jan 16 09:32 etcd-csr.json -rw------- 1 root root 1675 Jan 16 09:50 etcd-key.pem -rw-r--r-- 1 root root 1643 Jan 16 09:50 etcd.pem
使用openssl来查看证书信息
openssl x509 -in server.pem -noout -text
使用cfssl-certinfo来查看证书信息
可以通过命令行来动态设置SAN模块,如设置域名,IP
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-conf.json -profile=client -hostname=www.golang-node1.com,www.golang-node2.com,127.0.0.1,10.211.55.10 node-csr.json | cfssljson -bare node -
部署高可用的etcd集群
参考文档:https://www.cnblogs.com/hahaha111122222/p/16016386.html
下载并分发
#!/bin/bash cd /data/k8s/work source /data/k8s/bin/env.sh wget https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz tar -zxf etcd-v3.3.18-linux-amd64.tar.gz for node_ip in ${ETCD_IPS[@]} do echo ">>> ${node_ip}" scp etcd-v3.3.18-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin ssh root@${node_ip} "chmod +x /data/k8s/bin/*" done
创建启动脚本
etcd-1
]# cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=simple EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/usr/bin/etcd \ --name=${ETCD_NAME} \ --data-dir=${ETCD_DATA_DIR} \ --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \ --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --initial-cluster=${ETCD_INITIAL_CLUSTER} \ --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster-state=new \ --cert-file=/root/cert/etcd.pem \ --key-file=/root/cert/etcd-key.pem \ --peer-cert-file=/root/cert/etcd.pem \ --peer-key-file=/root/cert/etcd-key.pem \ --trusted-ca-file=/root/cert/ca.pem \ --peer-trusted-ca-file=/root/cert/ca.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
etcd-2
]# cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=simple EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/usr/bin/etcd \ --name=${ETCD_NAME} \ --data-dir=${ETCD_DATA_DIR} \ --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \ --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --initial-cluster=${ETCD_INITIAL_CLUSTER} \ --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster-state=new \ --cert-file=/root/cert/etcd.pem \ --key-file=/root/cert/etcd-key.pem \ --peer-cert-file=/root/cert/etcd.pem \ --peer-key-file=/root/cert/etcd-key.pem \ --trusted-ca-file=/root/cert/ca.pem \ --peer-trusted-ca-file=/root/cert/ca.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
etcd-3
]# cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=simple EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/usr/bin/etcd \ --name=${ETCD_NAME} \ --data-dir=${ETCD_DATA_DIR} \ --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \ --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --initial-cluster=${ETCD_INITIAL_CLUSTER} \ --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster-state=new \ --cert-file=/root/cert/etcd.pem \ --key-file=/root/cert/etcd-key.pem \ --peer-cert-file=/root/cert/etcd.pem \ --peer-key-file=/root/cert/etcd-key.pem \ --trusted-ca-file=/root/cert/ca.pem \ --peer-trusted-ca-file=/root/cert/ca.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
参数解释

创建配置文件
etcd-1
]# cat /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.50.220:2380" ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.220:2379" ETCD_NAME="etcd-cluster1" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.220:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.220:2379" ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380" ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
etcd-2
]# cat /etc/etcd/etcd.conf #[Member] ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.50.221:2380" ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.221:2379" ETCD_NAME="etcd-cluster2" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.221:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.221:2379" ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380" ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
etcd-3
]# cat /etc/etcd/etcd.conf #[Member] ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.50.222:2380" ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.222:2379" ETCD_NAME="etcd-cluster3" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.222:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.222:2379" ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380" ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
启动
444 systemctl daemon-reload 445 systemctl start etcd
报错
Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: newRaft 4a48efb1bd381d9c [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: 4a48efb1bd381d9c became follower at term 1 Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: 4a48efb1bd381d9c switched to configuration voters=(5352791703792655772) Jan 16 11:09:38 controller-2 etcd[77741]: simple token is not cryptographically signed Jan 16 11:09:38 controller-2 etcd[77741]: starting server... [version: 3.4.9, cluster version: to_be_decided] Jan 16 11:09:38 controller-2 etcd[77741]: 4a48efb1bd381d9c as single-node; fast-forwarding 7 ticks (election ticks 8) Jan 16 11:09:38 controller-2 etcd[77741]: failed to purge snap db file open /data/etcd-data/member/snap: no such file or directory Jan 16 11:09:38 controller-2 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE Jan 16 11:09:38 controller-2 systemd[1]: Failed to start Etcd Server. -- Subject: Unit etcd.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit etcd.service has failed. -- -- The result is failed. Jan 16 11:09:38 controller-2 systemd[1]: Unit etcd.service entered failed state. Jan 16 11:09:38 controller-2 systemd[1]: etcd.service failed. Jan 16 11:09:38 controller-2 polkitd[786]: Unregistered Authentication Agent for unix-process:77736:49916625 (system bus name :1.315, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_U [root@controller-2 CA]# rm -rf /data/etcd-data/member/ [root@controller-2 CA]# systemctl start etcd [root@controller-2 CA]# systemctl status etcd ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2023-01-16 11:09:46 CST; 12s ago Docs: https://github.com/coreos Main PID: 77757 (etcd) CGroup: /system.slice/etcd.service └─77757 /data/etcd/bin/etcd --data-dir=/data/etcd-data --wal-dir=/data/etcd-data --name=etcd2 --cert-file=/data/pki/etcd/etcd.pem --key-file=/data/pki/etcd/etcd-key.pem --trusted-ca-file=/data... Jan 16 11:09:46 controller-2 etcd[77757]: raft2023/01/16 11:09:46 INFO: raft.node: 4a48efb1bd381d9c elected leader 4a48efb1bd381d9c at term 2 Jan 16 11:09:46 controller-2 etcd[77757]: published {Name:etcd2 ClientURLs:[https://192.168.50.225:2379]} to cluster 7a42d6c524509625 Jan 16 11:09:46 controller-2 etcd[77757]: ready to serve client requests Jan 16 11:09:46 controller-2 etcd[77757]: setting up the initial cluster version to 3.4 Jan 16 11:09:46 controller-2 etcd[77757]: ready to serve client requests Jan 16 11:09:46 controller-2 systemd[1]: Started Etcd Server. Jan 16 11:09:46 controller-2 etcd[77757]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged! Jan 16 11:09:46 controller-2 etcd[77757]: serving client requests on 192.168.50.225:2379 Jan 16 11:09:46 controller-2 etcd[77757]: set the initial cluster version to 3.4 Jan 16 11:09:46 controller-2 etcd[77757]: enabled capabilities for version 3.4
集群健康监测
1 /usr/bin/etcdctl --ca-file=/root/cert/ca.pem --cert-file=/root/cert/etcd.pem --key-file=/root/cert/etcd-key.pem --endpoints="https://192.168.50.220:2379,https://192.168.50.221:2379,https://192.168.50.222:2379" cluster-health 2 member 1d0f59292d7002a6 is healthy: got healthy result from https://192.168.50.221:2379 3 member c3b43c4124d987ee is healthy: got healthy result from https://192.168.50.220:2379 4 member fd26658e8648d5e7 is healthy: got healthy result from https://192.168.50.222:2379 5 cluster is healthy
列出集群内所有节点
]# /usr/bin/etcdctl --ca-file=/root/cert/ca.pem --cert-file=/root/cert/etcd.pem --key-file=/root/cert/etcd-key.pem --endpoints="https://192.168.50.220:2379,https://192.168.50.221:2379,https://192.168.50.222:2379" member list 1d0f59292d7002a6: name=etcd-cluster2 peerURLs=https://192.168.50.221:2380 clientURLs=https://192.168.50.221:2379 isLeader=false c3b43c4124d987ee: name=etcd-cluster1 peerURLs=https://192.168.50.220:2380 clientURLs=https://192.168.50.220:2379 isLeader=false fd26658e8648d5e7: name=etcd-cluster3 peerURLs=https://192.168.50.222:2380 clientURLs=https://192.168.50.222:2379 isLeader=true
本文来自博客园,作者:不会跳舞的胖子,转载请注明原文链接:https://www.cnblogs.com/rtnb/p/17055414.html