二进制部署高可用的etcd集群

此文档已过时，仅作参考使用

使用cfssl生成证书

下载cfssl

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl_1.6.2_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssljson_1.6.2_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl-certinfo_1.6.2_linux_amd64

  356  mv cfssl_1.6.2_linux_amd64 /usr/local/bin/cfssl
  357  mv cfssl-certinfo_1.6.2_linux_amd64 /usr/sbin/certinfo
  358  mv cfssljson_1.6.2_linux_amd64 /usr/sbin/cfssljson

可以用以下命令生成默认格式

$ cfssl print-defaults config > ca-config.json
$ cfssl print-defaults csr > ca-csr.json

创建ca

制作证书模板配置文件
证书颁发机构，颁发证书时，需要参考不同模板的要求进行证书创建

[root@controller CA]# cat  ca-config.json
{
    "signing": {
        "default": {
            "expiry": "87600h"   # 默认过期时间
        },
        "profiles": {  # 设置了多个场景，创建时可以-profile=www/client指定
            "www": {
                "expiry": "87600h",  # 过期时间
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"  # server auth指定此证书只能用在服务器端
                ]
            },
            "client": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"  # 只能用在客户端
                ]
            },
            "all": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth",   # 既可以客户端，也可以服务器
                    "server auth"
                ]
            }
        }
    }
}

证书申请文件

申请证书配置文件
证书申请者申请证书时，要填写的基本信息
我们去办理身份证时向机关提供的基本信息

cat ca-csr.json

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
           "OU": "System"
        }
    ]
}

生成CA证书与私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

ca：生成文件名称前缀

制作kubernetes证书

使用自签名证书生成子证书方式

gencert: 生成新的key(密钥)和签名证书
-initca：初始化一个新ca
-ca：指明ca的证书
-ca-key：指明ca的私钥文件
-config：指明请求证书的json文件
-profile：与-config中的profile对应，是指根据config中的profile段来生成证书的相关信息

etcd证书生成

创建Etcd证书签名请求文件

cat etcd-csr.json

{
    "CN":"etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.50.220",
        "192.168.50.221",
        "192.168.50.222",
        "*.kubernetes.master",
        "localhost",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo":"rsa",
        "size": 2048
    },
    "names": [
        {
            "C":"CN",
            "ST":"BeiJing",
            "L":"BeiJing",
            "O":"k8s",
            "OU":"System"
        }
    ]
}

知识点：

这个证书目前专属于 apiserver，加了一个 *.kubernetes.master域名以便内部私有 DNS 解析使用(可删除)；kubernetes 这几个能不能删掉，答案是不可以的；因为当集群创建好后，default namespace 下会创建一个叫 kubenretes 的svc，有一些组件会直接连接这个 svc 来跟 api 通信的，证书如果不包含可能会出现无法连接的情况；其他几个 kubernetes 开头的域名作用相同
hosts包含的是授权范围，不在此范围的的节点或者服务使用此证书就会报证书不匹配错误。10.254.0.1是指kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP。

签发证书

  384  cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem  -config=ca-config.json  -profile=all  etcd-csr.json | cfssljson -bare etcd

[root@controller etcd]# ll
total 16
-rw-r--r-- 1 root root 1281 Jan 16 09:50 etcd.csr
-rw-r--r-- 1 root root  595 Jan 16 09:32 etcd-csr.json
-rw------- 1 root root 1675 Jan 16 09:50 etcd-key.pem
-rw-r--r-- 1 root root 1643 Jan 16 09:50 etcd.pem

使用openssl来查看证书信息

openssl x509 -in server.pem -noout -text

使用cfssl-certinfo来查看证书信息

可以通过命令行来动态设置SAN模块，如设置域名，IP

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem  -config=ca-conf.json -profile=client -hostname=www.golang-node1.com,www.golang-node2.com,127.0.0.1,10.211.55.10 node-csr.json | cfssljson -bare node -

部署高可用的etcd集群

参考文档：https://www.cnblogs.com/hahaha111122222/p/16016386.html

下载并分发

#!/bin/bash
cd /data/k8s/work
source /data/k8s/bin/env.sh
wget https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz
tar -zxf etcd-v3.3.18-linux-amd64.tar.gz

for node_ip in ${ETCD_IPS[@]}
do
  echo ">>> ${node_ip}"
  scp etcd-v3.3.18-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
  ssh root@${node_ip} "chmod +x /data/k8s/bin/*"
done

创建启动脚本

etcd-1

]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/root/cert/etcd.pem \
--key-file=/root/cert/etcd-key.pem \
--peer-cert-file=/root/cert/etcd.pem \
--peer-key-file=/root/cert/etcd-key.pem \
--trusted-ca-file=/root/cert/ca.pem \
--peer-trusted-ca-file=/root/cert/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

etcd-2

]#  cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/root/cert/etcd.pem \
--key-file=/root/cert/etcd-key.pem \
--peer-cert-file=/root/cert/etcd.pem \
--peer-key-file=/root/cert/etcd-key.pem \
--trusted-ca-file=/root/cert/ca.pem \
--peer-trusted-ca-file=/root/cert/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

etcd-3

]#  cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/root/cert/etcd.pem \
--key-file=/root/cert/etcd-key.pem \
--peer-cert-file=/root/cert/etcd.pem \
--peer-key-file=/root/cert/etcd-key.pem \
--trusted-ca-file=/root/cert/ca.pem \
--peer-trusted-ca-file=/root/cert/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

参数解释

创建配置文件

etcd-1

]# cat /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.50.220:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.220:2379"
ETCD_NAME="etcd-cluster1"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.220:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.220:2379"
ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380"
ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

etcd-2

]# cat /etc/etcd/etcd.conf
#[Member]
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.50.221:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.221:2379"
ETCD_NAME="etcd-cluster2"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.221:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.221:2379"
ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380"
ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

etcd-3

]# cat /etc/etcd/etcd.conf
#[Member]
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.50.222:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.222:2379"
ETCD_NAME="etcd-cluster3"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.222:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.222:2379"
ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380"
ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

启动

  444  systemctl  daemon-reload
  445  systemctl  start etcd

报错

Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: newRaft 4a48efb1bd381d9c [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: 4a48efb1bd381d9c became follower at term 1
Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: 4a48efb1bd381d9c switched to configuration voters=(5352791703792655772)
Jan 16 11:09:38 controller-2 etcd[77741]: simple token is not cryptographically signed
Jan 16 11:09:38 controller-2 etcd[77741]: starting server... [version: 3.4.9, cluster version: to_be_decided]
Jan 16 11:09:38 controller-2 etcd[77741]: 4a48efb1bd381d9c as single-node; fast-forwarding 7 ticks (election ticks 8)
Jan 16 11:09:38 controller-2 etcd[77741]: failed to purge snap db file open /data/etcd-data/member/snap: no such file or directory
Jan 16 11:09:38 controller-2 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Jan 16 11:09:38 controller-2 systemd[1]: Failed to start Etcd Server.
-- Subject: Unit etcd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcd.service has failed.
--
-- The result is failed.
Jan 16 11:09:38 controller-2 systemd[1]: Unit etcd.service entered failed state.
Jan 16 11:09:38 controller-2 systemd[1]: etcd.service failed.
Jan 16 11:09:38 controller-2 polkitd[786]: Unregistered Authentication Agent for unix-process:77736:49916625 (system bus name :1.315, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_U

[root@controller-2 CA]# rm -rf  /data/etcd-data/member/

[root@controller-2 CA]# systemctl  start etcd

[root@controller-2 CA]# systemctl  status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-01-16 11:09:46 CST; 12s ago
     Docs: https://github.com/coreos
 Main PID: 77757 (etcd)
   CGroup: /system.slice/etcd.service
           └─77757 /data/etcd/bin/etcd --data-dir=/data/etcd-data --wal-dir=/data/etcd-data --name=etcd2 --cert-file=/data/pki/etcd/etcd.pem --key-file=/data/pki/etcd/etcd-key.pem --trusted-ca-file=/data...

Jan 16 11:09:46 controller-2 etcd[77757]: raft2023/01/16 11:09:46 INFO: raft.node: 4a48efb1bd381d9c elected leader 4a48efb1bd381d9c at term 2
Jan 16 11:09:46 controller-2 etcd[77757]: published {Name:etcd2 ClientURLs:[https://192.168.50.225:2379]} to cluster 7a42d6c524509625
Jan 16 11:09:46 controller-2 etcd[77757]: ready to serve client requests
Jan 16 11:09:46 controller-2 etcd[77757]: setting up the initial cluster version to 3.4
Jan 16 11:09:46 controller-2 etcd[77757]: ready to serve client requests
Jan 16 11:09:46 controller-2 systemd[1]: Started Etcd Server.
Jan 16 11:09:46 controller-2 etcd[77757]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Jan 16 11:09:46 controller-2 etcd[77757]: serving client requests on 192.168.50.225:2379
Jan 16 11:09:46 controller-2 etcd[77757]: set the initial cluster version to 3.4
Jan 16 11:09:46 controller-2 etcd[77757]: enabled capabilities for version 3.4

集群健康监测

1 /usr/bin/etcdctl --ca-file=/root/cert/ca.pem --cert-file=/root/cert/etcd.pem --key-file=/root/cert/etcd-key.pem --endpoints="https://192.168.50.220:2379,https://192.168.50.221:2379,https://192.168.50.222:2379" cluster-health
2 member 1d0f59292d7002a6 is healthy: got healthy result from https://192.168.50.221:2379
3 member c3b43c4124d987ee is healthy: got healthy result from https://192.168.50.220:2379
4 member fd26658e8648d5e7 is healthy: got healthy result from https://192.168.50.222:2379
5 cluster is healthy

列出集群内所有节点

]# /usr/bin/etcdctl --ca-file=/root/cert/ca.pem --cert-file=/root/cert/etcd.pem --key-file=/root/cert/etcd-key.pem --endpoints="https://192.168.50.220:2379,https://192.168.50.221:2379,https://192.168.50.222:2379"  member list
1d0f59292d7002a6: name=etcd-cluster2 peerURLs=https://192.168.50.221:2380 clientURLs=https://192.168.50.221:2379 isLeader=false
c3b43c4124d987ee: name=etcd-cluster1 peerURLs=https://192.168.50.220:2380 clientURLs=https://192.168.50.220:2379 isLeader=false
fd26658e8648d5e7: name=etcd-cluster3 peerURLs=https://192.168.50.222:2380 clientURLs=https://192.168.50.222:2379 isLeader=true

posted @ 2023-01-16 15:00 不会跳舞的胖子阅读(19) 评论(0) 收藏举报

刷新页面返回顶部

不会跳舞的胖子

二进制部署高可用的etcd集群

公告