k8s之etcd集群部署

第一步：系统基本设置。（所有机器都要做）

1 关闭防火墙，selinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久

2 关闭swap

swapoff -a  # 临时

3 同步时间

ntpdate time.windows.com

4 配置host文件

vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.43.230  pc1.master01.com
192.168.43.232  pc1.node01.com
192.168.43.233  pc1.node02.com

 

第二步：cfssl自签证书（下载地址：https://pkg.cfssl.org/），任意一台机器都可做

下载cfssl工具，我这里直接下载到/usr/local/bin下，并重新命名了

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo

chmod 755 /usr/local/bin/*

创建目录test，在任意目录下生成证书。

mkdir test  && cd test

手动编写以下三个配置文件

vim ca-config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

vim server-csr.json     注意：此文件要修改ip

{
    "CN": "etcd",
    "hosts": [
        "192.168.43.63",
        "192.168.43.65",
        "192.168.43.66"
        ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

vim ca-csr.json

{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

 

 

 

 

 

第三步：部署etcd集群。

1. 下载etcd包，请根据系统下载对应的包。   # 备注：请使用3.13及以下版本，新版本出现启动报错。（新版本的集群部署，待续）

 

wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz
tar -zxvf etcd-v3.3.13-linux-amd64.tar.gz
mv etcd-v3.3.13-linux-amd64 /opt/etcd

 

 

解压后只有两个二进制的命令是有用的，如下图所示，可以把这两个命令放到环境变量里

mkdir /opt/etcd/{cfg,ssl,bin}          # cfg作为etcd服务的主目录,ssl目录放证书，个人喜欢

mv /opt/etcd/{etcd,etcdctl}  /opt/etcd/bin

 

2.  修改etcd服务的配置文件

vim /opt/etcd/cfg/etcd.conf     # 记得修改下面的ip，名字(名字随意）

 

#[Member]
ETCD_NAME="master"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.151:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.151:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.151:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.151:2379"
ETCD_INITIAL_CLUSTER="master=https://192.168.0.151:2380,node01=https://192.168.0.152:2380,node02=https://192.168.0.150:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

 

3. 把刚才生成的证书拷贝到etcd的ssl目录下。

cp test/{ca,server,server-key}.pem  etcd/ssl

备注：三台机器都要操作此步骤

 

4. 配置启动服务，且开机自启动。

编写启动服务的配置文件

vim /usr/lib/systemd/system/etcd.service

# 配置文件直接复制即可，如果服务文件目录不在opt下，则修改目录既可。

#
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
        --name=${ETCD_NAME} \
        --data-dir=${ETCD_DATA_DIR} \
        --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
        --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
        --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
        --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
        --initial-cluster=${ETCD_INITIAL_CLUSTER} \
        --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
        --initial-cluster-state=new \
        --cert-file=/opt/etcd/ssl/server.pem \
        --key-file=/opt/etcd/ssl/server-key.pem \
        --peer-cert-file=/opt/etcd/ssl/server.pem \
        --peer-key-file=/opt/etcd/ssl/server-key.pem \
        --trusted-ca-file=/opt/etcd/ssl/ca.pem \
        --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

 

启动服务：

systemctl daemon-reload
systemctl start etcd

启动时报错：request cluster ID mismatch (got db075567aa1b6248 want 3ccd9b211810a1e3)

解决办法：删除节点上的所有member，然后重新启动：mv /var/lib/etcd/default.etcd/ /tmp/

 

 

 

 

5. 检测三个节点的健康情况。

vim /usr/local/sbin/check_health.sh

#!/bin/bash
/opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.43.63:2379,https://192.168.43.65:2379,https://192.168.43.66:2379" \
cluster-health

chmod 755 /usr/local/sbin/check_health.sh

到此，etcd集群部署完毕。

 

[完]