etcd集群安装

etcd 是一个分布式一致性k-v存储系统，可用于服务注册发现与共享配置，具有以下优点：
1.简单：相比于晦涩难懂的paxos算法，etcd基于相对简单且易实现的raft算法实现一致性，并通过gRPC提供接口调用
2.安全：支持TLS通信，并可以针对不同的用户进行对key的读写控制
3.高性能：10,000/秒的写性能

 

开放端口:

firewall-cmd --zone=public --add-port=2379/tcp --permanent
firewall-cmd --zone=public --add-port=2380/tcp --permanent
firewall-cmd --zone=public --add-port=4001/tcp --permanent
firewall-cmd --reload

　　　　

设置hosts

172.16.150.25 etcd1
172.16.150.26 etcd2
172.16.150.27 etcd3

　　

创建用户

mkdir -p /opt/platform/etcd
useradd etcd -d /opt/platform/etcd -c "Etcd user" -r -s /sbin/nologin

　　

安装cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

　

创建CA证书配置，生成CA证书和私钥, 只需要在一台机器上操作即可

先用 cfssl 命令生成包含默认配置的 config.json和 csr.json文件

mkdir /opt/ssl
cd /opt/ssl
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json

　　

然后分别修改这两个文件为如下内容

vim /opt/ssl/config.json

{
    "signing": {
        "default": {
            "expiry": "87600h"
            },
        "profiles": {
            "kubernetes": {
                 "usages": [
                     "signing",
                     "key encipherment",
                     "server auth",
                     "client auth"
                  ],
                 "expiry": "87600h"
                }
            }
       }
}

config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示client可以用该 CA 对server提供的证书进行验证；
client auth：表示server可以用该CA对client提供的证书进行验证；

 

vim /opt/ssl/csr.json

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Wuhan",
      "L": "Hubei",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

CN：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (UserName)；浏览器使用该字段验证网站是否合法；
O：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组(Group)

 

生成CA 证书和私钥

cd /opt/ssl
cfssl gencert -initca csr.json | cfssljson -bare ca

　　

CA有关证书列表如下

[root@k8s-console ssl]# tree
.
├── ca.csr
├── ca-key.pem
├── ca.pem
├── config.json
└── csr.json

　　

创建etcd证书配置，生成 etcd 证书和私钥

在/opt/ssl 下添加文件 etcd-csr.json，内容如下

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.16.150.25",
    "172.16.150.26",
    "172.16.150.27"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
},
  "names": [
  {
    "C": "CN",
    "ST": "Shanghai",
    "L": "Shanghai",
    "O": "etcd",
    "OU": "Etcd Security"
    }
  ]
}

　　

生成etcd证书和密钥

cd /opt/ssl
cfssl gencert -ca=/opt/ssl/ca.pem \
-ca-key=/opt/ssl/ca-key.pem \
-config=/opt/ssl/config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd

　

给证书读权限

chmod 644 /opt/ssl/*

　　

etcd 有关证书证书列表如下

etcd.csr etcd-csr.json etcd-key.pem etcd.pem

　　

至此CA证书就弄完了, 将这台机器侠/opt/ssl下所有东西,拷贝到其他两台机器下面的/opt/ssl下即可

 

安装etcd

在三台上都安装etcd

yum install etcd -y

　

添加etcd配置

wget https://github.com/coreos/etcd/releases/download/v3.3.7/etcd-v3.3.7-linux-amd64.tar.gz
tar -xvf etcd-v3.3.7-linux-amd64.tar.gz
cd etcd-v3.3.7-linux-amd64
cp -a etcd* /opt/platform/etcd/

　

添加etcd为系统服务

vim /usr/lib/systemd/system/etcd.service　

etcd1

[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=etcd
WorkingDirectory=/opt/platform/etcd
EnvironmentFile=-/opt/platform/etcd/etcd.conf
ExecStart=/opt/platform/etcd/etcd \
  --name etcd1 \
  --cert-file=/opt/ssl/etcd.pem \
  --key-file=/opt/ssl/etcd-key.pem \
  --trusted-ca-file=/opt/ssl/ca.pem \
  --peer-cert-file=/opt/ssl/etcd.pem \
  --peer-key-file=/opt/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/opt/ssl/ca.pem \
  --initial-advertise-peer-urls https://172.16.150.25:2380 \
  --listen-peer-urls https://172.16.150.25:2380 \
  --listen-client-urls https://172.16.150.25:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://172.16.150.25:2379 \
  --initial-cluster-token consul_etcd \
  --initial-cluster etcd1=https://172.16.150.25:2380,etcd2=https://172.16.150.26:2380,etcd3=https://172.16.150.27:2380 \
  --initial-cluster-state new \
  --data-dir=/opt/platform/etcd/data
Restart=on-failure
RestartSec=5

　　

etcd2

[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=etcd
WorkingDirectory=/opt/platform/etcd
EnvironmentFile=-/opt/platform/etcd/etcd.conf
ExecStart=/opt/platform/etcd/etcd \
  --name etcd2 \
  --cert-file=/opt/ssl/etcd.pem \
  --key-file=/opt/ssl/etcd-key.pem \
  --trusted-ca-file=/opt/ssl/ca.pem \
  --peer-cert-file=/opt/ssl/etcd.pem \
  --peer-key-file=/opt/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/opt/ssl/ca.pem \
  --initial-advertise-peer-urls https://172.16.150.26:2380 \
  --listen-peer-urls https://172.16.150.26:2380 \
  --listen-client-urls https://172.16.150.26:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://172.16.150.26:2379 \
  --initial-cluster-token consul_etcd \
  --initial-cluster etcd1=https://172.16.150.25:2380,etcd2=https://172.16.150.26:2380,etcd3=https://172.16.150.27:2380 \
  --initial-cluster-state new \
  --data-dir=/opt/platform/etcd/data
Restart=on-failure
RestartSec=5

　　

etcd3

 

[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=etcd
WorkingDirectory=/opt/platform/etcd
EnvironmentFile=-/opt/platform/etcd/etcd.conf
ExecStart=/opt/platform/etcd/etcd \
  --name etcd3 \
  --cert-file=/opt/ssl/etcd.pem \
  --key-file=/opt/ssl/etcd-key.pem \
  --trusted-ca-file=/opt/ssl/ca.pem \
  --peer-cert-file=/opt/ssl/etcd.pem \
  --peer-key-file=/opt/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/opt/ssl/ca.pem \
  --initial-advertise-peer-urls https://172.16.150.27:2380 \
  --listen-peer-urls https://172.16.150.27:2380 \
  --listen-client-urls https://172.16.150.27:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://172.16.150.27:2379 \
  --initial-cluster-token consul_etcd \
  --initial-cluster etcd1=https://172.16.150.25:2380,etcd2=https://172.16.150.26:2380,etcd3=https://172.16.150.27:2380 \
  --initial-cluster-state new \
  --data-dir=/opt/platform/etcd/data
Restart=on-failure
RestartSec=5

　　

启动etcd:

mkdir -p /opt/platform/etcd/data && chown etcd:etcd -R /opt/platform/etcd
systemctl enable etcd.service && systemclt daemon-reload && systemctl start etcd.service

　

添加alias:

vim /etc/profile.d/alias_bash.sh
alias etcdctl='etcdctl --endpoints=https://172.16.150.25:2379,https://172.16.150.26:2379,https://172.16.150.27:2379 --cert-file=/opt/ssl/etcd.pem --ca-file=/opt/ssl/ca.pem --key-file=/opt/ssl/etcd-key.pem'
source /etc/profile.d/alias_bash.sh

 

验证etcd集群状态

etcdctl cluster-health

　　　

查看etcd 集群成员

etcdctl member list

 

测试验证:

etcdctl set test/testkey0 0 
etcdctl get test/testkey0