k8s v1.10.5 二进制集群安装

一. k8s 软件版本选择

名称 参数
系统版本 centos 7 64
docker版本 docker-ce-17.03.2
k8s系列版本 v1.10.0
pod-ip地址段 --cluster-cidr 10.229.0.0/16
service-ip地址段 service-cluster-ip-range 10.228.0.0/16
apiserver的vip 10.255.72.199

二. 系统环境

ip hostname 角色
10.255.72.189 qaz-bt-kvm-72-189 apiserver,controller,scheduler,kubelet,kube-proxy,etcd
10.255.72.190 qaz-bt-kvm-72-190 apiserver,controller,scheduler,kubelet,kube-proxy,etcd
10.255.72.191 qaz-bt-kvm-72-191 apiserver,controller,scheduler,kubelet,kube-proxy,etcd
10.255.72.192 qaz-bt-kvm-72-192 kubelet,kube-proxy

三. 系统需求

  • 1. host解析:
10.255.72.189   kube-master01
10.255.72.190   kube-master02
10.255.72.191   kube-master03
10.255.72.192   kube-minon02
10.255.72.189   etcd01
10.255.72.190   etcd02
10.255.72.191   etcd03

  • 2. 关闭系统selinux:

    /etc/selinux/config
    SELINUX=disabled
    setenforce 0

  • 3. 关闭系统swap：

    swapoff -a
    注释掉/etc/fstab下的所有的swap配置

  • 4. 添加内核参数：

    echo "
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
" > /etc/sysctl.d/k8s.conf
sysctl -p

  • 5. 系统更新到最新

    yum update -y && yum upgrade -y

  • 6. 安装docker

    #!/bin/bash
yum remove docker docker-common docker-selinux docker-engine -y
yum erase docker.x86_64 container-selinux.noarch
yum install -y yum-utils device-mapper-persistent-data lvm2
yum install https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm  -y
yum install https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/Packages/docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm  -y

  • 7.修改docker的service文件

    ExecStart=/usr/bin/dockerd \
    --exec-opt native.cgroupdriver=systemd

四.安装etcd+tls集群环境

角色 ip地址 配置主机名 客户端端口 集群端口
etcd01 10.255.72.189 etcd01 2379 2380
etcd02 10.255.72.190 etcd02 2379 2380
etcd03 10.255.72.191 etcd03 2379 2380

  • 1. 安装cfssl工具:

    go get -u github.com/cloudflare/cfssl/cmd/...
cp go/bin/cfssl* /usr/local/bin/
chmod +x /usr/local/bin/cfssl

  • 2. 创建etcd tls证书：

    cd /root/ ; mkdir ssl && cd ssl/

    ca-config.json

    {
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "example": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

    ca-csr.json

    {
  "CN": "example",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "example",
      "OU": "cloudnative"
    }
  ]
}

    创建ca证书

    cfssl gencert -initca ca-csr.json | cfssljson -bare ca

    etcd-csr.json

    {
    "CN": "example",
    "hosts": [
      "127.0.0.1",
      "10.255.72.189",
      "10.255.72.190",
      "10.255.72.191",
      "etcd01",
      "etcd02",
      "etcd03"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "example",
            "OU": "cloudnative"
        }
    ]
}

    创建etcd证书

    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example etcd-csr.json | cfssljson -bare etcd

    查看证书的信息

    cfssl-certinfo -cert etcd.pem

  • 3. 二进制文件下载

    curl -LO https://github.com/coreos/etcd/releases/download/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz
tar zxvf etcd-v3.2.9-linux-amd64.tar.gz
cd etcd-v3.2.9-linux-amd64 && mv etcd* /usr/bin/ && chmod +x /usr/bin/etcd*
#拷贝配置文件
mkdir -p /etc/etcd/ssl && cp /root/ssl/etcd*.pem /etc/etcd/ssl/
cp /root/ssl/ca.pem /etc/etcd/ssl/

  • 4. 系统service文件

    export nodeip=10.255.72.189
    export nodeip=10.255.72.190
    export nodeip=10.255.72.191

      echo "
  [Unit]
  Description=etcd server
  After=network.target
  After=network-online.target
  Wants=network-online.target

  [Service]
  Type=notify
  WorkingDirectory=/var/lib/etcd/
  EnvironmentFile=-/etc/etcd/etcd.conf
  ExecStart=/usr/bin/etcd \
    --name etcd01 \
    --cert-file=/etc/etcd/ssl/etcd.pem \
    --key-file=/etc/etcd/ssl/etcd-key.pem \
    --peer-cert-file=/etc/etcd/ssl/etcd.pem \
    --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
    --trusted-ca-file=/etc/etcd/ssl/ca.pem \
    --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
    --initial-advertise-peer-urls https://${nodeip}:2380 \
    --listen-peer-urls https://${nodeip}:2380 \
    --listen-client-urls https://${nodeip}:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://${nodeip}:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster etcd01=https://10.255.72.189:2380,etcd02=https://10.255.72.190:2380,etcd03=https://10.255.72.191:2380 \
    --initial-cluster-state new \
    --data-dir=/var/lib/etcd
  Restart=on-failure
  RestartSec=5
  LimitNOFILE=65536

  [Install]
  WantedBy=multi-user.target
  " >/usr/lib/systemd/system/etcd.service

  • 5. 启动etcd 集群

    systemctl daemon-reload
    systemctl start etcd.service
    systemctl status etcd.service

  • 6. 查看集群健康状态

    etcdctl \
--ca-file=/etc/etcd/ssl/ca.pem \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://10.255.72.189:2379,https://10.255.72.190:2379,https://10.255.72.191:2379 cluster-health

    输出结果为:

    member 11561ab2384b6c6d is healthy: got healthy result from https://10.255.72.191:2379
member 5b2cf22137bb72af is healthy: got healthy result from https://10.255.72.190:2379
member c88f6852e21dbb31 is healthy: got healthy result from https://10.255.72.189:2379
cluster is healthy

五. 在10.255.72.{189,190,191}下载kubernetes二进制文件,并复制到/usr/local/bin/下,在10.255.72.192上只需要kubelet二进制文件

curl - LO https://dl.k8s.io/v1.10.0/kubernetes-server-linux-amd64.tar.gz
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-controller-manager kube-scheduler kubelet /usr/local/bin/

六.配置kube-apiserver集群

角色 ip地址
kube-apiserver01 10.255.72.189
kube-apiserver02 10.255.72.190
kube-apiserver03 10.255.72.191
  • 1.生成tls证书
cd /root/ssl/ && cat apiserver-csr.json

apiserver-csr.json

{
   "CN": "kubernetes",
   "hosts": [
     "127.0.0.1",
     "10.255.72.189",
     "10.255.72.190",
     "10.255.72.191",
     "10.255.72.192",
     "10.255.72.199",
     "10.228.0.1",
     "kubernetes",
     "kubernetes.default",
     "kubernetes.default.svc",
     "kubernetes.default.svc.cluster",
     "kubernetes.default.svc.cluster.local"
   ],
   "key": {
       "algo": "rsa",
       "size": 2048
   },
   "names": [
       {
           "C": "CN",
           "ST": "BeiJing",
           "L": "BeiJing",
           "O": "k8s",
           "OU": "cloudnative"
       }
   ]
}

生成证书的命令：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example apiserver-csr.json | cfssljson -bare apiserver

  • 2.配置kube-apiserver证书文件
mkdir -p /etc/kubernetes/ssl/
cd /root/ssl/
cp apiserver-key.pem apiserver.pem ca.pem ca-key.pem /etc/kubernetes/ssl
  • 3.生成token文件
cd /etc/kubernetes/
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
  • 4.配置kube-apiserver的service文件

export nodeip=10.255.72.189
export nodeip=10.255.72.190
export nodeip=10.255.72.191

echo "
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --advertise-address=${nodeip} \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/lib/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=${nodeip}  \
  --secure-port=6442 \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --enable-swagger-ui=true \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://10.255.72.189:2379,https://10.255.72.190:2379,https://10.255.72.191:2379 \
  --event-ttl=1h \
  --kubelet-https=true \
  --insecure-bind-address=${nodeip}  \
  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-cluster-ip-range=10.228.0.0/16 \
  --service-node-port-range=30000-37000 \
  --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
  --enable-bootstrap-token-auth \
  --token-auth-file=/etc/kubernetes/token.csv \
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
">/usr/bin/systemd/system/kube-apiserver.service

  • 5. 启动并查看apiserver的集群状态

    systemctl daemon-reload
systemctl start kube-apiserver.service
systemctl status kube-apiserver.service

  • 6. 创建.kube/config

    创建admin证书

    admin-csr.json

    {
    "CN": "kubernetes-admin",
    "hosts": [
            "10.255.72.189",
            "10.255.72.190",
            "10.255.72.191",
            "10.255.72.192"
    ],
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:masters",
        "OU": "cloudnative"
      }
    ]
}

    生成证书

    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example admin-csr.json | cfssljson -bare admin
cp /root/ssl/admin.pem /root/ssl/admin-key.pem /etc/kubernetes/ssl/

    生成配置文件

    cd /etc/kubernetes
export KUBE_APISERVER="https://10.255.72.199:6443"

# set-cluster
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=admin.conf

# set-credentials
kubectl config set-credentials kubernetes-admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem \
  --kubeconfig=admin.conf

# set-context
kubectl config set-context kubernetes-admin@kubernetes \
  --cluster=kubernetes \
  --user=kubernetes-admin \
  --kubeconfig=admin.conf

# set default context
kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=admin.conf

# mv
cp admin.conf ~/.kube/config

    kubectl get cs

  • 7. 配置haproxy+keepalive 搭建高可用

    yum install -y keepalived haproxy

    /etc/haproxy/haproxy.conf

    global
      daemon
      maxconn 40960
      stats socket /etc/haproxy/haproxy.stats level operator
      stats timeout 2m
      log 127.0.0.1 local0

defaults
        log global
        mode http
        retries 3
        option redispatch
        timeout connect 60000ms
        timeout client 60000ms
        timeout server 60000ms

listen  kube-api *:6443 #代理配置
        mode tcp
        maxconn 4000
        balance roundrobin
        server  qaz-bt-kvm-72-189 10.255.72.189:6442 check inter 1500 fall 1 rise 2
        server  qaz-bt-kvm-72-190 10.255.72.190:6442 check inter 1500 fall 1 rise 2
        server  qaz-bt-kvm-72-191 10.255.72.191:6442 check inter 1500 fall 1 rise 2

listen stats *:7744 #status 状态配置
       mode http
       option httpclose
       balance roundrobin
       stats uri /
       stats realm Haproxy\ Statistics
       stats auth admin:nop@ss.1
       stats admin if TRUE

    /etc/keepalived/keepalived.conf

    global_defs {
}
vrrp_script chk_haproxy {
        script "/etc/keepalived/check_haproxy.sh"
        interval 2
        weight 2
        }
vrrp_instance VIP_1 {
          state BACKUP #配合不抢占一起配置。
          interface eth0
          virtual_router_id 50 #统一标示为一组广播的消息
          priority 100 #权重,只影响vip的抢占优先级,每节点设置不一样,这里每节点加50权重。
          advert_int 1
          nopreempt #不开启抢占,节省ip替换的耗费
          authentication {
            auth_type PASS
            auth_pass 321654
          }
  track_interface {
          eth0
  }
  virtual_ipaddress {
          10.255.72.199
  }
  track_script {
          chk_haproxy
  }
}

    /etc/keepalived/check_haproxy.sh

    #!/bin/bash
A=`ps -C haproxy --no-header |wc -l`
if [ $A -eq 0 ];then
  service haproxy start
  sleep 3
  if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then
    service keepalived stop
  fi
fi

    启动服务：

    service haproxy start
service keepalived start
ip a
netstat -lnpt

  • 8. 配置kube-apiserver的log归集

    vim /etc/rsyslog.conf
#添加配置 在messges log的定义之前。其他应用也一样。更改appname即可
$template app-template,"/var/log/%app-name%/%app-name%_%$YEAR%-%$MONTH%-%$DAY%.log"
if ( $app-name == "kube-apiserver") then {
                action(type="omfile" DynaFile="app-template")
                stop
}

七.配置kube-controller-manager

ip 角色
10.255.72.189 controller01
10.255.72.190 controller02
10.255.72.191 controller03
  • 1. 配置tls证书

controller-manager-csr.json

{
  "CN": "system:kube-controller-manager",
  "hosts": [
          "10.255.72.189",
          "10.255.72.190",
          "10.255.72.191",
          "10.255.72.192"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:kube-controller-manager",
      "OU": "cloudnative"
    }
  ]
}

生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example controller-manager-csr.json | cfssljson -bare controller-manager
  • 2. 移动证书文件
cp controller-manager-key.pem controller-manager.pem /etc/kubernetes/ssl/
scp controller-manager-key.pem controller-manager.pem  root@10.255.72.190:/etc/kubernetes/ssl/
scp controller-manager-key.pem controller-manager.pem  root@10.255.72.191:/etc/kubernetes/ssl/
  • 3. 生成controller-manager的kubeconfig文件controller-manager.conf
cd /etc/kubernetes
export KUBE_APISERVER="https://10.255.72.199:6443"

# set-cluster
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=controller-manager.conf

# set-credentials
kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=/etc/kubernetes/ssl/controller-manager.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/controller-manager-key.pem \
  --kubeconfig=controller-manager.conf
f
# set-context
kubectl config set-context system:kube-controller-manager@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=controller-manager.conf

# set default context9o9o99o=-098`214
kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=controller-manager.conf
  • 4.配置controller-manager的service文件

每节点配置一样

echo "
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
            --master=https://10.255.72.199:6443 \
            --kubeconfig=/etc/kubernetes/controller-manager.conf \
            --cluster-name=kubernetes \
            --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
            --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
            --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
            --root-ca-file=/etc/kubernetes/ssl/ca.pem \
            --service-cluster-ip-range=10.228.0.0/16 \
            --cluster-cidr=10.229.0.0/16 \
            --leader-elect=true \
            --controllers=* \
            --use-service-account-credentials=true
LimitNOFILE=65536
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target" > /usr/lib/systemd/system/kube-controller-manager.service

这里有一个参数需要注意： --use-service-account-credentials=true 参考：The Kubernetes controller manager runs core control loops. When invoked with --use-service-account-credentials, each control loop is started using a separate service account.

  • 5.启动服务
systemctl daemon-reload
systemctl start kube-controller-manager

八.配置scheduler服务集群

ip 角色
10.255.72.189 scheduler01
10.255.72.190 scheduler02
10.255.72.191 scheduler03
  • 1. 配置tls证书

scheduler-csr.json

{
    "CN": "system:kube-scheduler",
    "hosts": [
            "10.255.72.189",
            "10.255.72.190",
            "10.255.72.191",
            "10.255.72.192"
    ],
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-scheduler",
        "OU": "cloudnative"
      }
    ]
}

生成证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example scheduler-csr.json | cfssljson -bare scheduler
  • 2.移动证书文件
cp scheduler-key.pem scheduler.pem /etc/kubernetes/ssl/
scp scheduler-key.pem scheduler.pem  root@10.255.72.190:/etc/kubernetes/ssl/
scp scheduler-key.pem scheduler.pem  root@10.255.72.191:/etc/kubernetes/ssl/
  • 3.生成kube-scheduler的kubeconfig文件scheduler.conf
cd /etc/kubernetes
export KUBE_APISERVER="https://10.255.72.199:6443"

# set-cluster
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=scheduler.conf

# set-credentials
kubectl config set-credentials system:kube-scheduler \
  --client-certificate=/etc/kubernetes/ssl/scheduler.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/scheduler-key.pem \
  --kubeconfig=scheduler.conf

# set-context
kubectl config set-context system:kube-scheduler@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=scheduler.conf

# set default context
kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=scheduler.conf
  • 4.创建scheduler的service文件

每节点配置一样

/usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --master=https://10.255.72.199:6443 \
  --kubeconfig=/etc/kubernetes/scheduler.conf \
  --leader-elect=true \
  --v=2
LimitNOFILE=65536
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
  • 5.启动服务集群
systemctl daemon-reload
systemctl start kube-scheduler.service

九.部署kubelet客户端节点

以10.255.72.189节点为例

  • 1.生成kubelet客户端证书

kubelet-csr-189.json

{
  "CN": "system:node:kube-master01",
  "hosts": [
        "kube-master01",
        "10.255.72.189"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
    "O": "system:nodes",
      "OU": "cloudnative"
    }
  ]
}

生成证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example kubelet-csr-189.json | cfssljson -bare kubelet-189
cp kubelet-189.pem  /etc/kubernetes/ssl/kubelet.pem
cp kubectl-189-key.pem /etc/kubernetes/ssl/kubelet-key.pem
  • 2.创建 kubelet bootstrapping kubeconfig 文件
cd /etc/kubernetes
export KUBE_APISERVER="https://10.255.72.199:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
  • 3. 创建kubelet申请证书的权限用户
cd /etc/kubernetes
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

  • --user=kubelet-bootstrap 是在 /etc/kubernetes/token.csv 文件中指定的用户名，同时也写入了 /etc/kubernetes/bootstrap.kubeconfig 文件

  • 4. 创建kubelet的service文件

    创建必要的目录，不然服务启动会失败。

mkdir -p /etc/cni/net.d
mkdir -p /opt/cni/bin/
mkdir -p  /var/lib/kubelet

  [Unit]
  Description=Kubernetes Kubelet
  Documentation=https://github.com/GoogleCloudPlatform/kubernetes
  After=docker.service
  Requires=docker.service

  [Service]
  WorkingDirectory=/var/lib/kubelet
  ExecStart=/usr/local/bin/kubelet \
    --runtime-cgroups=/systemd/system.slice \
    --kubelet-cgroups=/systemd/system.slice \
    --cgroup-driver=systemd \
    --address=10.255.72.189 \
    --hostname-override=kube-master01 \
    --pod-infra-container-image=hub-dev.example.com/k8s/pause-amd64:3.0 \ #公司自建的仓库 搭建参考 harbor[https://github.com/vmware/harbor]
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
    --pod-manifest-path=/etc/kubernetes/manifests \
    --cert-dir=/etc/kubernetes/ssl \
    --cluster-dns=10.229.0.10 \
    --cluster-domain=cluster.local. \
    --allow-privileged=true \
    --rotate-certificates=true \
    --fail-swap-on=false \
    --serialize-image-pulls=false \
    --network-plugin=cni \
    --cni-conf-dir=/etc/cni/net.d \
    --cni-bin-dir=/opt/cni/bin \
    --max-pods=110 \
    --v=2
  Restart=on-failure
  RestartSec=5

  [Install]
  WantedBy=multi-user.target
  • 4. 启动服务
systemctl daemon-reload
systemctl start kubelet.service
  • 5.证书的查看和批准
kubectl get csr
kubectl certificate approve $csr
kubectl get nodes #查看加入的节点

十.部署kube-proxy

  • 1.配置ssl证书

kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:kube-proxy",
      "OU": "cloudnative"
    }
  ]
}
  • 2.生成证书文件
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=example kube-proxy-csr.json | cfssljson -bare kube-proxy
  • 3.生成kubeconfig文件kube-proxy.conf：
cd /etc/kubernetes
export KUBE_APISERVER="https://10.255.72.199:6443"
# set-cluster
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
# set-credentials
kubectl config set-credentials system:kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --kubeconfig=kube-proxy.kubeconfig
# set-context
kubectl config set-context system:kube-proxy@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
# set default context
kubectl config use-context system:kube-proxy@kubernetes --kubeconfig=kube-proxy.kubeconfig
  • 4.配置kube-proxy的系统service 文件

注意： 创建/var/lib/kube-proxy 目录，如果不存在 kube-proxy会起不来

mkdir /var/lib/kube-proxy

/usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=kube-proxy
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
EnvironmentFile=-/etc/kubernetes/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
    --logtostderr=true \
    --v=0 \
    --bind-address=10.255.72.189 \
    --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
    --cluster-cidr=10.229.0.0/16

Restart=on-failure

[Install]
WantedBy=multi-user.target

systemctl enable kube-proxy.service
systemctl start kube-proxy.service

十一.部署calico 网络插件

这里采用daemonset的方式部署:

calico.yaml

# Calico Version v2.3.0
# http://docs.projectcalico.org/v2.3/releases#v2.3.0
# This manifest includes the following component versions:
#   calico/node:v1.3.0
#   calico/cni:v1.9.1
#   calico/kube-policy-controller:v0.6.0

# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # The location of your etcd cluster.  This uses the Service clusterIP
  # defined below.
  etcd_endpoints: "http://10.228.0.222:6666"

  # Configure the Calico backend to use.
  calico_backend: "bird"

  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
        "name": "k8s-pod-network",
        "cniVersion": "0.1.0",
        "type": "calico",
        "etcd_endpoints": "__ETCD_ENDPOINTS__",
        "log_level": "info",
        "ipam": {
            "type": "calico-ipam"
        },
        "policy": {
            "type": "k8s",
             "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
             "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
        },
        "kubernetes": {
            "kubeconfig": "/etc/cni/net.d/__KUBECONFIG_FILENAME__"
        }
    }

---

# This manifest installs the Calico etcd on the kubeadm master.  This uses a DaemonSet
# to force it to run on the master even when the master isn't schedulable, and uses
# nodeSelector to ensure it only runs on the master.
apiVersion: extensions/v1beta1
#kind: DaemonSet
kind: Deployment
metadata:
  name: calico-etcd
  namespace: kube-system
  labels:
    k8s-app: calico-etcd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: calico-etcd
      annotations:
        # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler
        # reserves resources for critical add-on pods so that they can be rescheduled after
        # a failure.  This annotation works in tandem with the toleration below.
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      # Only run this pod on the master.
      #tolerations:
      #- key: node-role.kubernetes.io/master
      #  effect: NoSchedule
      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.
      # This, along with the annotation above marks this pod as a critical add-on.
      #- key: CriticalAddonsOnly
      #  operator: Exists
      #nodeSelector:
      #  node-role.kubernetes.io/master: ""
      hostNetwork: true
      containers:
        - name: calico-etcd
          image: harbor.example.com/k8s/etcd:2.2.1
          env:
            - name: CALICO_ETCD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          command: ["/bin/sh","-c"]
          args: ["/usr/local/bin/etcd --name=calico --data-dir=/var/etcd/calico-data --advertise-client-urls=http://$CALICO_ETCD_IP:6666 --listen-client-urls=http://0.0.0.0:6666 --listen-peer-urls=http://0.0.0.0:6667"]
          volumeMounts:
            - name: var-etcd
              mountPath: /var/etcd
      volumes:
        - name: var-etcd
          hostPath:
            path: /var/etcd

---

# This manifest installs the Service which gets traffic to the Calico
# etcd.
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: calico-etcd
  name: calico-etcd
  namespace: kube-system
spec:
  # Select the calico-etcd pod running on the master.
  selector:
    k8s-app: calico-etcd
  # This ClusterIP needs to be known in advance, since we cannot rely
  # on DNS to get access to etcd.
  clusterIP: 10.228.0.222
  ports:
    - port: 6666

---

# This manifest installs the calico/node container, as well
# as the Calico CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  template:
    metadata:
      labels:
        k8s-app: calico-node
      annotations:
        # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler
        # reserves resources for critical add-on pods so that they can be rescheduled after
        # a failure.  This annotation works in tandem with the toleration below.
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      hostNetwork: true
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.
      # This, along with the annotation above marks this pod as a critical add-on.
      - key: CriticalAddonsOnly
        operator: Exists
      serviceAccountName: calico-cni-plugin
      containers:
        # Runs calico/node container on each Kubernetes node.  This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: harbor.example.com/k8s/node:v1.3.0
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Enable BGP.  Disable to enforce policy only.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Configure the IP Pool from which Pod IPs will be chosen.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.229.0.0/16"
            - name: CALICO_IPV4POOL_IPIP
              value: "always"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            # Auto-detect the BGP IP address.
            - name: IP
              value: ""
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 300m
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
        # This container installs the Calico CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: harbor.example.com/k8s/cni:v1.9.1
          command: ["/install-cni.sh"]
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
      volumes:
        # Used by calico/node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d

---

# This manifest deploys the Calico policy controller on Kubernetes.
# See https://github.com/projectcalico/k8s-policy
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: calico-policy-controller
  namespace: kube-system
  labels:
    k8s-app: calico-policy
spec:
  # The policy controller can only have a single active instance.
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-policy-controller
      namespace: kube-system
      labels:
        k8s-app: calico-policy-controller
      annotations:
        # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler
        # reserves resources for critical add-on pods so that they can be rescheduled after
        # a failure.  This annotation works in tandem with the toleration below.
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      # The policy controller must run in the host network namespace so that
      # it isn't governed by policy that would prevent it from working.
      hostNetwork: true
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.
      # This, along with the annotation above marks this pod as a critical add-on.
      - key: CriticalAddonsOnly
        operator: Exists
      serviceAccountName: calico-policy-controller
      containers:
        - name: calico-policy-controller
          image: harbor.example.com/k8s/kube-policy-controller:v0.6.0
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # The location of the Kubernetes API.  Use the default Kubernetes
            # service for API access.
            - name: K8S_API
            #  value: "https://kubernetes.default:443"
              value: "https://10.255.72.199:6443"
            # Since we're running in the host namespace and might not have KubeDNS
            # access, configure the container's /etc/hosts to resolve
            # kubernetes.default to the correct service clusterIP.
            - name: CONFIGURE_ETC_HOSTS
              value: "true"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-cni-plugin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-cni-plugin
subjects:
- kind: ServiceAccount
  name: calico-cni-plugin
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-cni-plugin
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
    verbs:
      - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-cni-plugin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-policy-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-policy-controller
subjects:
- kind: ServiceAccount
  name: calico-policy-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-policy-controller
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    - extensions
    resources:
      - pods
      - namespaces
      - networkpolicies
    verbs:
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-policy-controller
  namespace: kube-system

～～注意：需要更改的地方～～

  • **1. etcd_endpoints: "http://10.228.0.222:6666" **
    第一次在这个yaml文件里定义 是创建的calico需要的etcd集群的ip地址和端口

  • ** 2. 这里写apiserver的集群ip地址，也就是haproxy+keepalived 搭建的代理的vip和端口**

- name: K8S_API
   value: "https://10.255.72.199:6443"
  • 3.calico-node的service定义的区域里需要再次更改etcd的clusterip 同1
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: calico-etcd
  name: calico-etcd
  namespace: kube-system
spec:
  # Select the calico-etcd pod running on the master.
  selector:
    k8s-app: calico-etcd
clusterIP: 10.228.0.222
  ports:
    - port: 6666
  • 4. 需要更改docker镜像的下载地址，贴出来的yaml文件里的地址为局域网的内部私有仓库地址，需要自己搭建docker私仓，这里不再展开
    类似这种的都得改 也可以使用开源的国内的源
    harbor.example.com/k8s/kube-policy-controller:v0.6.0
posted @ 2021-03-16 17:15  lei&dong  阅读(78)  评论(0)    收藏  举报