kubernetes集群部署
一、系统要求Centos7.2
二、需求三台服务器
192.168.1.151
192.168.1.152
192.168.1.153
三、etcd集群部署
1、三台服务器都需修改/etc/hosts主机名
192.168.1.151 k8s-master etcd1
192.168.1.152 k8s-node1 etcd2
192.168.1.153 k8s-node2 etcd3
2、三台服务器都需修改服务器主机名
hostnamectl --static set-hostname k8s-master
hostnamectl --static set-hostname k8s-node-1
hostnamectl --static set-hostname k8s-node-2
3、三台服务器都需确保时间同步
ntpdate -u cn.pool.ntp.org
4、三台服务器都需安装etcd服务
yum -y install etcd
5、192.168.1.151 修改etcd.conf配置文件
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" --数据存放位置
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380" -- 监听其他 Etcd 实例的地址
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001" -- 监听客户端地址
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.151:2380" -- 通知其他 Etcd 实例地址
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.151:2379,http://192.168.1.151:4001" -- 通知客户端地址
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.151:2380,etcd2=http://192.168.1.152:2380,etcd3=http://192.168.1.153:2380" -- 初始化集群内节点地址
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" -- 初始化集群 token
ETCD_INITIAL_CLUSTER_STATE="new" -- 初始化集群状态，new 表示新建
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
6、192.168.1.152 192.168.1.153修改etcd.conf配置文件
配置与192.168.1.151相同
只需将相应的ip地址及hostname修改为本机地址即可
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.151:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.151:2379,http://192.168.1.151:4001"
7、启动服务并查看集群状态
systmectl start etcd.service
etcdctl member list && etcdctl cluster-health
四、Kubernetes集群部署
1、192.168.1.151 Kubernetes-master安装部署
- yum -y install kubernetes-master flannel -- 安装服务
- vim /etc/kubernetes/apiserver -- 修改apiserver文件
###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#
# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
# The port on the local server to listen on.
KUBE_API_PORT="--port=8080"
# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"
# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://192.168.1.151:2379,http://192.168.1.152:2379,http://192.168.1.153:2379"
# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
# default admission control policies
#KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
# Add your own!
KUBE_API_ARGS=""
- 启动服务
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
2、192.168.1.152 192.168.1.153 Kubernetes-node安装部署
- yum -y install Kubernetes-node docker flannel
- vim /etc/kubernetes/config -- 修改config配置文件
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
# kube-apiserver.service
# kube-controller-manager.service
# kube-scheduler.service
# kubelet.service
# kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://192.168.1.151:8080"
- vim /etc/kubernetes/kubelet -- 修改kubelet配置文件
###
# kubernetes kubelet (minion) config
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=127.0.0.1"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.1.152"
# location of the api-server
KUBELET_API_SERVER="--api-servers=http://192.168.1.151:8080"
# pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
# Add your own!
KUBELET_ARGS=""
- 启动服务
systemctl start docker.service
systemctl start kubelet.service
systemctl start kube-proxy.service
3、查看集群部署信息
kubectl get nodes
五、网络配置
1、三台服务器都需修改
- vim /etc/sysconfig/flanneld
#Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://192.168.1.151:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/atomic.io/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""
2、启动flannel服务 #注意重启flannel服务需要重启docker及kubelet.service、kube-proxy.service服务
systemctl start flanneld.service
k8s-master
#!/bin/bash
ser_name=(flanneld etcd kube-apiserver kube-controller-manager kube-scheduler)
for i in ${ser_name[@]}
do
systemctl restart $i.service
done
k8s-node重启脚本
#!/bin/bash
ser_name=(flanneld etcd docker kubelet kube-proxy)
for i in ${ser_name[@]}
do
systemctl restart $i.service
done
六、注意事项
1、如果K8s-Master集群启动时flannel无法实现Pod内部跨主机通信，请执行以下命令并重启k8s-node所有服务
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F
iptables -L -n
2、如果执行kubctl create -f xxxx.yaml时Pod状态一直处于ContainerCreating，一下操作需要在k8s-node上执行
- 查看kubectl describe pod httpd-deployment错误信息,其中最主要的问题是：details: (open/etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file ordirectory)
- 查看/etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt（该链接就是上图中的说明） 是一个软链接，但是链接过去后并没有真实的/etc/rhsm，所以需要使用yum安装
yum -y install *rhsm* 安装完成后，执行一下docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest
如果依然报错，可参考下面的方案
- wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm
- rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pem
- 这两个命令会生成/etc/rhsm/ca/redhat-uep.pem文件
- 执行docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest
Trying to pull repositoryregistry.access.redhat.com/rhel7/pod-infrastructure ...
latest: Pulling fromregistry.access.redhat.com/rhel7/pod-infrastructure
26e5ed6899db: Pull complete
66dbe984a319: Pull complete
9138e7863e08: Pull complete
Digest:sha256:92d43c37297da3ab187fc2b9e9ebfb243c1110d446c783ae1b989088495db931
Status: Downloaded newer image forregistry.access.redhat.com/rhel7/pod-infrastructure:latest
- 如果pull成功后docker images查看，如果有registry.access.redhat.com/rhel7/pod-infrastructure则再次执行执行
kubctl create -f xxxx.yaml
kubectl get pods -- 查看正常
- 简单yaml文件实现启动httpd测试
- vim deployment.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: httpd-deployment
spec:
replicas: 2
template:
metadata:
labels:
app: httpd-deployment
spec:
containers:
- name: httpd
image: httpd:0.1
ports:
- containerPort: 80
hostPort: 80
浙公网安备 33010602011771号