用kubeadm在vagrant-centos7安装kubernetes

最近尝试用kubeadm安装kubernetes,碰到了几个坑,解决后顺利的安装上了,再此记录一下,给需要的同学查看.基本每一步都列出了参考文档,有搞不懂的也可以直接看参考文档. 另外kubernetes最近更新较快,注意教程的版本

机器: windows

kubernetes操作系统:centos7(使用vagrant)

kubernetes版本:1.7.3

特殊需求:需要开启vt-x. 需要FQ下载镜像

目标

  1. 搭建kubernetes的master(api-server+etcd)

准备

  1. 关闭hyperv. 开启vt-x
  2. 下载vagrant,virtualbox并安装. (可以手动下载安装也可以用chocolatey安装)
  3. 下载centos7的box
  4. (可选1)配置好FQ代理,VPN
  5. (可选2)从时速云或者国内的其他云拉取gcr的镜像,并一个个的打上gcr的tag. 列表:
gcr.io/google_containers/kube-controller-manager-amd64   v1.7.3              138MB
gcr.io/google_containers/kube-apiserver-amd64            v1.7.3              186MB
gcr.io/google_containers/kube-scheduler-amd64            v1.7.3              77.2MB
gcr.io/google_containers/kube-proxy-amd64                v1.7.3              115MB
gcr.io/google_containers/etcd-amd64                      3.0.17              169MB
gcr.io/google_containers/pause-amd64                     3.0                 747kB

配置代理(可选)

因为要下载google的镜像,所以必须要有代理或者从国内下载镜像后打上tag. 这里提供一种ss+proxifier配置方法

  1. 配置好SS. 本地代理为:http://127.0.0.1:1080(默认的)

  2. proxifier->配置文件(菜单)->代理服务器->添加ss->确定

  3. proxifier->配置文件(菜单)->代理规则->添加virtualbox->动作设为ss->确定

安装

参考文档主要参考kubernetes官方文档的Using kubeadm to Create a Cluster

  1. 打开powershell 创建目录
λ  C:\Users\mskmk
λ  cd d:\
D:\
λ  mkdir k8s
λ  cd k8s
D:\k8s
λ  vagrant init
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
  1. 把准备阶段下好的box复制到k8s目录 重命名为centos-7.0-x86_64.box(这里也可以先add box再直接指定名称)
  2. 删掉Vagrantfile内的文本 改为以下内容(熟悉的话可以自己改参数):
Vagrant.configure(2) do |config|
  (1..3).each do |i|
    config.vm.define "k8s#{i}" do |s|
      s.ssh.forward_agent = true
      s.vm.box = "centos-7.0-x86_64.box"
      s.vm.hostname = "k8s#{i}"
      s.vm.network "public_network"
      s.vm.provider "virtualbox" do |v|
        v.name = "k8s#{i}"
        v.memory = 2048
        v.gui = false
      end
    end
  end
end

PS:这里启动了3个机器,但是这篇教程只用到了第一个.后面2个是后面用来作为node的

  1. 启动集群 并ssh连上k8s1
vagrant up
vagrant ssh k8s1
  1. 检查语言(可选)
    发现语言不对 改成LANG="en_US.UTF-8"
$cat /etc/locale.conf
LANG="de_DE.UTF-8"
$sudo vim /etc/locale.conf
$sudo reboot
  1. 安装docker 参考文档
sudo yum remove docker \
                  docker-common \
                  docker-selinux \
                  docker-engine
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
sudo yum makecache fast
sudo yum install -y docker-ce
sudo systemctl start docker
sudo systemctl status docker //确认docker是否启动成功
  1. 安装kubectl 参考文档
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
  1. 安装kubelet and kubeadm 为了减少后面的麻烦.这里开始启用root. FQ注意！！ 这里要确认能访问packages.cloud.google.com.否则kubelet kubeadm无法安装成功(可以尝试其他安装kubelet kubeadm的方法.请自行搜索)。 参考文档
sudo passwd //给root设置密码
su //切换到root 后面的语句整个复制粘贴到shell
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm
systemctl enable kubelet && systemctl start kubelet
  1. 初始化master. 如果没有错误,这一步就过了,有错误,参考写出的解决办法
[root@k8s1 vagrant]# kubeadm init --pod-network-cidr=10.244.0.0/16 //加上--pod-network-cidr是为了使用flannel(根据参考文档)
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 1.12
[preflight] WARNING: docker service is not enabled, please run 'systemctl enable docker.service'
[preflight] WARNING: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Some fatal errors occurred:
        /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks`

有错误 修复一下继续

[root@k8s1 vagrant]# echo "1">/proc/sys/net/bridge/bridge-nf-call-iptables
[root@k8s1 vagrant]# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
kubeadm init --pod-network-cidr=10.244.0.0/16
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 1.12
[preflight] WARNING: docker service is not enabled, please run 'systemctl enable docker.service'
[preflight] WARNING: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [k8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.2.15]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready

卡在waiting for the control plane to become ready了.开一个新的shell2(powershell) 检查一下日志

cd D:\k8s
vagrant ssh k8s1
sudo journalctl -xeu kubelet
Aug 12 10:32:25 k8s1 kubelet[5054]: W0812 10:32:25.205653    5054 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
... // 略
Aug 12 10:32:35 k8s1 kubelet[5118]: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
Aug 12 10:32:35 k8s1 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Aug 12 10:32:35 k8s1 systemd[1]: Unit kubelet.service entered failed state.
Aug 12 10:32:35 k8s1 systemd[1]: kubelet.service failed.

又有错误 根据倒数第四行以及各种搜索到的信息,是kubelet的cgroup driver和docker不一致.那我们就把他们改一致. shell2:

[vagrant@k8s1 ~]$ sudo cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" //这里systemd改成docker的cgroupfs
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS
[vagrant@k8s1 ~]$sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[vagrant@k8s1 ~]$ sudo cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS

重启kubelet

systemctl daemon-reload

切回shell1 CTRL+C停止等待

[root@k8s1 vagrant]# kubeadm reset
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/lib/etcd]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[root@k8s1 vagrant]# kubeadm init --pod-network-cidr=10.244.0.0/16
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 1.12
[preflight] WARNING: docker service is not enabled, please run 'systemctl enable docker.service'
[preflight] WARNING: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [k8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.2.15]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[apiclient] Created API client, waiting for the control plane to become ready

依旧阻塞 切回shell2 检查有没有kubelet.service failed的信息

sudo journalctl -xeu kubelet

没有kubelet.service failed失败的信息.
可以认为这时候正在从google pull 镜像.确认到gcr的连接(需要FQ).这时候等待就好了

  1. 成功的提示. 成功后会从waiting for the control plane to become ready继续往下走,出现以下提示,这表示master已经安装成功！
[token] Using token: 3a9431.40d61ebb7d46d770
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 3a9431.40d61ebb7d46d770 10.0.2.15:6443

可以用docker看看运行的镜像

[root@k8s1 ~]# docker ps
CONTAINER ID        IMAGE                                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
102f0eac3c7a        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 11 hours ago        Up 11 hours                             k8s_POD_kube-flannel-ds-4pmjq_kube-system_b9600c62-7ebc-11e7-aca1-080027de0e0e_0
e5ed24159528        gcr.io/google_containers/kube-proxy-amd64                "/usr/local/bin/ku..."   11 hours ago        Up 11 hours                             k8s_kube-proxy_kube-proxy-3nql2_kube-system_6aadfc16-7ebc-11e7-aca1-080027de0e0e_0
e05e44d290c1        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 11 hours ago        Up 11 hours                             k8s_POD_kube-proxy-3nql2_kube-system_6aadfc16-7ebc-11e7-aca1-080027de0e0e_0
4d7c2de686ef        gcr.io/google_containers/kube-scheduler-amd64            "kube-scheduler --..."   11 hours ago        Up 11 hours                             k8s_kube-scheduler_kube-scheduler-k8s1_kube-system_3386b6e53bb1022a6eaf6f513b40c9e3_0
6b813c586547        gcr.io/google_containers/kube-apiserver-amd64            "kube-apiserver --..."   11 hours ago        Up 11 hours                             k8s_kube-apiserver_kube-apiserver-k8s1_kube-system_75fb805115b08772c080549caee03a55_1
5cdc7b32a966        gcr.io/google_containers/etcd-amd64                      "etcd --listen-cli..."   11 hours ago        Up 11 hours                             k8s_etcd_etcd-k8s1_kube-system_9ef6d25e21bb4befeabe4d0e4f72d1ca_0
969dbd445570        gcr.io/google_containers/kube-controller-manager-amd64   "kube-controller-m..."   11 hours ago        Up 11 hours                             k8s_kube-controller-manager_kube-controller-manager-k8s1_kube-system_97d7d7a1a3a9104117271303948bcc5b_0
2a93e0d3a2f9        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 11 hours ago        Up 11 hours                             k8s_POD_kube-apiserver-k8s1_kube-system_75fb805115b08772c080549caee03a55_0
55dd62d8de39        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 11 hours ago        Up 11 hours                             k8s_POD_kube-controller-manager-k8s1_kube-system_97d7d7a1a3a9104117271303948bcc5b_0
edbd05038c42        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 11 hours ago        Up 11 hours                             k8s_POD_etcd-k8s1_kube-system_9ef6d25e21bb4befeabe4d0e4f72d1ca_0
5c222faf3e57        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 11 hours ago        Up 11 hours                             k8s_POD_kube-scheduler-k8s1_kube-system_3386b6e53bb1022a6eaf6f513b40c9e3_0

总结

kubernetes发展太快,各种教程甚至官方文档都不能保证正确性和及时性,只有实践才能发现各种坑。这也算是提供一次自己的实践经历，给大家总结一些遇到的坑,说不定哪天就用上了呢

常出现错误

  1. yum install -y kubelet kubeadm可能找不到需要安装的程序.解决:FQ
  2. kubeadm init提示/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1. 解决:echo "1">/proc/sys/net/bridge/bridge-nf-call-iptables
  3. kubeadm init卡在waiting for the control plane to become ready. 处理:
sudo journalctl -xeu kubelet

检查输出日志中最后一行是否:kubelet.service failed:

  1. 是的话往上看日志,一般在倒数第四行,比如docker cgroup dirver和kubelet不匹配之类的。不匹配按前面提到的方法处理,其他错误请自己去谷歌百度
  2. 最后一行不是kubelet.service failed. 等待下载镜像即可.(注意！需要确定能FQ,不然等多久都没用)

posted on 2017-08-12 17:06  alamisw  阅读(1125)  评论(0)    收藏  举报

导航