Kubeadm安装Kubernetes环境

 Kubeadm方式号称一键安装部署,很多人也试过并且顺利成功,可到了我这里因为折腾系统问题,倒腾出不少的坑出来。

  • kubeadm好处是自动配置了必要的服务,以及缺省配置了安全的认证,etcd,apiserver,controller-manager,Schedule,kube-proxy都变成pod而非操作系统进程可以不断检测其状态并且进行迁移(能否迁移不确定)
  • kubeadm上有很多组件配置直接拿来可用。
  • 缺点是缺乏集群高可用模式,以及目前的定位是beta版。
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.

  • 准备工作

关掉selinux

vi /etc/selinux/config
disabled

关掉firewalld,iptables

systemctl disable firewalld
systemctl stop firewalld
systemctl disable iptables
systemctl stop iptables

 

先设置主机名

hostnamectl set-hostname k8s-1

修改/etc/hosts文件

cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.105    k8s-1
192.168.0.106    k8s-2
192.168.0.107    k8s-3

修改网络配置成静态ip,然后

service network restart
  • 安装docker,kubectl,kubelet,kubeadm

安装docker

yum install docker

验证docker version

[root@k8s-master1 ~]# service docker start
Redirecting to /bin/systemctl start  docker.service
[root@k8s-master1 ~]# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-61.git85d7426.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      85d7426/1.12.6
 Built:           Tue Oct 24 15:40:21 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-61.git85d7426.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      85d7426/1.12.6
 Built:           Tue Oct 24 15:40:21 2017
 OS/Arch:         linux/amd64

 

开机启动

[root@k8s-master1 ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@k8s-master1 ~]# systemctl start docker

 

编辑生成kubernetes的yum源

[root@k8s-1 network-scripts]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0

安装kubelet,kubectl,kubenetes-cni,kubeadm,缺省安装的是1.7.5版本

yum install kubectl kubelet kubernetes-cni kubeadm 
sysctl net.bridge.bridge-nf-call-iptables=1

如果需要安装其他版本,可以用yum remove移除

修改kubelet启动配置文件,主要是将--cgroup-driver改为cgroupfs(确保和/usr/lib/systemd/system/docker.service的用户一致就可以了,不需要修改!)

[root@k8s-1 bin]# cat  /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS

 

启动docker和kubelet

systemctl enable docker
systemctl enable kubelet
systemctl start docker
systemctl start kubelet

 

  • 下载镜像

在运行kubeadm之前,需要在本地先下载一系列images,这些images名称和版本,可以运行kubeadm init,然后中断运行得到

具体会生成在/etc/kubernetes/manifest目录下,通过grep命令可以列出,比如

cat etcd.yaml | grep gcr*
    image: gcr.io/google_containers/etcd-amd64:3.0.17

那具体需要下载哪些images和相应的版本呢? 可以参照kubernetes kubeadm手册,具体地址

https://kubernetes.io/docs/admin/kubeadm/

 这里就有比较清楚的版本和对应关系。

如何获取镜像

国内因为gcr.io被墙,所以要么通过代理翻墙获取,要么寻找其他办法。我的办法是访问

https://hub.docker.com/,然后搜索kube-apiserver-amd64,会列出各位大神已经build好的images

 

 选择相应的版本,进行pull

docker pull cloudnil/etcd-amd64:3.0.17
docker pull cloudnil/pause-amd64:3.0
docker pull cloudnil/kube-proxy-amd64:v1.7.2
docker pull cloudnil/kube-scheduler-amd64:v1.7.2
docker pull cloudnil/kube-controller-manager-amd64:v1.7.2
docker pull cloudnil/kube-apiserver-amd64:v1.7.2
docker pull cloudnil/kubernetes-dashboard-amd64:v1.6.1
docker pull cloudnil/k8s-dns-sidecar-amd64:1.14.4
docker pull cloudnil/k8s-dns-kube-dns-amd64:1.14.4
docker pull cloudnil/k8s-dns-dnsmasq-nanny-amd64:1.14.4

docker tag cloudnil/etcd-amd64:3.0.17    gcr.io/google_containers/etcd-amd64:3.0.17
docker tag cloudnil/pause-amd64:3.0     gcr.io/google_containers/pause-amd64:3.0
docker tag cloudnil/kube-proxy-amd64:v1.7.2   gcr.io/google_containers/kube-proxy-amd64:v1.7.2
docker tag cloudnil/kube-scheduler-amd64:v1.7.2  gcr.io/google_containers/kube-scheduler-amd64:v1.7.2
docker tag cloudnil/kube-controller-manager-amd64:v1.7.2 gcr.io/google_containers/kube-controller-manager-amd64:v1.7.2
docker tag cloudnil/kube-apiserver-amd64:v1.7.2 gcr.io/google_containers/kube-apiserver-amd64:v1.7.2
docker tag cloudnil/kubernetes-dashboard-amd64:v1.6.1 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1
docker tag cloudnil/k8s-dns-sidecar-amd64:1.14.4 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4
docker tag cloudnil/k8s-dns-kube-dns-amd64:1.14.4 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4
docker tag cloudnil/k8s-dns-dnsmasq-nanny-amd64:1.14.4 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4

最后

[root@k8s-1 ~]# docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/kube-apiserver-amd64            v1.7.2              25c5958099a8        3 months ago        186.1 MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.7.2              83d607ba9358        3 months ago        138 MB
gcr.io/google_containers/kube-scheduler-amd64            v1.7.2              6282cca6de74        3 months ago        77.18 MB
gcr.io/google_containers/kube-proxy-amd64                v1.7.2              69f8faa3d08d        3 months ago        114.7 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.4              2d6a3bea02c4        3 months ago        49.38 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.4              13117b1d461f        3 months ago        41.41 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.4              c413c7235eb4        3 months ago        41.81 MB
gcr.io/google_containers/etcd-amd64                      3.0.17              393e48d05c4e        4 months ago        168.9 MB
gcr.io/google_containers/kubernetes-dashboard-amd64      v1.6.1              c14ffb751676        4 months ago        134.4 MB
gcr.io/google_containers/pause-amd64                     3.0                 66c684b679d2        4 months ago        746.9 kB

 

  • 主节点初始化

 

镜像准备完成,准备开始init

kubeadm init --kubernetes-version=v1.7.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=192.168.0.105,192.168.0.106,192.168.0.107,127.0.0.1,k8s-1,k8s-2,k8s-3,192.168.0.1 --skip-preflight-checks

 

[root@k8s-1 network-scripts]# kubeadm init --kubernetes-version=v1.7.2 --pod-network-cidr=10.244.0.0/12 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=192.168.0.105,192.168.0.106,192.168.0.107,127.0.0.1,k8s-1,k8s-2,k8s-3,192.168.0.1 --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Skipping pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Using the existing CA certificate and key.
[certificates] Using the existing API Server certificate and key.
[certificates] Using the existing API Server kubelet client certificate and key.
[certificates] Using the existing service account token signing key.
[certificates] Using the existing front-proxy CA certificate and key.
[certificates] Using the existing front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[apiclient] Created API client, waiting for the control plane to become ready

坑来了。。。卡在这一句上,通过journalctl看日志

journalctl -xeu kubelet > a

 

Oct 30 10:01:30 k8s-1 systemd[1]: Starting kubelet: The Kubernetes Node Agent...
-- Subject: Unit kubelet.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun starting up.
Oct 30 10:01:30 k8s-1 kubelet[4646]: I1030 10:01:30.326586    4646 feature_gate.go:144] feature gates: map[]
Oct 30 10:01:30 k8s-1 kubelet[4646]: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory
Oct 30 10:01:30 k8s-1 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Oct 30 10:01:30 k8s-1 systemd[1]: Unit kubelet.service entered failed state.
Oct 30 10:01:30 k8s-1 systemd[1]: kubelet.service failed.
Oct 30 10:01:40 k8s-1 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Oct 30 10:01:40 k8s-1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
Oct 30 10:01:40 k8s-1 systemd[1]: Starting kubelet: The Kubernetes Node Agent...
-- Subject: Unit kubelet.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun starting up.
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.709684    4676 feature_gate.go:144] feature gates: map[]
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.712602    4676 client.go:72] Connecting to docker on unix:///var/run/docker.sock
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.712647    4676 client.go:92] Start docker client with request timeout=2m0s
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.714086    4676 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.725461    4676 manager.go:143] cAdvisor running in container: "/"
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.752809    4676 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.762789    4676 fs.go:117] Filesystem partitions: map[/dev/mapper/cl-root:{mountpoint:/ major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:xfs blockSize:0}]
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.763579    4676 manager.go:198] Machine: {NumCores:1 CpuFrequency:2496238 MemoryCapacity:1041182720 MachineID:a146a47b0c6b4c28a794c88309119e62 SystemUUID:B9DF3269-4A23-458F-8717-21EC1D216DD4 BootID:62e18038-ea14-438f-9688-e6a4abf265a1 Filesystems:[{Device:/dev/mapper/cl-root DeviceMajor:253 DeviceMinor:0 Capacity:39700664320 Type:vfs Inodes:19394560 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:1063256064 Type:vfs Inodes:524288 HasInodes:true}] DiskMap:map[253:1:{Name:dm-1 Major:253 Minor:1 Size:2147483648 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:42949672960 Scheduler:cfq} 253:0:{Name:dm-0 Major:253 Minor:0 Size:39720058880 Scheduler:none}] NetworkDevices:[{Name:enp0s3 MacAddress:08:00:27:e2:ae:0a Speed:1000 Mtu:1500} {Name:virbr0 MacAddress:52:54:00:ed:58:71 Speed:0 Mtu:1500} {Name:virbr0-nic MacAddress:52:54:00:ed:58:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:1073274880 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:3145728 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.765607    4676 manager.go:204] Version: {KernelVersion:3.10.0-514.21.1.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.12.6 DockerAPIVersion:1.24 CadvisorVersion: CadvisorRevision:}
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.766218    4676 server.go:536] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.767731    4676 container_manager_linux.go:218] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.767779    4676 container_manager_linux.go:246] container manager verified user specified cgroup-root exists: /
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.767789    4676 container_manager_linux.go:251] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[]}
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.767924    4676 kubelet.go:263] Adding manifest file: /etc/kubernetes/manifests
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.767935    4676 kubelet.go:273] Watching apiserver
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.782325    4676 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: Failed to list *v1.Node: Get https://192.168.0.105:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-1&resourceVersion=0: dial tcp 192.168.0.105:6443: getsockopt: connection refused
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.782380    4676 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: Failed to list *v1.Service: Get https://192.168.0.105:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.105:6443: getsockopt: connection refused
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.782413    4676 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.0.105:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-1&resourceVersion=0: dial tcp 192.168.0.105:6443: getsockopt: connection refused
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.783607    4676 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.783625    4676 kubelet.go:508] Hairpin mode set to "hairpin-veth"
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.784179    4676 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
orks found in /etc/cni/net.d
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.784915    4676 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.793823    4676 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.793839    4676 docker_service.go:208] Docker cri networking managed by cni
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.798395    4676 docker_service.go:225] Setting cgroupDriver to cgroupfs
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.804276    4676 remote_runtime.go:42] Connecting to runtime service unix:///var/run/dockershim.sock
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.806221    4676 kuberuntime_manager.go:166] Container runtime docker initialized, version: 1.12.6, apiVersion: 1.24.0
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.807620    4676 server.go:943] Started kubelet v1.7.5
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.808001    4676 kubelet.go:1229] Image garbage collection failed once. Stats initialization may not have completed yet: unable to find data for container /
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.808008    4676 kubelet_node_status.go:247] Setting node annotation to enable volume controller attach/detach
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.808464    4676 server.go:132] Starting to listen on 0.0.0.0:10250
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.809166    4676 server.go:310] Adding debug handlers to kubelet server.
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.811544    4676 event.go:209] Unable to write event: 'Post https://192.168.0.105:6443/api/v1/namespaces/default/events: dial tcp 192.168.0.105:6443: getsockopt: connection refused' (may retry after sleeping)
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.818965    4676 kubelet.go:1729] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.818965    4676 kubelet.go:1737] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.826012    4676 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.826058    4676 status_manager.go:140] Starting to sync pod status with apiserver
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.826130    4676 kubelet.go:1809] Starting kubelet main sync loop.
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.826196    4676 kubelet.go:1820] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.826424    4676 container_manager_linux.go:747] CPUAccounting not enabled for pid: 980
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.826429    4676 container_manager_linux.go:750] MemoryAccounting not enabled for pid: 980
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.826465    4676 container_manager_linux.go:747] CPUAccounting not enabled for pid: 4676
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.826429    4676 container_manager_linux.go:750] MemoryAccounting not enabled for pid: 980
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.826465    4676 container_manager_linux.go:747] CPUAccounting not enabled for pid: 4676
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.826468    4676 container_manager_linux.go:750] MemoryAccounting not enabled for pid: 4676
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.826495    4676 container_manager_linux.go:543] [ContainerManager]: Fail to get rootfs information unable to find data for container /
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.826504    4676 volume_manager.go:245] Starting Kubelet Volume Manager
Oct 30 10:01:40 k8s-1 kubelet[4676]: W1030 10:01:40.829827    4676 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.829892    4676 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Oct 30 10:01:40 k8s-1 kubelet[4676]: E1030 10:01:40.844934    4676 factory.go:336] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
Oct 30 10:01:40 k8s-1 kubelet[4676]: I1030 10:01:40.845787    4676 factory.go:351] Registering Docker factory

 

看起来是cni初始化的问题,网上帖子一大堆,但解决方案都不work。

=============================================================================

反复折腾搞不定,觉得可能是自己的OS有问题,重新安装了个CentOS7.4,步骤一样,结果秒过,真的是崩溃啊,前面那个问题折腾了一天!

同时抄了个脚本,自动化一下镜像下载

images=(etcd-amd64:3.0.17 pause-amd64:3.0 kube-proxy-amd64:v1.7.2 kube-scheduler-amd64:v1.7.2 kube-controller-manager-amd64:v1.7.2 kube-apiserver-amd64:v1.7.2 kubernetes-dashboard-amd64:v1.6.1 k8s-dns-sidecar-amd64:1.14.4 k8s-dns-kube-dns-amd64:1.14.4 k8s-dns-dnsmasq-nanny-amd64:1.14.4)
for imageName in ${images[@]} ; do
  docker pull cloudnil/$imageName
  docker tag cloudnil/$imageName gcr.io/google_containers/$imageName
  docker rmi cloudnil/$imageName
done

 

 

[root@k8s-1 ~]# kubeadm init --kubernetes-version=v1.7.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=192.168.0.105,192.168.0.106,192.168.0.107,127.0.0.1,k8s-1,k8s-2,k8s-3,192.168.0.1 
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [k8s-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-1 k8s-2 k8s-3] and IPs [192.168.0.105 192.168.0.106 192.168.0.107 127.0.0.1 192.168.0.1 10.96.0.1 192.168.0.105]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 55.001211 seconds
[token] Using token: 22d578.d921a7cf51352441
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 22d578.d921a7cf51352441 192.168.0.105:6443

 

然后

export KUBECONFIG=/etc/kubernetes/admin.conf 

[root@k8s-1 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-1                      1/1       Running   0          5m
kube-system   kube-apiserver-k8s-1            1/1       Running   0          4m
kube-system   kube-controller-manager-k8s-1   1/1       Running   0          4m
kube-system   kube-dns-2425271678-j8mnw       0/3       Pending   0          5m
kube-system   kube-proxy-6k4sb                1/1       Running   0          5m
kube-system   kube-scheduler-k8s-1            1/1       Running   0          4m

 

  • 安装flanneld网络

启动kube-dns的服务无法启动,因为网络尚未配置。

配置flannel网络

在https://github.com/winse/docker-hadoop/tree/master/kube-deploy/kubeadm 中下载kube-flannel.yml和kube-flannel-rbac.yml

 然后运行:

[root@k8s-1 ~]# kubectl apply -f kube-flannel.yml
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created
[root@k8s-1 ~]# kubectl apply -f kube-flannel-rbac.yml 
clusterrole "flannel" created
clusterrolebinding "flannel" created

等待一段时间后pod启动,配置完成

[root@k8s-1 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-1                      1/1       Running   1          3h
kube-system   kube-apiserver-k8s-1            1/1       Running   1          3h
kube-system   kube-controller-manager-k8s-1   1/1       Running   1          3h
kube-system   kube-dns-2425271678-j8mnw       3/3       Running   0          3h
kube-system   kube-flannel-ds-j491k           2/2       Running   0          1h
kube-system   kube-proxy-6k4sb                1/1       Running   1          3h
kube-system   kube-scheduler-k8s-1            1/1       Running   1          3h

 

节点

安装images

images=(pause-amd64:3.0 kube-proxy-amd64:v1.7.2)
for imageName in ${images[@]} ; do
  docker pull cloudnil/$imageName
  docker tag cloudnil/$imageName gcr.io/google_containers/$imageName
  docker rmi cloudnil/$imageName
done
root@k8s-3 ~]# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/kube-proxy-amd64   v1.7.2              69f8faa3d08d        3 months ago        114.7 MB
gcr.io/google_containers/pause-amd64        3.0                 66c684b679d2        4 months ago        746.9 kB

 

加入集群

[root@k8s-2 ~]# kubeadm join --token 22d578.d921a7cf51352441 192.168.0.105:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.0.105:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.105:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://192.168.0.105:6443"
[discovery] Successfully established connection with API Server "192.168.0.105:6443"
[bootstrap] Detected server version: v1.7.2
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

 

验证

[root@k8s-1 ~]# kubectl get nodes
NAME      STATUS    AGE       VERSION
k8s-1     Ready     4h        v1.7.5
k8s-2     Ready     1m        v1.7.5

 

加入节点3后验证

[root@k8s-1 ~]# kubectl get nodes
NAME      STATUS    AGE       VERSION
k8s-1     Ready     4h        v1.7.5
k8s-2     Ready     5m        v1.7.5
k8s-3     Ready     50s       v1.7.5
[root@k8s-1 ~]# kubectl get pods -n kube-system -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP              NODE
etcd-k8s-1                      1/1       Running   1          4h        192.168.0.105   k8s-1
kube-apiserver-k8s-1            1/1       Running   1          4h        192.168.0.105   k8s-1
kube-controller-manager-k8s-1   1/1       Running   1          4h        192.168.0.105   k8s-1
kube-dns-2425271678-j8mnw       3/3       Running   0          4h        10.244.0.2      k8s-1
kube-flannel-ds-d8vvr           2/2       Running   0          1m        192.168.0.107   k8s-3
kube-flannel-ds-fgvr1           2/2       Running   0          5m        192.168.0.106   k8s-2
kube-flannel-ds-j491k           2/2       Running   0          1h        192.168.0.105   k8s-1
kube-proxy-6k4sb                1/1       Running   1          4h        192.168.0.105   k8s-1
kube-proxy-p6v69                1/1       Running   0          5m        192.168.0.106   k8s-2
kube-proxy-tk2jq                1/1       Running   0          1m        192.168.0.107   k8s-3
kube-scheduler-k8s-1            1/1       Running   1          4h        192.168.0.105   k8s-1

 

  • 建立一个dashborad

在三台机器上运行

images=(kubernetes-dashboard-amd64:v1.6.0)
for imageName in ${images[@]} ; do
  docker pull k8scn/$imageName
  docker tag k8scn/$imageName gcr.io/google_containers/$imageName
  docker rmi k8scn/$imageName
done

 

然后再https://github.com/winse/docker-hadoop/tree/master/kube-deploy/kubeadm下载一个kubernetes-dashboard.yaml文件

root@k8s-1 ~]# kubectl create -f kubernetes-dashboard.yaml 
serviceaccount "kubernetes-dashboard" created
clusterrolebinding "kubernetes-dashboard" created
deployment "kubernetes-dashboard" created
service "kubernetes-dashboard" created
[root@k8s-1 ~]# kubectl get pods -n kube-system -o wide
NAME                                    READY     STATUS    RESTARTS   AGE       IP              NODE
etcd-k8s-1                              1/1       Running   1          4h        192.168.0.105   k8s-1
kube-apiserver-k8s-1                    1/1       Running   1          4h        192.168.0.105   k8s-1
kube-controller-manager-k8s-1           1/1       Running   1          4h        192.168.0.105   k8s-1
kube-dns-2425271678-j8mnw               3/3       Running   0          4h        10.244.0.2      k8s-1
kube-flannel-ds-d8vvr                   2/2       Running   0          13m       192.168.0.107   k8s-3
kube-flannel-ds-fgvr1                   2/2       Running   0          18m       192.168.0.106   k8s-2
kube-flannel-ds-j491k                   2/2       Running   0          2h        192.168.0.105   k8s-1
kube-proxy-6k4sb                        1/1       Running   1          4h        192.168.0.105   k8s-1
kube-proxy-p6v69                        1/1       Running   0          18m       192.168.0.106   k8s-2
kube-proxy-tk2jq                        1/1       Running   0          13m       192.168.0.107   k8s-3
kube-scheduler-k8s-1                    1/1       Running   1          4h        192.168.0.105   k8s-1
kubernetes-dashboard-3044843954-42k3c   1/1       Running   0          4s        10.244.2.2      k8s-3

 

firefox上运行http://10.244.2.2:9090/,秒出这一大堆的Pods.

 

 谢谢帮助我指引我爬坑的大神们:

 

http://www.cnblogs.com/liangDream/p/7358847.html

http://www.winseliu.com/blog/2017/08/13/kubeadm-install-k8s-on-centos7-with-resources/

 

posted @ 2017-10-29 11:11  ericnie  阅读(50970)  评论(10编辑  收藏