K8S实用整理(08)-kubelet启动参数修改方法(配置Enabling Unsafe Sysctls)

暂基于kubespary自动部署的1.9.0-coreos版本,kubelet服务相关配置文件:

文件1:/etc/systemd/system/kubelet.service 

/etc/systemd/system/kubelet.service

文件内容为:

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Wants=docker.socket

[Service]
EnvironmentFile=-/etc/kubernetes/kubelet.env
ExecStartPre=-/bin/mkdir -p /var/lib/kubelet/volume-plugins
ExecStart=/usr/local/bin/kubelet \
                $KUBE_LOGTOSTDERR \
                $KUBE_LOG_LEVEL \
                $KUBELET_API_SERVER \
                $KUBELET_ADDRESS \
                $KUBELET_PORT \
                $KUBELET_HOSTNAME \
                $KUBE_ALLOW_PRIV \
                $KUBELET_ARGS \
                $DOCKER_SOCKET \
                $KUBELET_NETWORK_PLUGIN \
                $KUBELET_VOLUME_PLUGIN \
                $KUBELET_CLOUDPROVIDER
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

 文件2:/etc/kubernetes/kubelet.env

/etc/kubernetes/kubelet.env

文件内容:
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=2"
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.28.2.211 --node-ip=172.28.2.211"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node1"






KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests \
--cadvisor-port=0 \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--node-status-update-frequency=10s \
--docker-disable-shared-pid=True \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--tls-cert-file=/etc/kubernetes/ssl/node-node1.pem \
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem \
--anonymous-auth=false \
--cgroup-driver=cgroupfs \
--cgroups-per-qos=True \
--fail-swap-on=True \
--enforce-node-allocatable=""  --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true  --feature-gates=Initializers=False,PersistentLocalVolumes=False  "
KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volume-plugins"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBELET_CLOUDPROVIDER=""

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~                                                                                 

 

修改需求:

Enabling Unsafe Sysctls

With the warning above in mind, the cluster admin can allow certain unsafe sysctls for very special situations like e.g. high-performance or real-time application tuning. Unsafe sysctls are enabled on a node-by-node basis with a flag of the kubelet, e.g.:

$ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.ipv4.route.min_pmtu' ...
修改需求:
kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'
修改方法:修改环境变量文件/etc/kubernetes/kubelet.env,修改为
(添加了--experimental-allowed-unsafe-sysctls='kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'"
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=2"
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.28.2.211 --node-ip=172.28.2.211"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node1"






KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests \
--cadvisor-port=0 \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--node-status-update-frequency=10s \
--docker-disable-shared-pid=True \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--tls-cert-file=/etc/kubernetes/ssl/node-node1.pem \
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem \
--anonymous-auth=false \
--cgroup-driver=cgroupfs \
--cgroups-per-qos=True \
--fail-swap-on=True \
--enforce-node-allocatable=""  --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true  --feature-gates=Initializers=False,PersistentLocalVolumes=False  \
--experimental-allowed-unsafe-sysctls='kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'"
KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volume-plugins"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBELET_CLOUDPROVIDER=""

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~                                                                                

 重启kubelet服务,查看是否修改成功:

systemctl restart kubelet

systemctl status kubelet

ps aux | grep kubelet | grep kernel

root@node1:~# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since 四 2018-03-08 17:20:38 CST; 2min 51s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 14844 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 14851 (kubelet)
    Tasks: 17
   Memory: 50.4M
      CPU: 27.792s
   CGroup: /system.slice/kubelet.service
           └─14851 /usr/local/bin/kubelet --logtostderr=true --v=2 --address=172.28.2.211 --node-ip=172.28.2.211 -

3月 08 17:23:28 node1 kubelet[14851]: E0308 17:23:28.518287   14851 pod_workers.go:186] Error syncing pod 082ad73d
3月 08 17:23:28 node1 kubelet[14851]: W0308 17:23:28.518432   14851 container.go:393] Failed to create summary rea
3月 08 17:23:28 node1 kubelet[14851]: I0308 17:23:28.887368   14851 kubelet.go:1881] SyncLoop (PLEG): "centos1_def
3月 08 17:23:28 node1 kubelet[14851]: W0308 17:23:28.887505   14851 pod_container_deletor.go:77] Container "84c856
3月 08 17:23:29 node1 kubelet[14851]: I0308 17:23:29.188203   14851 kuberuntime_manager.go:403] No ready sandbox f
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624710   14851 remote_runtime.go:92] RunPodSandbox from runti
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624792   14851 kuberuntime_sandbox.go:54] CreatePodSandbox fo
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624814   14851 kuberuntime_manager.go:647] createPodSandbox f
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624923   14851 pod_workers.go:186] Error syncing pod 082ad73d
3月 08 17:23:29 node1 kubelet[14851]: W0308 17:23:29.625543   14851 container.go:393] Failed to create summary rea
root@node1:~# ps aux | grep kubelet | grep kernel
root     14851 12.5  0.7 696144 121368 ?       Ssl  17:20   0:24 /usr/local/bin/kubelet --logtostderr=true --v=2 --address=172.28.2.211 
--node-ip=172.28.2.211 --hostname-override=node1 --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests
--cadvisor-port=0 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --node-status-update-frequency=10s
--docker-disable-shared-pid=True --client-ca-file=/etc/kubernetes/ssl/ca.pem --tls-cert-file=/etc/kubernetes/ssl/node-node1.pem
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem --anonymous-auth=false --cgroup-driver=cgroupfs --cgroups-per-qos=True
--fail-swap-on=True --enforce-node-allocatable= --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf
--kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true
--feature-gates=Initializers=False,PersistentLocalVolumes=False
--experimental-allowed-unsafe-sysctls=kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu
--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin
--volume-plugin-dir=/var/lib/kubelet/volume-plugins

如上修改成功。

创建POD,参考页面:K8S实用整理(10)-Kubernetes配置POD内核参数sysctl  http://www.cnblogs.com/DaweiJ/articles/8528687.html


Kubelet Configurations We Should Care About

下面是我梳理的,我认为必须关注的flag。

 

flagvalue
--address 0.0.0.0
--allow-privileged false
--cadvisor-port int32 4194
--cgroup-driver string cgroupfs
--cluster-dns stringSlice 10.0.0.10 //todo
--cluster-domain string caas.vivo.com
--cni-bin-dir string /opt/cni/bin
--cni-conf-dir string /etc/cni/net.d
--docker-endpoint string unix:///var/run/docker.sock
--eviction-hard string memory.available<4Gi,<br/> nodefs.available<20Gi,<br/> imagefs.available<5Gi
--eviction-max-pod-grace-period int32 30
--eviction-minimum-reclaim string memory.available=500Mi,<br/> nodefs.available=2Gi,,<br/> imagefs.available=2Gi
--eviction-pressure-transition-periodduration 5m0s
--eviction-soft string memory.available<8Gi,<br/> nodefs.available<100Gi,<br/> imagefs.available<20Gi
--eviction-soft-grace-period string memory.available=30s,<br/> nodefs.available=2m,<br/> imagefs.available=2m
--experimental-fail-swap-on +
--experimental-kernel-memcg-notification +
--feature-gates string AllAlpha=false
--file-check-frequency duration 20s
--hairpin-mode string promiscuous-bridge
--healthz-port int32 10248
--image-gc-high-threshold int32 60
--image-gc-low-threshold int32 40
--image-pull-progress-deadline duration 2m0s
--kube-api-qps int32 5
--kube-reserved mapStringString cpu=200m,memory=16G
--kubeconfig string /var/lib/kubelet/kubeconfig
--max-pods int32 50
--minimum-image-ttl-duration duration 1h
--network-plugin string cni
--pod-infra-container-image string vivo.registry.com/google_containers/pause-amd64:3.0
--pod-manifest-path string /var/lib/kubelet/pod_manifest
--port int32 10250
--protect-kernel-defaults +
--read-only-port int32 10255
--require-kubeconfig +
--root-dir string /var/lib/kubelet
--runtime-request-timeout duration 2m0s
--serialize-image-pulls false
--sync-frequency duration 1m0s
--system-reserved mapStringString cpu=100m,memory=32G
--volume-plugin-dir string /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
--volume-stats-agg-period duration 1m0s

 

 

下面是我最终梳理的,认为需要真正显示设置的flag,如下:

/usr/bin/kubelet —address=0.0.0.0  --port=10250  --allow-privileged=false --cluster-dns=10.0.0.1  --cluster-domain=caas.vivo.com --max-pods=50  --network-plugin=cni  --require-kubeconfig  --pod-manifest-path=/etc/kubelet.d/ --pod-infra-container-image=vivo.registry.com/google_containers/pause-amd64:3.0  --eviction-hard=memory.available<4Gi,nodefs.available<20Gi,imagefs.available<5Gi  --eviction-max-pod-grace-period=30  --eviction-minimum-reclaim=memory.available=500Mi,nodefs.available=2Gi,imagefs.available=2Gi  --eviction-pressure-transition-period=5m0s  --eviction-soft=memory.available<8Gi,nodefs.available<100Gi,imagefs.available<20Gi  --eviction-soft-grace-period=memory.available=30s,nodefs.available=2m,imagefs.available=2m  --experimental-kernel-memcg-notification  --experimental-fail-swap-on  --system-reserved=cpu=100m,memory=8G  --kube-reserved=cpu=200m,memory=16G --hairpin-mode=promiscuous-bridge  --image-gc-high-threshold=60  --image-gc-low-threshold=40  --serialize-image-pulls=false --protect-kernel-defaults  --feature-gates=AllAlpha=false 

  

posted @ 2018-03-08 17:27  Cslc-DaweiJ  阅读(12957)  评论(0编辑  收藏  举报