Kubernetes CKA考试之Killer Simulator（上）
写在前面
- 个人微信公众号：密码应用技术实战
- 个人博客园首页：https://www.cnblogs.com/informatics/
- 注：学习交流使用
CKA Simulator Kubernetes 1.25
- 写在前面
- Pre Setup
- Question 1 | Contexts
- Question 2 | Schedule Pod on Master Node
- Question 3 | Scale down StatefulSet
- Question 4 | Pod Ready if Service is reachable
- Question 5 | Kubectl sorting
- Question 6 | Storage, PV, PVC, Pod volume
- Question 7 | Node and Pod Resource Usage
- Question 8 | Get Master Information
- Question 9 | Kill Scheduler, Manual Scheduling
- Question 10 | RBAC ServiceAccount Role RoleBinding
- Question 11 | DaemonSet on all Nodes
- Question 12 | Deployment on all Nodes
- Question 13 | Multi Containers and Pod shared Volume
- Question 14 | Find out Cluster Information
- Question 15 | Cluster Event Logging
Pre Setup
Once you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these:
kubectl
alias k=kubectl # will already be pre-configured
export do="--dry-run=client -o yaml" # k create deploy nginx --image=nginx $do
export now="--force --grace-period 0" # k delete pod x $now
Vim
The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:
set tabstop=2
set expandtab
set shiftwidth=2
More setup suggestions are in the tips section.
Question 1 | Contexts
Task weight: 1%
You have access to multiple clusters from your main terminal through kubectl contexts. Write all those context names into /opt/course/1/contexts.
Next write a command to display the current context into /opt/course/1/context_default_kubectl.sh, the command should use kubectl.
Finally write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh, but without the use of kubectl.
Answer:
Maybe the fastest way is just to run:
k config get-contexts # copy manually
k config get-contexts -o name > /opt/course/1/contexts
Or using jsonpath:
k config view -o yaml # overview
k config view -o jsonpath="{.contexts[*].name}"
k config view -o jsonpath="{.contexts[*].name}" | tr " " "\n" # new lines
k config view -o jsonpath="{.contexts[*].name}" | tr " " "\n" > /opt/course/1/contexts
The content should then look like:
# /opt/course/1/contexts
k8s-c1-H
k8s-c2-AC
k8s-c3-CCC
Next create the first command:
# /opt/course/1/context_default_kubectl.sh
kubectl config current-context
➜ sh /opt/course/1/context_default_kubectl.sh
k8s-c1-H
And the second one:
# /opt/course/1/context_default_no_kubectl.sh
cat ~/.kube/config | grep current
➜ sh /opt/course/1/context_default_no_kubectl.sh
current-context: k8s-c1-H
In the real exam you might need to filter and find information from bigger lists of resources, hence knowing a little jsonpath and simple bash filtering will be helpful.
The second command could also be improved to:
# /opt/course/1/context_default_no_kubectl.sh
cat ~/.kube/config | grep current | sed -e "s/current-context: //"
Question 2 | Schedule Pod on Master Node
Task weight: 3%
Use context: kubectl config use-context k8s-c1-H
Create a single Pod of image httpd:2.4.41-alpine in Namespace default. The Pod should be named pod1 and the container should be named pod1-container. This Pod should only be scheduled on a master node, do not add new labels any nodes.
Answer:
First we find the master node(s) and their taints:
k get node # find master node
k describe node cluster1-master1 | grep Taint -A1 # get master node taints
k get node cluster1-master1 --show-labels # get master node labels
Next we create the Pod template:
# check the export on the very top of this document so we can use $do
k run pod1 --image=httpd:2.4.41-alpine $do > 2.yaml
vim 2.yaml
Perform the necessary changes manually. Use the Kubernetes docs and search for example for tolerations and nodeSelector to find examples:
# 2.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: pod1
name: pod1
spec:
containers:
- image: httpd:2.4.41-alpine
name: pod1-container # change
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
tolerations: # add
- effect: NoSchedule # add
key: node-role.kubernetes.io/control-plane # add
nodeSelector: # add
node-role.kubernetes.io/control-plane: "" # add
status: {}
Important here to add the toleration for running on master nodes, but also the nodeSelector to make sure it only runs on master nodes. If we only specify a toleration the Pod can be scheduled on master or worker nodes.
Now we create it:
k -f 2.yaml create
Let's check if the pod is scheduled:
➜ k get pod pod1 -o wide
NAME READY STATUS RESTARTS ... NODE NOMINATED NODE
pod1 1/1 Running 0 ... cluster1-master1 <none>
Question 3 | Scale down StatefulSet
Task weight: 1%
Use context: kubectl config use-context k8s-c1-H
There are two Pods named o3db-* in Namespace project-c13. C13 management asked you to scale the Pods down to one replica to save resources.
Answer:
If we check the Pods we see two replicas:
➜ k -n project-c13 get pod | grep o3db
o3db-0 1/1 Running 0 52s
o3db-1 1/1 Running 0 42s
From their name it looks like these are managed by a StatefulSet. But if we're not sure we could also check for the most common resources which manage Pods:
➜ k -n project-c13 get deploy,ds,sts | grep o3db
statefulset.apps/o3db 2/2 2m56s
Confirmed, we have to work with a StatefulSet. To find this out we could also look at the Pod labels:
➜ k -n project-c13 get pod --show-labels | grep o3db
o3db-0 1/1 Running 0 3m29s app=nginx,controller-revision-hash=o3db-5fbd4bb9cc,statefulset.kubernetes.io/pod-name=o3db-0
o3db-1 1/1 Running 0 3m19s app=nginx,controller-revision-hash=o3db-5fbd4bb9cc,statefulset.kubernetes.io/pod-name=o3db-1
To fulfil the task we simply run:
➜ k -n project-c13 scale sts o3db --replicas 1
statefulset.apps/o3db scaled
➜ k -n project-c13 get sts o3db
NAME READY AGE
o3db 1/1 4m39s
C13 Mangement is happy again.
Question 4 | Pod Ready if Service is reachable
Task weight: 4%
Use context: kubectl config use-context k8s-c1-H
Do the following in Namespace default. Create a single Pod named ready-if-service-ready of image nginx:1.16.1-alpine. Configure a LivenessProbe which simply runs true. Also configure a ReadinessProbe which does check if the url http://service-am-i-ready:80 is reachable, you can use wget -T2 -O- http://service-am-i-ready:80 for this. Start the Pod and confirm it isn't ready because of the ReadinessProbe.
Create a second Pod named am-i-ready of image nginx:1.16.1-alpine with label id: cross-server-ready. The already existing Service service-am-i-ready should now have that second Pod as endpoint.
Now the first Pod should be in ready state, confirm that.
Answer:
It's a bit of an anti-pattern for one Pod to check another Pod for being ready using probes, hence the normally available readinessProbe.httpGet doesn't work for absolute remote urls. Still the workaround requested in this task should show how probes and Pod<->Service communication works.
First we create the first Pod:
k run ready-if-service-ready --image=nginx:1.16.1-alpine $do > 4_pod1.yaml
vim 4_pod1.yaml
Next perform the necessary additions manually:
# 4_pod1.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: ready-if-service-ready
name: ready-if-service-ready
spec:
containers:
- image: nginx:1.16.1-alpine
name: ready-if-service-ready
resources: {}
livenessProbe: # add from here
exec:
command:
- 'true'
readinessProbe:
exec:
command:
- sh
- -c
- 'wget -T2 -O- http://service-am-i-ready:80' # to here
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
Then create the Pod:
k -f 4_pod1.yaml create
And confirm its in a non-ready state:
➜ k get pod ready-if-service-ready
NAME READY STATUS RESTARTS AGE
ready-if-service-ready 0/1 Running 0 7s
We can also check the reason for this using describe:
➜ k describe pod ready-if-service-ready
...
Warning Unhealthy 18s kubelet, cluster1-worker1 Readiness probe failed: Connecting to service-am-i-ready:80 (10.109.194.234:80)
wget: download timed out
Now we create the second Pod:
➜k run am-i-ready --image=nginx:1.16.1-alpine --labels="id=cross-server-ready"
The already existing Service service-am-i-ready should now have an Endpoint:
➜k describe svc service-am-i-ready
➜k get ep # also possible
Which will result in our first Pod being ready, just give it a minute for the Readiness probe to check again:
➜ k get pod ready-if-service-ready
NAME READY STATUS RESTARTS AGE
ready-if-service-ready 1/1 Running 0 53s
Look at these Pods coworking together!
Question 5 | Kubectl sorting
Task weight: 1%
Use context: kubectl config use-context k8s-c1-H
There are various Pods in all namespaces. Write a command into /opt/course/5/find_pods.sh which lists all Pods sorted by their AGE (metadata.creationTimestamp).
Write a second command into /opt/course/5/find_pods_uid.sh which lists all Pods sorted by field metadata.uid. Use kubectl sorting for both commands.
Answer:
A good resources here (and for many other things) is the kubectl-cheat-sheet. You can reach it fast when searching for "cheat sheet" in the Kubernetes docs.
# /opt/course/5/find_pods.sh
kubectl get pod -A --sort-by=.metadata.creationTimestamp
And to execute:
➜ sh /opt/course/5/find_pods.sh
NAMESPACE NAME ... AGE
kube-system kube-scheduler-cluster1-master1 ... 63m
kube-system etcd-cluster1-master1 ... 63m
kube-system kube-apiserver-cluster1-master1 ... 63m
kube-system kube-controller-manager-cluster1-master1 ... 63m
...
For the second command:
# /opt/course/5/find_pods_uid.sh
kubectl get pod -A --sort-by=.metadata.uid
And to execute:
➜ sh /opt/course/5/find_pods_uid.sh
NAMESPACE NAME ... AGE
kube-system coredns-5644d7b6d9-vwm7g ... 68m
project-c13 c13-3cc-runner-heavy-5486d76dd4-ddvlt ... 63m
project-hamster web-hamster-shop-849966f479-278vp ... 63m
project-c13 c13-3cc-web-646b6c8756-qsg4b ... 63m
Question 6 | Storage, PV, PVC, Pod volume
Task weight: 8%
Use context: kubectl config use-context k8s-c1-H
Create a new PersistentVolume named safari-pv. It should have a capacity of 2Gi, accessMode ReadWriteOnce, hostPath /Volumes/Data and no storageClassName defined.
Next create a new PersistentVolumeClaim in Namespace project-tiger named safari-pvc . It should request 2Gi storage, accessMode ReadWriteOnce and should not define a storageClassName. The PVC should bound to the PV correctly.
Finally create a new Deployment safari in Namespace project-tiger which mounts that volume at /tmp/safari-data. The Pods of that Deployment should be of image httpd:2.4.41-alpine.
Answer
➜ vim 6_pv.yaml
Find an example from https://kubernetes.io/docs and alter it:
# 6_pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: safari-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/Volumes/Data"
Then create it:
➜ k -f 6_pv.yaml create
Next the PersistentVolumeClaim:
➜ vim 6_pvc.yaml
Find an example from https://kubernetes.io/docs and alter it:
# 6_pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: safari-pvc
namespace: project-tiger
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
Then create:
➜ k -f 6_pvc.yaml create
And check that both have the status Bound:
➜ k -n project-tiger get pv,pvc
NAME CAPACITY ... STATUS CLAIM ...
persistentvolume/safari-pv 2Gi ... Bound project-tiger/safari-pvc ...
NAME STATUS VOLUME CAPACITY ...
persistentvolumeclaim/safari-pvc Bound safari-pv 2Gi ...
Next we create a Deployment and mount that volume:
➜ k -n project-tiger create deploy safari \
--image=httpd:2.4.41-alpine $do > 6_dep.yaml
➜ vim 6_dep.yaml
Alter the yaml to mount the volume:
# 6_dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: safari
name: safari
namespace: project-tiger
spec:
replicas: 1
selector:
matchLabels:
app: safari
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: safari
spec:
volumes: # add
- name: data # add
persistentVolumeClaim: # add
claimName: safari-pvc # add
containers:
- image: httpd:2.4.41-alpine
name: container
volumeMounts: # add
- name: data # add
mountPath: /tmp/safari-data # add
k -f 6_dep.yaml create
We can confirm its mounting correctly:
➜ k -n project-tiger describe pod safari-5cbf46d6d-mjhsb | grep -A2 Mounts:
Mounts:
/tmp/safari-data from data (rw) # there it is
/var/run/secrets/kubernetes.io/serviceaccount from default-token-n2sjj (ro)
Question 7 | Node and Pod Resource Usage
Task weight: 1%
Use context: kubectl config use-context k8s-c1-H
The metrics-server has been installed in the cluster. Your college would like to know the kubectl commands to:
- show Nodes resource usage
- show Pods and their containers resource usage
Please write the commands into /opt/course/7/node.sh and /opt/course/7/pod.sh.
Answer:
The command we need to use here is top:
➜ k top -h
Display Resource (CPU/Memory/Storage) usage.
The top command allows you to see the resource consumption for nodes or pods.
This command requires Metrics Server to be correctly configured and working on the server.
Available Commands:
node Display Resource (CPU/Memory/Storage) usage of nodes
pod Display Resource (CPU/Memory/Storage) usage of pods
We see that the metrics server provides information about resource usage:
➜ k top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
cluster1-master1 178m 8% 1091Mi 57%
cluster1-worker1 66m 6% 834Mi 44%
cluster1-worker2 91m 9% 791Mi 41%
We create the first file:
# /opt/course/7/node.sh
kubectl top node
For the second file we might need to check the docs again:
➜ k top pod -h
Display Resource (CPU/Memory/Storage) usage of pods.
...
Namespace in current context is ignored even if specified with --namespace.
--containers=false: If present, print usage of containers within a pod.
--no-headers=false: If present, print output without headers.
...
With this we can finish this task:
# /opt/course/7/pod.sh
kubectl top pod --containers=true
Question 8 | Get Master Information
Task weight: 2%
Use context: kubectl config use-context k8s-c1-H
Ssh into the master node with ssh cluster1-master1. Check how the master components kubelet, kube-apiserver, kube-scheduler, kube-controller-manager and etcd are started/installed on the master node. Also find out the name of the DNS application and how it's started/installed on the master node.
Write your findings into file /opt/course/8/master-components.txt. The file should be structured like:
# /opt/course/8/master-components.txt
kubelet: [TYPE]
kube-apiserver: [TYPE]
kube-scheduler: [TYPE]
kube-controller-manager: [TYPE]
etcd: [TYPE]
dns: [TYPE] [NAME]
Choices of [TYPE] are: not-installed, process, static-pod, pod
Answer:
We could start by finding processes of the requested components, especially the kubelet at first:
➜ ssh cluster1-master1
root@cluster1-master1:~# ps aux | grep kubelet # shows kubelet process
We can see which components are controlled via systemd looking at /etc/systemd/system directory:
➜ root@cluster1-master1:~# find /etc/systemd/system/ | grep kube
/etc/systemd/system/kubelet.service.d
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
/etc/systemd/system/multi-user.target.wants/kubelet.service
➜ root@cluster1-master1:~# find /etc/systemd/system/ | grep etcd
This shows kubelet is controlled via systemd, but no other service named kube nor etcd. It seems that this cluster has been setup using kubeadm, so we check in the default manifests directory:
➜ root@cluster1-master1:~# find /etc/kubernetes/manifests/
/etc/kubernetes/manifests/
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/etcd.yaml
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
(The kubelet could also have a different manifests directory specified via parameter --pod-manifest-path in it's systemd startup config)
This means the main 4 master services are setup as static Pods. Actually, let's check all Pods running on in the kube-system Namespace on the master node:
➜ root@cluster1-master1:~# kubectl -n kube-system get pod -o wide | grep master1
coredns-5644d7b6d9-c4f68 1/1 Running ... cluster1-master1
coredns-5644d7b6d9-t84sc 1/1 Running ... cluster1-master1
etcd-cluster1-master1 1/1 Running ... cluster1-master1
kube-apiserver-cluster1-master1 1/1 Running ... cluster1-master1
kube-controller-manager-cluster1-master1 1/1 Running ... cluster1-master1
kube-proxy-q955p 1/1 Running ... cluster1-master1
kube-scheduler-cluster1-master1 1/1 Running ... cluster1-master1
weave-net-mwj47 2/2 Running ... cluster1-master1
There we see the 5 static pods, with -cluster1-master1 as suffix.
We also see that the dns application seems to be coredns, but how is it controlled?
➜ root@cluster1-master1$ kubectl -n kube-system get ds
NAME DESIRED CURRENT ... NODE SELECTOR AGE
kube-proxy 3 3 ... kubernetes.io/os=linux 155m
weave-net 3 3 ... <none> 155m
➜ root@cluster1-master1$ kubectl -n kube-system get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
coredns 2/2 2 2 155m
Seems like coredns is controlled via a Deployment. We combine our findings in the requested file:
# /opt/course/8/master-components.txt
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: pod coredns
You should be comfortable investigating a running cluster, know different methods on how a cluster and its services can be setup and be able to troubleshoot and find error sources.
Question 9 | Kill Scheduler, Manual Scheduling
Task weight: 5%
Use context: kubectl config use-context k8s-c2-AC
Ssh into the master node with ssh cluster2-master1. Temporarily stop the kube-scheduler, this means in a way that you can start it again afterwards.
Create a single Pod named manual-schedule of image httpd:2.4-alpine, confirm its created but not scheduled on any node.
Now you're the scheduler and have all its power, manually schedule that Pod on node cluster2-master1. Make sure it's running.
Start the kube-scheduler again and confirm its running correctly by creating a second Pod named manual-schedule2 of image httpd:2.4-alpine and check if it's running on cluster2-worker1.
Answer:
Stop the Scheduler
First we find the master node:
➜ k get node
NAME STATUS ROLES AGE VERSION
cluster2-master1 Ready master 26h v1.25.2
cluster2-worker1 Ready <none> 26h v1.25.2
Then we connect and check if the scheduler is running:
➜ ssh cluster2-master1
➜ root@cluster2-master1:~# kubectl -n kube-system get pod | grep schedule
kube-scheduler-cluster2-master1 1/1 Running 0 6s
Kill the Scheduler (temporarily):
➜ root@cluster2-master1:~# cd /etc/kubernetes/manifests/
➜ root@cluster2-master1:~# mv kube-scheduler.yaml ..
And it should be stopped:
➜ root@cluster2-master1:~# kubectl -n kube-system get pod | grep schedule
Create a Pod
Now we create the Pod:
➜ k run manual-schedule --image=httpd:2.4-alpine
And confirm it has no node assigned:
➜ k get pod manual-schedule -o wide
NAME READY STATUS ... NODE NOMINATED NODE
manual-schedule 0/1 Pending ... <none> <none>
Manually schedule the Pod
Let's play the scheduler now:
k get pod manual-schedule -o yaml > 9.yaml
# 9.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2020-09-04T15:51:02Z"
labels:
run: manual-schedule
managedFields:
...
manager: kubectl-run
operation: Update
time: "2020-09-04T15:51:02Z"
name: manual-schedule
namespace: default
resourceVersion: "3515"
selfLink: /api/v1/namespaces/default/pods/manual-schedule
uid: 8e9d2532-4779-4e63-b5af-feb82c74a935
spec:
nodeName: cluster2-master1 # add the master node name
containers:
- image: httpd:2.4-alpine
imagePullPolicy: IfNotPresent
name: manual-schedule
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-nxnc7
readOnly: true
dnsPolicy: ClusterFirst
...
The only thing a scheduler does, is that it sets the nodeName for a Pod declaration. How it finds the correct node to schedule on, that's a very much complicated matter and takes many variables into account.
As we cannot kubectl apply or kubectl edit , in this case we need to delete and create or replace:
k -f 9.yaml replace --force
How does it look?
➜ k get pod manual-schedule -o wide
NAME READY STATUS ... NODE
manual-schedule 1/1 Running ... cluster2-master1
It looks like our Pod is running on the master now as requested, although no tolerations were specified. Only the scheduler takes tains/tolerations/affinity into account when finding the correct node name. That's why its still possible to assign Pods manually directly to a master node and skip the scheduler.
Start the scheduler again
➜ ssh cluster2-master1
➜ root@cluster2-master1:~# cd /etc/kubernetes/manifests/
➜ root@cluster2-master1:~# mv ../kube-scheduler.yaml .
Checks its running:
➜ root@cluster2-master1:~# kubectl -n kube-system get pod | grep schedule
kube-scheduler-cluster2-master1 1/1 Running 0 16s
Schedule a second test Pod:
➜ k run manual-schedule2 --image=httpd:2.4-alpine
➜ k get pod -o wide | grep schedule
manual-schedule 1/1 Running ... cluster2-master1
manual-schedule2 1/1 Running ... cluster2-worker1
Back to normal.
Question 10 | RBAC ServiceAccount Role RoleBinding
Task weight: 6%
Use context: kubectl config use-context k8s-c1-H
Create a new ServiceAccount processor in Namespace project-hamster. Create a Role and RoleBinding, both named processor as well. These should allow the new SA to only create Secrets and ConfigMaps in that Namespace.
Answer:
Let's talk a little about RBAC resources
A ClusterRole|Role defines a set of permissions and where it is available, in the whole cluster or just a single Namespace.
A ClusterRoleBinding|RoleBinding connects a set of permissions with an account and defines where it is applied, in the whole cluster or just a single Namespace.
Because of this there are 4 different RBAC combinations and 3 valid ones:
- Role + RoleBinding (available in single Namespace, applied in single Namespace)
- ClusterRole + ClusterRoleBinding (available cluster-wide, applied cluster-wide)
- ClusterRole + RoleBinding (available cluster-wide, applied in single Namespace)
- Role + ClusterRoleBinding (NOT POSSIBLE: available in single Namespace, applied cluster-wide)
To the solution
We first create the ServiceAccount:
➜ k -n project-hamster create sa processor
serviceaccount/processor created
Then for the Role:
➜ k -n project-hamster create role -h # examples
So we execute:
k -n project-hamster create role processor \
--verb=create \
--resource=secret \
--resource=configmap
Which will create a Role like:
# kubectl -n project-hamster create role processor --verb=create --resource=secret --resource=configmap
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: processor
namespace: project-hamster
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- create
Now we bind the Role to the ServiceAccount:
➜ k -n project-hamster create rolebinding -h # examples
So we create it:
k -n project-hamster create rolebinding processor \
--role processor \
--serviceaccount project-hamster:processor
This will create a RoleBinding like:
# kubectl -n project-hamster create rolebinding processor --role processor --serviceaccount project-hamster:processor
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: processor
namespace: project-hamster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: processor
subjects:
- kind: ServiceAccount
name: processor
namespace: project-hamster
To test our RBAC setup we can use kubectl auth can-i:
k auth can-i -h # examples
Like this:
➜ k -n project-hamster auth can-i create secret \
--as system:serviceaccount:project-hamster:processor
yes
➜ k -n project-hamster auth can-i create configmap \
--as system:serviceaccount:project-hamster:processor
yes
➜ k -n project-hamster auth can-i create pod \
--as system:serviceaccount:project-hamster:processor
no
➜ k -n project-hamster auth can-i delete secret \
--as system:serviceaccount:project-hamster:processor
no
➜ k -n project-hamster auth can-i get configmap \
--as system:serviceaccount:project-hamster:processor
no
Done.
Question 11 | DaemonSet on all Nodes
Task weight: 4%
Use context: kubectl config use-context k8s-c1-H
Use Namespace project-tiger for the following. Create a DaemonSet named ds-important with image httpd:2.4-alpine and labels id=ds-important and uuid=18426a0b-5f59-4e10-923f-c0e078e82462. The Pods it creates should request 10 millicore cpu and 10 mebibyte memory. The Pods of that DaemonSet should run on all nodes, master and worker.
Answer:
As of now we aren't able to create a DaemonSet directly using kubectl, so we create a Deployment and just change it up:
➜ k -n project-tiger create deployment --image=httpd:2.4-alpine ds-important $do > 11.yaml
➜ vim 11.yaml
(Sure you could also search for a DaemonSet example yaml in the Kubernetes docs and alter it.)
Then we adjust the yaml to:
# 11.yaml
apiVersion: apps/v1
kind: DaemonSet # change from Deployment to Daemonset
metadata:
creationTimestamp: null
labels: # add
id: ds-important # add
uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 # add
name: ds-important
namespace: project-tiger # important
spec:
#replicas: 1 # remove
selector:
matchLabels:
id: ds-important # add
uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 # add
#strategy: {} # remove
template:
metadata:
creationTimestamp: null
labels:
id: ds-important # add
uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 # add
spec:
containers:
- image: httpd:2.4-alpine
name: ds-important
resources:
requests: # add
cpu: 10m # add
memory: 10Mi # add
tolerations: # add
- effect: NoSchedule # add
key: node-role.kubernetes.io/control-plane # add
#status: {} # remove
It was requested that the DaemonSet runs on all nodes, so we need to specify the toleration for this.
Let's confirm:
k -f 11.yaml create
➜ k -n project-tiger get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
ds-important 3 3 3 3 3 <none> 8s
➜ k -n project-tiger get pod -l id=ds-important -o wide
NAME READY STATUS NODE
ds-important-6pvgm 1/1 Running ... cluster1-worker1
ds-important-lh5ts 1/1 Running ... cluster1-master1
ds-important-qhjcq 1/1 Running ... cluster1-worker2
Question 12 | Deployment on all Nodes
Task weight: 6%
Use context: kubectl config use-context k8s-c1-H
Use Namespace project-tiger for the following. Create a Deployment named deploy-important with label id=very-important (the Pods should also have this label) and 3 replicas. It should contain two containers, the first named container1 with image nginx:1.17.6-alpine and the second one named container2 with image kubernetes/pause.
There should be only ever one Pod of that Deployment running on one worker node. We have two worker nodes: cluster1-worker1 and cluster1-worker2. Because the Deployment has three replicas the result should be that on both nodes one Pod is running. The third Pod won't be scheduled, unless a new worker node will be added.
In a way we kind of simulate the behaviour of a DaemonSet here, but using a Deployment and a fixed number of replicas.
Answer:
There are two possible ways, one using podAntiAffinity and one using topologySpreadConstraint.
- PodAntiAffinity
The idea here is that we create a "Inter-pod anti-affinity" which allows us to say a Pod should only be scheduled on a node where another Pod of a specific label (here the same label) is not already running.
Let's begin by creating the Deployment template:
k -n project-tiger create deployment \
--image=nginx:1.17.6-alpine deploy-important $do > 12.yaml
vim 12.yaml
Then change the yaml to:
# 12.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
id: very-important # change
name: deploy-important
namespace: project-tiger # important
spec:
replicas: 3 # change
selector:
matchLabels:
id: very-important # change
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
id: very-important # change
spec:
containers:
- image: nginx:1.17.6-alpine
name: container1 # change
resources: {}
- image: kubernetes/pause # add
name: container2 # add
affinity: # add
podAntiAffinity: # add
requiredDuringSchedulingIgnoredDuringExecution: # add
- labelSelector: # add
matchExpressions: # add
- key: id # add
operator: In # add
values: # add
- very-important # add
topologyKey: kubernetes.io/hostname # add
status: {}
Specify a topologyKey, which is a pre-populated Kubernetes label, you can find this by describing a node.
- TopologySpreadConstraints
We can achieve the same withtopologySpreadConstraints. Best to try out and play with both.
# 12.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
id: very-important # change
name: deploy-important
namespace: project-tiger # important
spec:
replicas: 3 # change
selector:
matchLabels:
id: very-important # change
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
id: very-important # change
spec:
containers:
- image: nginx:1.17.6-alpine
name: container1 # change
resources: {}
- image: kubernetes/pause # add
name: container2 # add
topologySpreadConstraints: # add
- maxSkew: 1 # add
topologyKey: kubernetes.io/hostname # add
whenUnsatisfiable: DoNotSchedule # add
labelSelector: # add
matchLabels: # add
id: very-important # add
status: {}
Apply and Run
Let's run it:
➜ k -f 12.yaml create
Then we check the Deployment status where it shows 2/3 ready count:
➜ k -n project-tiger get deploy -l id=very-important
NAME READY UP-TO-DATE AVAILABLE AGE
deploy-important 2/3 3 2 2m35s
And running the following we see one Pod on each worker node and one not scheduled.
➜ k -n project-tiger get pod -o wide -l id=very-important
NAME READY STATUS ... NODE
deploy-important-58db9db6fc-9ljpw 2/2 Running ... cluster1-worker1
deploy-important-58db9db6fc-lnxdb 0/2 Pending ... <none>
deploy-important-58db9db6fc-p2rz8 2/2 Running ... cluster1-worker2
If we kubectl describe the Pod deploy-important-58db9db6fc-lnxdb it will show us the reason for not scheduling is our implemented podAntiAffinity ruling:
Warning FailedScheduling 63s (x3 over 65s) default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/control-plane: }, that the pod didn't tolerate, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't satisfy existing pods anti-affinity rules.
Or our topologySpreadConstraints:
Warning FailedScheduling 16s default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/control-plane: }, that the pod didn't tolerate, 2 node(s) didn't match pod topology spread constraints.
Question 13 | Multi Containers and Pod shared Volume
Task weight: 4%
Use context: kubectl config use-context k8s-c1-H
Create a Pod named multi-container-playground in Namespace default with three containers, named c1, c2 and c3. There should be a volume attached to that Pod and mounted into every container, but the volume shouldn't be persisted or shared with other Pods.
Container c1 should be of image nginx:1.17.6-alpine and have the name of the node where its Pod is running available as environment variable MY_NODE_NAME.
Container c2 should be of image busybox:1.31.1 and write the output of the date command every second in the shared volume into file date.log. You can use while true; do date >> /your/vol/path/date.log; sleep 1; done for this.
Container c3 should be of image busybox:1.31.1 and constantly send the content of file date.log from the shared volume to stdout. You can use tail -f /your/vol/path/date.log for this.
Check the logs of container c3 to confirm correct setup.
Answer:
First we create the Pod template:
➜ k run multi-container-playground --image=nginx:1.17.6-alpine $do > 13.yaml
vim 13.yaml
And add the other containers and the commands they should execute:
# 13.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: multi-container-playground
name: multi-container-playground
spec:
containers:
- image: nginx:1.17.6-alpine
name: c1 # change
resources: {}
env: # add
- name: MY_NODE_NAME # add
valueFrom: # add
fieldRef: # add
fieldPath: spec.nodeName # add
volumeMounts: # add
- name: vol # add
mountPath: /vol # add
- image: busybox:1.31.1 # add
name: c2 # add
command: ["sh", "-c", "while true; do date >> /vol/date.log; sleep 1; done"] # add
volumeMounts: # add
- name: vol # add
mountPath: /vol # add
- image: busybox:1.31.1 # add
name: c3 # add
command: ["sh", "-c", "tail -f /vol/date.log"] # add
volumeMounts: # add
- name: vol # add
mountPath: /vol # add
dnsPolicy: ClusterFirst
restartPolicy: Always
volumes: # add
- name: vol # add
emptyDir: {} # add
status: {}
➜ k -f 13.yaml create
Oh boy, lot's of requested things. We check if everything is good with the Pod:
➜ k get pod multi-container-playground
NAME READY STATUS RESTARTS AGE
multi-container-playground 3/3 Running 0 95s
Good, then we check if container c1 has the requested node name as env variable:
➜ k exec multi-container-playground -c c1 -- env | grep MY
MY_NODE_NAME=cluster1-worker2
And finally we check the logging:
➜ k logs multi-container-playground -c c3
Sat Dec 7 16:05:10 UTC 2077
Sat Dec 7 16:05:11 UTC 2077
Sat Dec 7 16:05:12 UTC 2077
Sat Dec 7 16:05:13 UTC 2077
Sat Dec 7 16:05:14 UTC 2077
Sat Dec 7 16:05:15 UTC 2077
Sat Dec 7 16:05:16 UTC 2077
Question 14 | Find out Cluster Information
Task weight: 2%
Use context: kubectl config use-context k8s-c1-H
You're ask to find out following information about the cluster k8s-c1-H:
- How many master nodes are available?
- How many worker nodes are available?
- What is the Service CIDR?
- Which Networking (or CNI Plugin) is configured and where is its config file?
- Which suffix will static pods have that run on cluster1-worker1?
Write your answers into file /opt/course/14/cluster-info, structured like this:
# /opt/course/14/cluster-info
1: [ANSWER]
2: [ANSWER]
3: [ANSWER]
4: [ANSWER]
5: [ANSWER]
Answer:
How many master and worker nodes are available?
➜ k get node
NAME STATUS ROLES AGE VERSION
cluster1-master1 Ready master 27h v1.25.2
cluster1-worker1 Ready <none> 27h v1.25.2
cluster1-worker2 Ready <none> 27h v1.25.2
We see one master and two workers.
What is the Service CIDR?
➜ ssh cluster1-master1
➜ root@cluster1-master1:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep range
- --service-cluster-ip-range=10.96.0.0/12
Which Networking (or CNI Plugin) is configured and where is its config file?
➜ root@cluster1-master1:~# find /etc/cni/net.d/
/etc/cni/net.d/
/etc/cni/net.d/10-weave.conflist
➜ root@cluster1-master1:~# cat /etc/cni/net.d/10-weave.conflist
{
"cniVersion": "0.3.0",
"name": "weave",
...
By default the kubelet looks into /etc/cni/net.d to discover the CNI plugins. This will be the same on every master and worker nodes.
Which suffix will static pods have that run on cluster1-worker1?
The suffix is the node hostname with a leading hyphen. It used to be -static in earlier Kubernetes versions.
Result
The resulting /opt/course/14/cluster-info could look like:
# /opt/course/14/cluster-info
# How many master nodes are available?
1: 1
# How many worker nodes are available?
2: 2
# What is the Service CIDR?
3: 10.96.0.0/12
# Which Networking (or CNI Plugin) is configured and where is its config file?
4: Weave, /etc/cni/net.d/10-weave.conflist
# Which suffix will static pods have that run on cluster1-worker1?
5: -cluster1-worker1
Question 15 | Cluster Event Logging
Task weight: 3%
Use context: kubectl config use-context k8s-c2-AC
Write a command into /opt/course/15/cluster_events.sh which shows the latest events in the whole cluster, ordered by time (metadata.creationTimestamp). Use kubectl for it.
Now kill the kube-proxy Pod running on node cluster2-worker1 and write the events this caused into /opt/course/15/pod_kill.log.
Finally kill the containerd container of the kube-proxy Pod on node cluster2-worker1 and write the events into /opt/course/15/container_kill.log.
Do you notice differences in the events both actions caused?
Answer:
# /opt/course/15/cluster_events.sh
➜ kubectl get events -A --sort-by=.metadata.creationTimestamp
Now we kill the kube-proxy Pod:
➜ k -n kube-system get pod -o wide | grep proxy # find pod running on cluster2-worker1
➜ k -n kube-system delete pod kube-proxy-z64cg
Now check the events:
sh /opt/course/15/cluster_events.sh
Write the events the killing caused into /opt/course/15/pod_kill.log:
# /opt/course/15/pod_kill.log
kube-system 9s Normal Killing pod/kube-proxy-jsv7t ...
kube-system 3s Normal SuccessfulCreate daemonset/kube-proxy ...
kube-system <unknown> Normal Scheduled pod/kube-proxy-m52sx ...
default 2s Normal Starting node/cluster2-worker1 ...
kube-system 2s Normal Created pod/kube-proxy-m52sx ...
kube-system 2s Normal Pulled pod/kube-proxy-m52sx ...
kube-system 2s Normal Started pod/kube-proxy-m52sx ...
Finally we will try to provoke events by killing the container belonging to the container of the kube-proxy Pod:
➜ ssh cluster2-worker1
➜ root@cluster2-worker1:~# crictl ps | grep kube-proxy
1e020b43c4423 36c4ebbc9d979 About an hour ago Running kube-proxy ...
➜ root@cluster2-worker1:~# crictl rm 1e020b43c4423
1e020b43c4423
➜ root@cluster2-worker1:~# crictl ps | grep kube-proxy
0ae4245707910 36c4ebbc9d979 17 seconds ago Running kube-proxy ...
We killed the main container (1e020b43c4423), but also noticed that a new container (0ae4245707910) was directly created. Thanks Kubernetes!
Now we see if this caused events again and we write those into the second file:
sh /opt/course/15/cluster_events.sh
# /opt/course/15/container_kill.log
kube-system 13s Normal Created pod/kube-proxy-m52sx ...
kube-system 13s Normal Pulled pod/kube-proxy-m52sx ...
kube-system 13s Normal Started pod/kube-proxy-m52sx ...
Comparing the events we see that：
- when we deleted the whole Pod there were more things to be done, hence more events. For example was the DaemonSet in the game to re-create the missing Pod.
- Where when we manually killed the main container of the Pod, the Pod would still exist but only its container needed to be re-created, hence less events.
