分析kube-apiserver单次创建namespace耗时

日志输出

# 业务日志
I0220 22:12:39.149364   40965 multi_config_multi_clientset.go:63] begin to wait cache sync
I0220 22:12:39.250461   40965 multi_config_multi_clientset.go:67] wait cache sync end
I0220 22:12:39.256440   40965 multi_config_multi_clientset.go:90] create ns test success
I0220 22:12:39.256459   40965 multi_config_multi_clientset.go:92] cost time 5 ms

# kube-apiserver日志
I0220 22:12:39.252062   40933 handler.go:153] kube-aggregator: POST "/api/v1/namespaces" satisfied by nonGoRestful
I0220 22:12:39.252216   40933 pathrecorder.go:253] kube-aggregator: "/api/v1/namespaces" satisfied by NotFoundHandler
I0220 22:12:39.252234   40933 handler.go:143] kube-apiserver: POST "/api/v1/namespaces" satisfied by gorestful with webservice /api/v1
I0220 22:12:39.255738   40933 httplog.go:89] "HTTP" verb="POST" URI="/api/v1/namespaces" latency="3.902358ms" userAgent="test/v0.0.0 (linux/amd64) kubernetes/$Format" srcIP="192.168.0.105:57696" resp=201
I0220 22:12:39.263293   40933 shared_informer.go:270] caches populated

不足以确定详细流程，后面通过kube-apiserver和etcd tracing来分析。

安装k8s v1.27.0

准备

查看k8s v1.27.0镜像版本
kubeadm config images list --kubernetes-version=v1.27.0

registry.k8s.io/kube-apiserver:v1.27.0
registry.k8s.io/kube-controller-manager:v1.27.0
registry.k8s.io/kube-scheduler:v1.27.0
registry.k8s.io/kube-proxy:v1.27.0
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1

安装Docker

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
apt-get update
apt-get install ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update

apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

安装k8s

cat <<EOF | kind create cluster --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.27.0
- role: worker
  image: kindest/node:v1.27.0
EOF

启动分布式追踪UI jaeger容器

docker run -d --name jaeger \
  --network host \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.43

配置kube-apiserver

vim /etc/kubernetes/pki/apitracing-config.yaml
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
endpoint: <JaegerIP:4317>
samplingRatePerMillion: 100000

特性门控APIServerTracing从1.27开始默认是true。

kube-apiserver启动参数增加--tracing-config-file=/etc/kubernetes/pki/apitracing-config.yaml

配置etcd

etcd启动参数增加
--experimental-distributed-tracing-address=<JaegerIP:4317>
--experimental-distributed-tracing-service-name=etcd
--experimental-enable-distributed-tracing=true

获取kind k8s的kubeconfig文件

kubectl config view --raw

打开jaeger页面查看

业务日志

etcd耗时很少，主要时间花费在kube-apiserver。

参考资料

https://blog.csdn.net/DaoCloud_daoke/article/details/131174429