Kubernetes 单节点部署步骤（kubeadm）

主机环境预设

本示例中的Kubernetes集群部署将基于以下环境进行。

OS: Ubuntu 20.04.5
Kubernetes：v1.26.0
Container Runtime: Docker CE 20.10.22
CRI：cri-dockerd v0.3.0

设定时钟同步

若节点可直接访问互联网，安装chrony程序包后，可直接启动chronyd系统服务，并设定其随系统引导而启动。随后，chronyd服务即能够从默认的时间服务器同步时间。

root@k8s-master:~# apt install chrony

root@k8s-master:~# systemctl start chrony.service
root@k8s-master:~# systemctl enable chrony.service

root@K8S-master:~# vi /etc/hosts
127.0.0.1 localhost
127.0.1.1 awen

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.4.7.132 K8S-master.awen.com K8S-master kubeapi.awen.com
10.4.7.131 K8S-node01.awen.com K8S-node01
10.4.7.133 K8S-node02.awen.com K8S-node02
10.4.7.134 K8S-node03.awen.com K8S-node03

root@k8s-master:~# systemctl --type swap --all

UNIT LOAD ACTIVE SUB DESCRIPTION

swap.img.swap loaded active active /swap.img

LOAD = Reflects whether the unit definition was properly loaded.

ACTIVE = The high-level unit activation state, i.e. generalization of SUB.

SUB = The low-level unit activation state, values depend on unit type.

1 loaded units listed.

To show all installed unit files use 'systemctl list-unit-files'.

root@k8s-master:~# sed -r -i '/\/swap/s@^@#@' /etc/fstab

root@k8s-master:~# swapoff -a
Killed

root@k8s-master:~# vi /etc/sysctl.conf

vm.overcommit_memory=0

root@k8s-master:~# reboot

reboot

禁用默认的防火墙服务

Ubuntu和Debian等Linux发行版默认使用ufw（Uncomplicated FireWall）作为前端来简化 iptables的使用，处于启用状态时，它默认会生成一些规则以加强系统安全。出于降低配置复杂度之目的，本文选择直接将其禁用。

~# ufw disable

~# ufw status

安装程序包

提示：以下操作需要在本示例中的所有四台主机上分别进行。

安装并启动docker

root@k8s-master:~# apt -y install apt-transport-https ca-certificates curl software-properties-common

root@k8s-master:~# curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add -

root@k8s-master:~# add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable".

root@k8s-master:~# apt update

root@k8s-master:~# apt install docker-ce

kubelet需要让docker容器引擎使用systemd作为CGroup的驱动，其默认值为cgroupfs，因而，我们还需要编辑docker的配置文件/etc/docker/daemon.json，添加如下内容，其中的registry-mirrors用于指明使用的镜像加速服务。

root@k8s-master:/etc/docker# cat /etc/docker/daemon.json
{
"registry-mirrors": [
"https://registry.docker-cn.com"
],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "200m"
},
"storage-driver": "overlay2"
}

root@k8s-master:/etc/docker# systemctl daemon-reload
root@k8s-master:/etc/docker# systemctl start docker.service
root@k8s-master:/etc/docker# systemctl restart docker.service
root@k8s-master:/etc/docker# systemctl status docker.service

安装cri-dockerd

Kubernetes自v1.24移除了对docker-shim的支持，而Docker Engine默认又不支持CRI规范，因而二者将无法直接完成整合。为此，Mirantis和Docker联合创建了cri-dockerd项目，用于为Docker Engine提供一个能够支持到CRI规范的垫片，从而能够让Kubernetes基于CRI控制Docker 。

项目地址：https://github.com/Mirantis/cri-dockerd

cri-dockerd项目提供了预制的二进制格式的程序包，用户按需下载相应的系统和对应平台的版本即可完成安装，这里以Ubuntu 2004 64bits系统环境，以及cri-dockerd目前最新的程序版本v0.3.0为例。

root@k8s-master:~# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.0/cri-dockerd_0.3.0.3-0.ubuntu-jammy_amd64.deb

root@k8s-master:~# apt install ./cri-dockerd_0.3.0.3-0.ubuntu-jammy_amd64.deb

root@K8S-master:~# apt --fix-broken install

完成安装后，相应的服务cri-dockerd.service便会自动启动。我们也可以使用如下命令进行验证，若服务处于Running状态即可进行后续步骤。

root@k8s-master:~# systemctl status cri-docker.service

安装kubelet、kubeadm和kubectl

首先，在各主机上生成kubelet和kubeadm等相关程序包的仓库，这里以阿里云的镜像服务为例：

root@k8s-master:~# apt update && apt install -y apt-transport-https curl

root@k8s-master:~# curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

root@k8s-master:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list

deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main

EOF

root@k8s-master:~# apt update

接着，在各主机安装kubelet、kubeadm和kubectl等程序包，并将其设置为随系统启动而自动引导：

root@k8s-master:~# apt install -y kubelet kubeadm kubectl

root@k8s-master:~# apt install kubeadm=1.26.0-00 kubectl=1.26.0-00 kubelet=1.26.0-00

安装完成后，要确保kubeadm等程序文件的版本，这将也是后面初始化Kubernetes集群时需要明确指定的版本号。

整合kubelet和cri-dockerd

仅支持CRI规范的kubelet需要经由遵循该规范的cri-dockerd完成与docker-ce的整合。

配置cri-dockerd

配置cri-dockerd，确保其能够正确加载到CNI插件。编辑/usr/lib/systemd/system/cri-docker.service文件，确保其[Service]配置段中的ExecStart的值类似如下内容。

root@k8s-master:~# vi /usr/lib/systemd/system/cri-docker.service

添加如下

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d

需要添加的各配置参数（各参数的值要与系统部署的CNI插件的实际路径相对应）：

--network-plugin：指定网络插件规范的类型，这里要使用CNI；
--cni-bin-dir：指定CNI插件二进制程序文件的搜索目录；
--cni-cache-dir：CNI插件使用的缓存目录；
--cni-conf-dir：CNI插件加载配置文件的目录；

配置完成后，重载并重启cri-docker.service服务。

root@k8s-master:~# mkdir /etc/sysconfig/

root@k8s-master:~# vim /etc/sysconfig/kubelet
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/cri-dockerd.sock"

root@k8s-master:~# systemctl daemon-reload && systemctl restart cri-docker.service

初始化第一个主节点

该步骤开始尝试构建Kubernetes集群的master节点，配置完成后，各worker节点直接加入到集群中的即可。需要特别说明的是，由kubeadm部署的Kubernetes集群上，集群核心组件kube-apiserver、kube-controller-manager、kube-scheduler和etcd等均会以静态Pod的形式运行，它们所依赖的镜像文件默认来自于registry.k8s.io这一Registry服务之上。但我们无法直接访问该服务，常用的解决办法有如下两种，本示例将选择使用更易于使用的前一种方式。

使用能够到达该服务的代理服务；
使用国内的镜像服务器上的服务，例如registry.aliyuncs.com/google_containers等。

初始化master节点（在master01上完成如下操作）

在运行初始化命令之前先运行如下命令单独获取相关的镜像文件，而后再运行后面的kubeadm init命令，以便于观察到镜像文件的下载过程。

root@k8s-master:~# kubeadm config images list
registry.k8s.io/kube-apiserver:v1.26.1
registry.k8s.io/kube-controller-manager:v1.26.1
registry.k8s.io/kube-scheduler:v1.26.1
registry.k8s.io/kube-proxy:v1.26.1
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.6-0
registry.k8s.io/coredns/coredns:v1.9.3

root@k8s-master:~# kubeadm config images pull --cri-socket unix:///run/cri-dockerd.sock 会超时

无法访问grc.io时，可以在上面的命令中使用“--image-repository=registry.aliyuncs.com/google_containers”选项，以便从国内的镜像服务中获取各Image；

root@k8s-master:~# kubeadm config images pull --cri-socket unix:///run/cri-dockerd.sock

root@K8S-master:~# docker pull registry.aliyuncs.com/google_containers/pause:3.6

root@K8S-master:~# docker tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6

docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.26.0

docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.26.0

docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.26.0

docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.26.0

docker pull registry.aliyuncs.com/google_containers/etcd:3.5.6-0

docker pull registry.aliyuncs.com/google_containers/pause:3.9

docker pull registry.aliyuncs.com/google_containers/coredns:v1.9.3

docker pull registry.aliyuncs.com/google_containers/pause:3.6

Bootstrap token 引导认证 --upload-certs 便捷加入节点成为控制平面

pod所使用的网络IP 与所使用的网络插件匹配 flannl默认使用10.244 calico默认使用192.168 都是16位的掩码

root@k8s-master:~# kubeadm init --control-plane-endpoint="kubeapi.awen.com" --kubernetes-version=v1.26.0 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --token-ttl=0 --cri-socket unix:///run/cri-dockerd.sock --upload-certs --image-repository=registry.aliyuncs.com/google_containers

root@K8S-master:~# mkdir -p $HOME/.kube
root@K8S-master:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@K8S-master:~# chown $(id -u):$(id -g) $HOME/.kube/config

root@K8S-master:~# kubectl get nodes

NAME STATUS ROLES AGE VERSION

k8s-master.awen.com NotReady control-plane 50m v1.26.0

部署网络插件

Kubernetes系统上Pod网络的实现依赖于第三方插件进行，这类插件有近数十种之多，较为著名的有flannel、calico、canal和kube-router等，简单易用的实现是为CoreOS提供的flannel项目。下面的命令用于在线部署flannel至Kubernetes系统之上：

首先，下载适配系统及硬件平台环境的flanneld至每个节点，并放置于/opt/bin/目录下。我们这里选用flanneld-amd64，目前最新的版本为v0.20.2，因而，我们需要在集群的每个节点上执行如下命令：

root@K8S-master:~# mkdir /opt/bin/

root@K8S-master:~# wget https://github.com/flannel-io/flannel/releases/download/v0.20.2/flanneld-amd64

root@K8S-master:~# cp flanneld-amd64 /opt/bin/flanneld
root@K8S-master:~# chmod +x /opt/bin/flanneld

拷贝其他服务器

root@K8S-master:~# docker pull docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2

root@K8S-master:~# docker pull docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0

root@K8S-master:~# kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

root@K8S-master:~# kubectl get pods -n kube-flannel

NAME READY STATUS RESTARTS AGE

kube-flannel-ds-xtrq4 1/1 Running 0 71s

root@K8S-master:~# kubectl get nodes

NAME STATUS ROLES AGE VERSION

k8s-master.awen.com Ready control-plane 3h19m v1.26.0

root@K8S-node01:~# kubeadm join kubeapi.awen.com:6443 --token liulgc.qa6gyf4mfvb8tgoi --discovery-token-ca-cert-hash sha256:5b7f4f7fa0ebadee5867afd80f1d8a16ae8c5c50c94bf86b53fd9e951745d9c0 --cri-socket unix:///run/cri-dockerd.sock

root@K8S-master:~# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE

coredns-5bbd96d687-2gl77 1/1 Running 0 7h10m

coredns-5bbd96d687-r7tt4 1/1 Running 0 7h10m

etcd-k8s-master.awen.com 1/1 Running 0 7h10m

kube-apiserver-k8s-master.awen.com 1/1 Running 0 7h10m

kube-controller-manager-k8s-master.awen.com 1/1 Running 0 7h10m

kube-proxy-fhzdw 1/1 Running 0 3m13s

kube-proxy-gbvbl 1/1 Running 0 60s

kube-proxy-lw7vv 1/1 Running 0 7h10m

kube-proxy-ww2tc 1/1 Running 0 12m

kube-scheduler-k8s-master.awen.com 1/1 Running 0 7h10m

root@K8S-master:~# kubectl api-versions
admissionregistration.k8s.io/v1
apiextensions.k8s.io/v1
apiregistration.k8s.io/v1
apps/v1
authentication.k8s.io/v1
authorization.k8s.io/v1
autoscaling/v1
autoscaling/v2
batch/v1
certificates.k8s.io/v1
coordination.k8s.io/v1
discovery.k8s.io/v1
events.k8s.io/v1
flowcontrol.apiserver.k8s.io/v1beta2
flowcontrol.apiserver.k8s.io/v1beta3
networking.k8s.io/v1
node.k8s.io/v1
policy/v1
rbac.authorization.k8s.io/v1
scheduling.k8s.io/v1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

一个群组之内有多少个资源类型

root@K8S-master:~# kubectl api-resources --api-group=apps

NAME SHORTNAMES APIVERSION NAMESPACED KIND

controllerrevisions apps/v1 true ControllerRevision

daemonsets ds apps/v1 true DaemonSet

deployments deploy apps/v1 true Deployment

replicasets rs apps/v1 true ReplicaSet

statefulsets sts apps/v1 true StatefulSet

资源类型的字段

命令式命令部署容器化应用：
创建一个Deployment，部署无状态应用
~# kubectl create deployment --help
几个常用选项：
--image=IMAGE：编排运行的Pod模板上的容器默认使用Image；
--replicas=Num：运行的Pod副本数量；
--dry-run=client：仅测试创建是否能够成功完成，但并不真正创建资源；

root@K8S-master:~# kubectl create deployment demoapp --image=ikubernetes/demoapp:v1.1 --replicas=3 --dry-run=client -o yaml

apiVersion: apps/v1

kind: Deployment

metadata: 定义对象元数据

creationTimestamp: null

labels:

app: demoapp

spec: 定义资源的终态

replicas: 3

selector:

matchLabels:

app: demoapp

strategy: {}

template:

metadata:

creationTimestamp: null

labels:

app: demoapp

spec:

containers:

- image: ikubernetes:v1.1

resources: {}

status: {}

root@K8S-master:~# kubectl get pod demoapp-55b98fb4d5-cm82z -o yaml

apiVersion: v1

kind: Pod

metadata:

creationTimestamp: "2023-01-24T11:37:30Z"

generateName: demoapp-55b98fb4d5-

labels:

app: demoapp

pod-template-hash: 55b98fb4d5

namespace: default

ownerReferences:

- apiVersion: apps/v1

blockOwnerDeletion: true

controller: true

kind: ReplicaSet

uid: 737642f1-5938-4dab-a6eb-2625f66f80a1

resourceVersion: "54816"

uid: 2a3be2e4-dfb9-4570-b157-2e959dbdd937

spec:

containers:

- image: ikubernetes/demoapp:v1.1

imagePullPolicy: IfNotPresent

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

readOnly: true

dnsPolicy: ClusterFirst

enableServiceLinks: true

nodeName: k8s-node02.awen.com

preemptionPolicy: PreemptLowerPriority

priority: 0

restartPolicy: Always

schedulerName: default-scheduler

securityContext: {}

serviceAccount: default

serviceAccountName: default

terminationGracePeriodSeconds: 30

tolerations:

- effect: NoExecute

key: node.kubernetes.io/not-ready

operator: Exists

tolerationSeconds: 300

- effect: NoExecute

key: node.kubernetes.io/unreachable

operator: Exists

tolerationSeconds: 300

volumes:

- name: kube-api-access-d2ll8

projected:

defaultMode: 420

sources:

- serviceAccountToken:

expirationSeconds: 3607

path: token

- configMap:

items:

- key: ca.crt

path: ca.crt

- downwardAPI:

items:

- fieldRef:

apiVersion: v1

fieldPath: metadata.namespace

path: namespace

status:

conditions:

- lastProbeTime: null

lastTransitionTime: "2023-01-24T11:37:30Z"

status: "True"

type: Initialized

- lastProbeTime: null

lastTransitionTime: "2023-01-24T11:39:34Z"

status: "True"

type: Ready

- lastProbeTime: null

lastTransitionTime: "2023-01-24T11:39:34Z"

status: "True"

type: ContainersReady

- lastProbeTime: null

lastTransitionTime: "2023-01-24T11:37:30Z"

status: "True"

type: PodScheduled

containerStatuses:

- containerID: docker://543597f45306c51cf961cf6d709e1d3ed5bff31ae798bcd178b8c77bb5c3cc7e

image: ikubernetes/demoapp:v1.1

imageID: docker-pullable://ikubernetes/demoapp@sha256:3d7df5578a07f4e8d1c6b1e87204b359262b34ab07434635d5119df375afaac4

lastState: {}

ready: true

restartCount: 0

started: true

state:

running:

startedAt: "2023-01-24T11:39:33Z"

hostIP: 10.4.7.138

phase: Running

podIP: 10.244.6.7

podIPs:

- ip: 10.244.6.7

qosClass: BestEffort

startTime: "2023-01-24T11:37:30Z"

列出特定资源类型下的所有对象：

~# kubectl get TYPE [NAME] -o {json|yaml|wide}

root@K8S-master:~# kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

demoapp-55b98fb4d5-2qdd9 1/1 Running 0 9m27s 10.244.4.7 k8s-node01.awen.com <none> <none>

demoapp-55b98fb4d5-2rxsg 1/1 Running 0 9m27s 10.244.5.6 k8s-node03.awen.com <none> <none>

demoapp-55b98fb4d5-89qd8 1/1 Running 0 9m27s 10.244.6.8 k8s-node02.awen.com <none> <none>

root@K8S-master:~# curl 10.244.4.7

iKubernetes demoapp v1.1 !! ClientIP: 10.244.0.0, ServerName: demoapp-55b98fb4d5-2qdd9, ServerIP: 10.244.4.7!

创建Service资源对象：

root@K8S-master:~# kubectl create service nodeport demoapp --tcp=80:80 --dry-run=client -o yaml

apiVersion: v1

kind: Service

metadata:

creationTimestamp: null

labels:

app: demoapp

spec:

ports:

- name: 80-80

port: 80

protocol: TCP

targetPort: 80

selector:

app: demoapp

type: NodePort

status:

loadBalancer: {}

查看标签

root@K8S-master:~# kubectl get pods --show-labels

NAME READY STATUS RESTARTS AGE LABELS

demoapp-55b98fb4d5-2qdd9 1/1 Running 0 20m app=demoapp,pod-template-hash=55b98fb4d5

demoapp-55b98fb4d5-2rxsg 1/1 Running 0 20m app=demoapp,pod-template-hash=55b98fb4d5

demoapp-55b98fb4d5-89qd8 1/1 Running 0 20m app=demoapp,pod-template-hash=55b98fb4d5

创建Service资源对象：

root@K8S-master:~# kubectl create service nodeport demoapp --tcp=80:80
service/demoapp created

root@K8S-master:~# kubectl get service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

demoapp NodePort 10.106.9.91 <none> 80:31623/TCP 80s

kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10h

创建service 自动对应创建endpoints

root@K8S-master:~# kubectl get service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

demoapp NodePort 10.106.9.91 <none> 80:31623/TCP 5m33s

kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10h

root@K8S-master:~# kubectl get endpoints

NAME ENDPOINTS AGE

demoapp 10.244.4.7:80,10.244.5.6:80,10.244.6.8:80 5m36s

kubernetes 10.4.7.136:6443 10h

RESTful API Server上的所资源对象的操作：

增删改查

root@K8S-master:~# cat deployment-demoapp.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

creationTimestamp: null

labels:

app: nginx

spec:

replicas: 3

selector:

matchLabels:

app: nginx

strategy: {}

template:

metadata:

creationTimestamp: null

labels:

app: nginx

spec:

containers:

- image: nginx:alpine

root@K8S-master:~# kubectl create -f deployment-demoapp.yaml
deployment.apps/nginx created

root@K8S-master:~# kubectl get Deployment

NAME READY UP-TO-DATE AVAILABLE AGE

demoapp 3/3 3 3 97m

nginx 3/3 3 3 5m44s

root@K8S-master:~# cat service-demonapp.yaml

apiVersion: v1

kind: Service

metadata:

creationTimestamp: null

labels:

app: nginx

spec:

ports:

- name: 80-80

port: 80

protocol: TCP

targetPort: 80

selector:

app: nginx

type: NodePort

root@K8S-master:~# kubectl create -f service-demonapp.yaml
service/nginx created

root@K8S-master:~# kubectl get service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

demoapp NodePort 10.106.9.91 <none> 80:31623/TCP 82m

kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 11h

nginx NodePort 10.102.182.152 <none> 80:32588/TCP 18s

查看pod日志

root@K8S-master:~# kubectl logs nginx-6c557cc74d-t2728

工作负载控制器Deployment 由工作负载控制器创建pod, 创建service 作为访问入口

pod扩容

root@K8S-master:~# kubectl scale --help

root@K8S-master:~# kubectl scale deployment nginx --replicas=6

deployment.apps/nginx scaled

root@K8S-master:~# kubectl get pods

NAME READY STATUS RESTARTS AGE

demoapp-55b98fb4d5-2qdd9 1/1 Running 0 118m

demoapp-55b98fb4d5-2rxsg 1/1 Running 0 118m

demoapp-55b98fb4d5-89qd8 1/1 Running 0 118m

nginx-6c557cc74d-7xdx8 1/1 Running 0 26m

nginx-6c557cc74d-cl5tn 1/1 Running 0 11s

nginx-6c557cc74d-jbgp7 1/1 Running 0 11s

nginx-6c557cc74d-nmzpf 1/1 Running 0 26m

nginx-6c557cc74d-sgfjp 1/1 Running 0 11s

nginx-6c557cc74d-t2728 1/1 Running 0 23m

root@K8S-master:~# kubectl scale deployment nginx --replicas=3

deployment.apps/nginx scaled

root@K8S-master:~# kubectl get pods

NAME READY STATUS RESTARTS AGE

demoapp-55b98fb4d5-2qdd9 1/1 Running 0 119m

demoapp-55b98fb4d5-2rxsg 1/1 Running 0 119m

demoapp-55b98fb4d5-89qd8 1/1 Running 0 119m

nginx-6c557cc74d-7xdx8 1/1 Running 0 26m

nginx-6c557cc74d-nmzpf 1/1 Running 0 26m

nginx-6c557cc74d-t2728 1/1 Running 0 23m

posted on 2023-01-24 21:55 倔强不屈的阿文阅读(5) 评论(0) 收藏举报

刷新页面返回顶部

tshxawen