|NO.Z.00090|——————————|^^ 升级 ^^|——|KuberNetes&二进制升级.V01|——|kubernetes_master|etcd.v3.4.13——>v3.4.7|

一、二进制kubernetes升级说明

### --- kubernetes升级说明

~~~     # kubeadm方式：
~~~     kubernetes官方不建议跨大版本升级，建议升级小版本升级:1.17.0——>1.17.1

~~~     # 二进制的方式：
~~~     可以直接进行跨大版本升级，直接替换配置文件即可:1.17——>.1.19
~~~     kubernetes是节点，在1.16之前是可以使用bata，而1.16之后直接启用了，
~~~     升级的过程中要注意apiversion更改，防止升级完成之后不能增删改查。
~~~     升级的过程中，停掉完所有的节点再启动controller-manager
~~~     # 升级是从下往上升级：
~~~     etcd——>k8s-master组件：apiserver、controller-manager、scheduler——>kubelet：升级时涉及到容器的迁移——>END

### --- 查看kubernetes版本并确认升级的版本号
~~~     查看kubernetes版本号
~~~     将kubernetes版本升级为v1.17.0——>1.19.5版本

[root@k8s-master01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.17.0", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.17.0", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

### --- 查看kubernetes升级版本所对应的组件版本 

~~~     https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md

二、kubernetes-etcd升级说明

### --- 升级etcd流etcd升级流程

~~~     ——>备份etcd数据
~~~     ——>下载新版etcd包
~~~     ——>停止etcd
~~~     ——>替换etcd和etcdctl
~~~     ——>启动etcd

### --- 查看kubernetes1.19下etcd对应的版本号
~~~     # kubernetes升级到1.19版本对应可升级的etcd版本为3.4.7
~~~     注：kubernetes.V1.19.5版本对应的etcd版本号为3.4.7  

Update corefile-migration library to 1.0.8 (#91856, @wawa0210) [SIG Node]
Update default etcd server version to 3.4.4 (#89214, @jingyih) [SIG API Machinery, Cluster Lifecycle and Testing]
Update default etcd server version to 3.4.7 (#89895, @jingyih) [SIG API Machinery, Cluster Lifecycle and Testing]
Update default etcd server version to 3.4.9 (#92349, @jingyih) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]
Update go.etcd.io/bbolt to v1.3.5 (#92350, @justaugustus) [SIG API Machinery and Cloud Provider]
Update opencontainers/runtime-spec dependency to v1.0.2 (#89644, @saschagrunert) [SIG Node]
beta.kubernetes.io/os and beta.kubernetes.io/arch node labels are deprecated. Update node selectors to use kubernetes.io/os and kubernetes.io/arch. (#91046, @wawa0210) [SIG Apps and Node]
kubectl config view now redacts bearer tokens by default, similar to client certificates. The --raw flag can still be used to output full content. (#88985, @puerco)

### --- 下载etcd.v3.4.7版本包
~~~     # 下载etcd版本包

[root@k8s-master01 etcd]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.7/etcd-v3.4.7-linux-amd64.tar.gz

~~~     # 查看下载的版本包
[root@k8s-master01 etcd]# ll
-rw-r--r-- 1 root root 17310840 Apr  2  2020 etcd-v3.4.7-linux-amd64.tar.gz

~~~     # 解压版本包
[root@k8s-master01 etcd]# tar -zxvf etcd-v3.4.7-linux-amd64.tar.gz

### --- 查看etcd集群状态
~~~     # 使用etcdctl命令V2版本查看etcd集群信息

[root@k8s-master01 etcd]# export ETCDCTL_API=2
[root@k8s-master01 etcd]# etcdctl --ca-file /etc/kubernetes/pki/etcd/etcd-ca.pem --key-file /etc/kubernetes/pki/etcd/etcd-key.pem --cert-file /etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 member list
ac7e57d44f030e8: name=k8s-master02 peerURLs=https://192.168.1.12:2380 clientURLs=https://192.168.1.12:2379 isLeader=false
40ba37809e1a423f: name=k8s-master03 peerURLs=https://192.168.1.13:2380 clientURLs=https://192.168.1.13:2379 isLeader=true
ace8d5b0766b3d92: name=k8s-master01 peerURLs=https://192.168.1.11:2380 clientURLs=https://192.168.1.11:2379 isLeader=false

~~~     # 使用etcdctl命令V3版本查看etcd集群信息
~~~     注：可以确定k8s-master03为主节点
~~~     注：可以先升级k8s-master02和k8s-master03节点的etcd版本

[root@k8s-master01 etcd]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 member list
ac7e57d44f030e8, started, k8s-master02, https://192.168.1.12:2380, https://192.168.1.12:2379
40ba37809e1a423f, started, k8s-master03, https://192.168.1.13:2380, https://192.168.1.13:2379
ace8d5b0766b3d92, started, k8s-master01, https://192.168.1.11:2380, https://192.168.1.11:2379

### --- 备份etcd集群存储数据 
~~~     # 备份所有节点etcd集群存储数据

[root@k8s-master01 etcd]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 snapshot -h
COMMANDS:
    restore Restores an etcd member snapshot to an etcd directory       // 回滚
    save    Stores an etcd node backend snapshot to a given file        // 保存
    status  Gets backend snapshot status of a given file                // 状态
[root@k8s-master01 etcd]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379 snapshot save 20210701-0701-11.db
Snapshot saved at 20210701-0701-11.db
[root@k8s-master01 etcd]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.12:2379 snapshot save 20210701-0701-12.db
Snapshot saved at 20210701-0701-12.db
[root@k8s-master01 etcd]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.13:2379 snapshot save 20210701-0701-13.db
Snapshot saved at 20210701-0701-13.db

三、k8s-master01节点kubernetes-etcd升级etcd.v3.4.13——>3.4.7

### --- 备份版本包
~~~     # 查看etcd部署在什么位置

[root@k8s-master01 ~]# which etcd
/usr/local/bin/etcd

~~~     # 备份版本包

[root@k8s-master01 ~]# mkdir /usr/local/bin/bak/
[root@k8s-master01 ~]# cp /usr/local/bin/* /usr/local/bin/bak/

### --- 停止当前节点etcd服务
~~~     # 停止当前节点etcd服务

[root@k8s-master01 ~]# systemctl stop etcd

~~~     # 查看etcd集群状态
~~~     注：11状态不健康，已经被停止运行状态
 
[root@k8s-master01 ~]# etcdctl --endpoints="192.168.1.13:2379,192.168.1.12:2379,192.168.1.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.13:2379 | 40ba37809e1a423f |  3.4.13 |  9.5 MB |      true |      false |       248 |     223109 |             223109 |        |
| 192.168.1.12:2379 |  ac7e57d44f030e8 |  3.4.13 |  9.5 MB |     false |      false |       248 |     223109 |             223109 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
~~~     # OR
[root@k8s-master01 ~]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 endpoint health
https://192.168.1.13:2379 is healthy: successfully committed proposal: took = 37.560337ms
https://192.168.1.12:2379 is healthy: successfully committed proposal: took = 43.665877ms
https://192.168.1.11:2379 is unhealthy: failed to commit proposal: context deadline exceeded

~~~     # 升级etcd版本
~~~     切换到etcd版本包目录下

[root@k8s-master01 ~]# cd /root/upgrade/etcd-v3.4.7-linux-amd64

~~~     拷贝etcd  etcdctl文件到/usr/local/bin目录下
[root@k8s-master01 etcd-v3.4.7-linux-amd64]# cp etcd* /usr/local/bin/
cp: overwrite ‘/usr/local/bin/etcd’? y
cp: overwrite ‘/usr/local/bin/etcdctl’? y

~~~     查看配置文件是否为切片形式
[root@k8s-master01 ~]# vim /etc/etcd/etcd.config.yml 
log-outputs: ["default"]

~~~     # 启动etcd服务

[root@k8s-master01 ~]# systemctl restart etcd

~~~     # 查看更新后etcd版本

[root@k8s-master01 ~]# etcdctl version
etcdctl version: 3.4.7
API version: 3.4

### --- 查看集群状态
~~~     # 查看etcd集群状态

[root@k8s-master01 ~]# etcdctl --endpoints="192.168.1.13:2379,192.168.1.12:2379,192.168.1.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.13:2379 | 40ba37809e1a423f |  3.4.13 |  9.5 MB |      true |      false |       248 |     226533 |             226533 |        |
| 192.168.1.12:2379 |  ac7e57d44f030e8 |  3.4.13 |  9.5 MB |     false |      false |       248 |     226533 |             226533 |        |
| 192.168.1.11:2379 | ace8d5b0766b3d92 |   3.4.7 |  9.8 MB |     false |      false |       248 |     226533 |             226533 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
~~~     # OR   
[root@k8s-master01 ~]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 endpoint health
https://192.168.1.12:2379 is healthy: successfully committed proposal: took = 31.109889ms
https://192.168.1.13:2379 is healthy: successfully committed proposal: took = 32.625376ms
https://192.168.1.11:2379 is healthy: successfully committed proposal: took = 32.204066ms

四、k8s-master02节点kubernetes-etcd升级

### --- 备份版本包
~~~     # 查看etcd部署在什么位置

[root@k8s-master02 ~]# which etcd
/usr/local/bin/etcd

~~~     # 备份版本包

[root@k8s-master02 ~]# mkdir /usr/local/bin/bak/
[root@k8s-master02 ~]# cp /usr/local/bin/* /usr/local/bin/bak/

### --- 停止当前节点etcd服务

[root@k8s-master02 ~]# systemctl stop etcd

~~~     # 查看etcd集群状态
~~~     注：12状态不健康，已经被停止运行状态

[root@k8s-master01 ~]# etcdctl --endpoints="192.168.1.13:2379,192.168.1.12:2379,192.168.1.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.13:2379 | 40ba37809e1a423f |  3.4.13 |  9.5 MB |      true |      false |       248 |     223109 |             223109 |        |
| 192.168.1.11:2379 |  ac7e57d44f030e8 |  3.4.17 |  9.5 MB |     false |      false |       248 |     223109 |             223109 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
~~~     # OR
[root@k8s-master01 ~]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 endpoint health
https://192.168.1.13:2379 is healthy: successfully committed proposal: took = 22.136296ms
https://192.168.1.11:2379 is healthy: successfully committed proposal: took = 26.003475ms
https://192.168.1.12:2379 is unhealthy: failed to commit proposal: context deadline exceeded

### --- 升级etcd版本

~~~     # 查看当前节点etcd集群版本
[root@k8s-master02 ~]# etcdctl version
etcdctl version: 3.4.13
API version: 3.4

~~~     # 升级etcd版本
~~~     切换到版本包目录

[root@k8s-master01 ~]# cd /root/upgrade/etcd-v3.4.7-linux-amd64

~~~     拷贝etcd  etcdctl文件到/usr/local/bin目录下
[root@k8s-master01 etcd-v3.4.7-linux-amd64]# scp etcd* k8s-master03:/usr/local/bin/
etcd                                                                                                                                                          100%   23MB  21.9MB/s   00:01    
etcdctl   

~~~     查看配置文件是否为切片形式
[root@k8s-master02 ~]# vim /etc/etcd/etcd.config.yml 
log-outputs: ["default"]

~~~     # 启动etcd服务

[root@k8s-master02 ~]# systemctl restart etcd

~~~     # 查看更新后etcd版本

[root@k8s-master02 ~]# etcdctl version
etcdctl version: 3.4.7
API version: 3.4

### --- 查看集群状态
~~~     # 查看etcd集群状态

[root@k8s-master01 ~]# etcdctl --endpoints="192.168.1.13:2379,192.168.1.12:2379,192.168.1.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.13:2379 | 40ba37809e1a423f |  3.4.13 |  9.5 MB |      true |      false |       248 |     227928 |             227928 |        |
| 192.168.1.12:2379 |  ac7e57d44f030e8 |   3.4.7 |  9.5 MB |     false |      false |       248 |     227928 |             227928 |        |
| 192.168.1.11:2379 | ace8d5b0766b3d92 |   3.4.7 |  9.8 MB |     false |      false |       248 |     227928 |             227928 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
~~~     # OR   
[root@k8s-master01 ~]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 endpoint health
https://192.168.1.11:2379 is healthy: successfully committed proposal: took = 28.622819ms
https://192.168.1.13:2379 is healthy: successfully committed proposal: took = 57.660454ms
https://192.168.1.12:2379 is healthy: successfully committed proposal: took = 65.698448ms

五、k8s-master03节点kubernetes-etcd升级

### --- 备份版本包
~~~     # 查看etcd部署在什么位置

[root@k8s-master03 ~]# which etcd
/usr/local/bin/etcd

~~~     # 备份版本包

[root@k8s-master03 ~]# mkdir /usr/local/bin/bak/
[root@k8s-master03 ~]# cp /usr/local/bin/* /usr/local/bin/bak/

### --- 停止当前节点etcd服务
~~~     # 停止当前节点etcd服务
~~~     升级主节点：主节点被stop后会重新选主，重新选择出来一个新的主节点

[root@k8s-master03 ~]# systemctl stop etcd

~~~     # 查看etcd集群状态
~~~     注：13状态不健康，已经被停止运行状态

[root@k8s-master01 ~]# etcdctl --endpoints="192.168.1.13:2379,192.168.1.12:2379,192.168.1.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.12:2379 |  ac7e57d44f030e8 |   3.4.7 |  9.5 MB |     false |      false |       249 |     228492 |             228492 |        |
| 192.168.1.11:2379 | ace8d5b0766b3d92 |   3.4.7 |  9.8 MB |      true |      false |       249 |     228492 |             228492 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
~~~     # OR
[root@k8s-master01 ~]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 endpoint health
https://192.168.1.11:2379 is healthy: successfully committed proposal: took = 39.586651ms
https://192.168.1.12:2379 is healthy: successfully committed proposal: took = 41.327247ms
https://192.168.1.13:2379 is unhealthy: failed to commit proposal: context deadline exceeded

### --- 升级etcd版本
~~~     # 查看当前节点etcd集群版本

[root@k8s-master03 ~]# etcdctl version
etcdctl version: 3.4.13
API version: 3.4

~~~     # 升级etcd版本 
~~~     切换到版本包目录

[root@k8s-master01 ~]# cd /root/upgrade/etcd-v3.4.7-linux-amd64

~~~     拷贝etcd  etcdctl文件到/usr/local/bin目录下
[root@k8s-master01 etcd-v3.4.7-linux-amd64]# scp etcd* k8s-master02:/usr/local/bin/
etcd                                                                                                                                                          100%   23MB  21.9MB/s   00:01    
etcdctl   

~~~     查看配置文件是否为切片形式
[root@k8s-master03 ~]# vim /etc/etcd/etcd.config.yml 
log-outputs: ["default"]

~~~     # 启动etcd服务

[root@k8s-master03 ~]# systemctl restart etcd

~~~     # 查看更新后etcd版本

[root@k8s-master03 ~]# etcdctl version
etcdctl version: 3.4.7
API version: 3.4

### --- 查看集群状态
~~~     # 查看etcd集群状态

[root@k8s-master01 ~]# etcdctl --endpoints="192.168.1.13:2379,192.168.1.12:2379,192.168.1.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.13:2379 | 40ba37809e1a423f |   3.4.7 |  9.5 MB |     false |      false |       249 |     229007 |             229007 |        |
| 192.168.1.12:2379 |  ac7e57d44f030e8 |   3.4.7 |  9.5 MB |     false |      false |       249 |     229007 |             229007 |        |
| 192.168.1.11:2379 | ace8d5b0766b3d92 |   3.4.7 |  9.8 MB |      true |      false |       249 |     229007 |             229007 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
~~~     # OR   
[root@k8s-master01 ~]# etcdctl --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --endpoints https://192.168.1.11:2379,https://192.168.1.12:2379,https://192.168.1.13:2379 endpoint health
https://192.168.1.11:2379 is healthy: successfully committed proposal: took = 33.137392ms
https://192.168.1.13:2379 is healthy: successfully committed proposal: took = 34.387544ms
https://192.168.1.12:2379 is healthy: successfully committed proposal: took = 34.056139ms

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

——W.S.Landor

posted on 2022-03-29 13:51 yanqi_vip 阅读(107) 评论(0) 收藏举报

刷新页面返回顶部

|NO.Z.00090|——————————|^^ 升级 ^^|——|KuberNetes&二进制升级.V01|——|kubernetes_master|etcd.v3.4.13——>v3.4.7|

导航

公告