K8S集群CNI网络插件之Calico底层原理

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

目录

一.Calico及calicoctl部署

1.Calico介绍

Calico是一个开源的三层虚拟化网络方案，用于为云原生应用实现互联及策略控制，相较于Flannel来说，Calico的优势是对网络策略(Network policy)，它允许用户动态定义ACL规则进出容器的数据报文，实现为Pod间的通信按需施加安全策略。不仅如此，Calico还可以整合进大多数具备编排能力的环境， 可以为虚机和容器提供多主机间通信的功能。

Calico本身是一个三层的虚拟网络方案，利用Linux内核实现了一个高效的虚拟路由器(vRouter)进行报文转发，各节点路由器通过BGP(Border Gateway Protocol)协议负责把自身所属节点上运行的Pod资源的IP地址信息及基于节点的agent程序(Felix)直接由vRouter生成路由规则向整个Calico网络内传播，从而将不同节点上的容器链接起来。

综上所述，Calico方案其实是一个纯三层的解决方案，通过每个节点协议栈的三层(网络层)确保容器之间的连通性，这摆脱了Flannel host-gw类型的所有节点必须位于同一二层网络的限制， 从而极大地扩展了网络规模和网络边界。

官网地址:
	https://www.tigera.io/project-calico/

2.Calico的工作机制

Calico把Kubernetes集群环境中的每个节点上的Pod所组成的网络视为一个自治系统，各节点也就是自治系统的边界网关，它们彼此间通过BGP协议交换路由信息生成路由规则。

考虑到并非所有网络都支持BGP，以及BGP路由模型要求所有节点必须要位于同一个二层网络，Calico还支持基于IPIP和VXLAN的叠加网络模型。

类似于Flannel在VXLAN后端启用DirectRouting的网络模型，Calico也支持混合使用路由和叠加网络模型，BGP路由模型用于二层网络的高性能通信，IPIP或VXLAN用于跨子网的节点间(Cross-Subnet)报文转发。

Calico不使用隧道或者NAT来实现转发，而是基于BGP来实现各节点的数据转发:
    underlay network:
        BGP，三层虚拟网络解决方案，大规模场景推荐使用这种模式。

    overlay network:
        - IPIP，双层IP实现跨网段效果。【默认模式】
        - VXLAN，数据包标识实现大二层上的跨网段通信。

3.Calico程序组件及架构

参考链接:
	https://docs.tigera.io/calico/latest/reference/architecture/overview
	
		
	
Clico程序组件说明:
	Calico API服务器:
		主要任务：允许您直接使用kubectl管理Calico资源。
		
	Felix: (Calico Agent)
		主要任务：作为守护进程运行在每个节点，主要负责维护虚拟接口设备，路由信息，ACL，以及主机上为该主机上的端点提供所需连接所需的任何其他内容。
		
	BIRD
		主要任务：在承载Felix代理的每个节点上运行,从Felix获取路由，并分发到网络上的BGP对等端进行主机间路由。
		
	confd
		主要任务：监控Calico数据存储中BGP配置和全局默认值（如as号、日志级别和IPAM信息）的更改。开源、轻量级的配置管理工具。

	Dikastes(Optional) :
		主要任务：执行Istio服务网格的网络策略。作为Istio Envoy的sidecar代理在集群上运行。
		
	CNI plugin
		主要任务：为Kubernetes集群提供Calico网络。
		
	Datastore plugin
		主要任务：它是Calico CNI插件之一，负责存储Calico配置，路由，策略及其他信息，通常它们表现为Calico CRD资源对象。
		支持的Calico CRD有很多，此处我们找几个常用的CRD功能举例说明:
			- BGPConfiguration:
				全局BGP配置，用于设定AS(自治系统)编号，node mesh，以及用于通告ClusterIP的设置。
			- FelixConfiguration:
				Felix相关的低级别配置，包括iptables，MTU和路由协议等。
			- GlobalNetworkPolicy:
				全局网络策略，生效于整个集群级别。
			- GlobalNetworkSet:
				全局网络集，是指可由GlobalNetworkPolicy引用的外部网络IP列表或CIDR列表。
			- IPPool:
				IP地址池及相关选项，包括要使用的路由协议(IPIP,VXLAN或Native)，一个集群支持使用多个Pool。

	IPAM plugin
		主要任务：使用Calico的IP池资源来控制如何将IP地址分配给集群内的Pod。它是大多数Calico安装使用的默认插件。它是Calico CNI插件之一。

	kube-controllers
		主要任务：监控Kubernetes API，并根据集群状态执行操作。kube控制器。

	Typha
		主要任务：通过减少每个节点对数据存储的影响来增加规模。作为数据存储和Felix实例之间的守护进程运行。默认情况下已安装，但未配置。
		各Calico node示例同Calico Datastore通信的中间层。
		由其负责将Calico Datastore中生成的更改信息分发给各Calico node，以减轻50个节点以上规模集群的Calico Datastore的负载。

	calicoctl
		主要任务：命令行界面，用于创建、读取、更新和删除Calico对象。
		calicontl命令行可以在任何可以通过网络访问Calico数据存储的主机上以二进制或容器的形式使用。需要单独安装。
		
	Plugins for cloud orchestrators
		主要任务：将用于管理网络的编排器API转换为Calico数据模型和数据存储。
		
	BGP Route Reflector(Optional) 
		主要任务: BGP路由反射器，可选组件，用于较大规模的网络场景。
		假设集群有N个节点，每个节点默认要维护N*(N-1)个路由信息，随着节点数量的增多，每个节点维护的路由信息也会增加。
		而使用BGP路由反射器相当于配置了一个路由中心，每个节点将路由信息通报给该节点即可，本地没有路由记录时来这里查询即可。
		
	
温馨提示:
	对于Calico在K8S集群上的部署来说，会涉及到两个部署组件:
		- calico-node:
			需要部署到所有集群节点上的代理守护进程，提供封装好的Felix和BIRD。
			
		- calico-kube-controller:
			专用于K8S上对Calico所有节点给管理的中央控制器。
			负责Calico与K8S集群的协同及Calico核心功能实现。

4.Calico数据存储模式

尽管使用独立的etcd可以减轻kube-apiserver的压力。但通常情况下，都建议使用Kubernetes数据存储而非使用独立的etcd存储。

基于Kubernetes存储数据:
	- 1.数据通过CRD存储于kube-apiserver组件;
	- 2.对Calico资源的访问可由Kubernetes RBAC控制;
	- 3.但成百上千个Felix实例同时与kube-apiserver交互，会带来负载影响，因而必须要使用typha中间层;
	
基于etcd存储数据:
	- 1.可减轻kube-apiserver的压力，但也会引入不必要的复杂性和安全性;
	- 2.不可直接使用kubernetes etcd作为calico的数据存储;

5.Calico的Pod网络接口

calico网络插件如何为Pod配置网络接口:
	- 1.为每个Pod创建一组veth pair，一端注入Pod网络名称空间，另一端驻留在worker宿主机上;
		- Pod内的一端，通常名格式为"eth0@ifN"，其中N是驻留在宿主机上的另一端的IP link编号;
		- 驻留宿主机的一端，名称格式为"caliXXXXXXXXXXX@ifN"，其中11为X是经由函数计算生成，而N则是注入到Pod网络名称空间中的对端IP link编号;
		
	- 2.Pod网络名称空间中，会生成独特的默认路由，将网关指向"169.254.1.1"，表示宿主机本身;
[root@master231 ~]# kubectl get pods -o wide
NAME         READY   STATUS    RESTARTS        AGE   IP               NODE        NOMINATED NODE   READINESS GATES
xiuxian-v1   1/1     Running   1 (3h50m ago)   13d   10.100.203.151   worker232   <none>           <none>
xiuxian-v2   1/1     Running   1 (3h49m ago)   13d   10.100.140.86    worker233   <none>           <none>
[root@master231 ~]# 
[root@master231 ~]# kubectl exec -it xiuxian-v1  -- route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         169.254.1.1     0.0.0.0         UG    0      0        0 eth0
169.254.1.1     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
[root@master231 ~]# 

	-3.宿主机为每个"caliXXXXXXXXXXX"接口都开启了ARP Proxy功能，从而让宿主机扮演网关设备，并以自己的MAC地址代为应答对端Pod中发来的所有ARP请求。
[root@master231 ~]# ip a
...
9: cali2e51c138fab@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: calic929f404ac0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: cali92282748908@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
12: cali253a24d7fb3@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3
[root@master231 ~]# 
[root@master231 ~]# 
[root@master231 ~]# ll -d  /proc/sys/net/ipv4/conf/cali*
dr-xr-xr-x 1 root root 0 Mar  7 16:09 /proc/sys/net/ipv4/conf/cali253a24d7fb3/
dr-xr-xr-x 1 root root 0 Mar  7 16:08 /proc/sys/net/ipv4/conf/cali2e51c138fab/
dr-xr-xr-x 1 root root 0 Mar  7 16:09 /proc/sys/net/ipv4/conf/cali92282748908/
dr-xr-xr-x 1 root root 0 Mar  7 16:08 /proc/sys/net/ipv4/conf/calic929f404ac0/
[root@master231 ~]# 
[root@master231 ~]# cat  /proc/sys/net/ipv4/conf/cali*/proxy_arp  # "1"表示开启了arp代理功能
1
1
1
1
[root@master231 ~]# 



同一个节点上的Pod通信依赖于为每个Pod单独配置的路由规则:
[root@master231 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
...
10.100.160.128  0.0.0.0         255.255.255.192 U     0      0        0 *
10.100.160.153  0.0.0.0         255.255.255.255 UH    0      0        0 cali2e51c138fab
10.100.160.154  0.0.0.0         255.255.255.255 UH    0      0        0 calic929f404ac0
10.100.160.155  0.0.0.0         255.255.255.255 UH    0      0        0 cali92282748908
10.100.160.156  0.0.0.0         255.255.255.255 UH    0      0        0 cali253a24d7fb3
[root@master231 ~]# 


不同节点上的Pod间通信，则由Calico的路由模式决定。

6.Calico支持的路由模式

Calico支持多种路由模式:
	- Native Routing:
		原生路由，无隧道封装，类似于Flannel的host-gw模式。
		此模式BIRD组件默认使用TCP的179端口。
		
	- IP-in-IP：
		IPIP隧道(基于"tunl0"设备)模式，开销小的隧道协议，适用性好，尤其是跨多子网的网络环境。
		存在额外开销，MTO一般要设置为1480，IPIP模式同样需要使用BGP协议，以完成路由分发。
		
	- VXLAN:
		VXLAN隧道(基于"vxlan.calico"设备)模式，完全不需要依赖于BGP，开销较之IPIP略大(MTU需要设置为1450)，但功能也更加强大。
		默认使用UDP的4789端口。

难以使用Native Routing模式的常见场景:
	- 1.集群节点跨越多个子网或路由器，这些路由器依赖目标IP来确定目标主机;
	- 2.对于入站数据包强制执行源地址和目标地址核验;
	- 3.阻止所有BGP报文的网络环境;


难以使用IP-in-IP模式的常见场景:
	- 1.禁止使用IPIP协议的环境，例如"Azure";
	- 2.阻止所有BGP报文的网络环境;

二.部署Calico

1.部署Calico插件方式

部署Calico的典型方式有Operator和Manifest。
	- Operator:
		由专门的Operator和CRD管理。
	
	- Manifest:
		基于配置清单进行部署，常见的三种选择方式如下:
            - 1.将Kubernetes API作为存储，节点数小于等于50;
            - 2.将Kubernetes API作为存储，节点数大于50，重点在于启用Typha组件;
            - 3.抓浓郁etcd存储的方式，不推荐;
            
参考链接:
	https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico


	
推荐阅读:
	https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/config-options
	https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart

2.部署Calico

部署资源清单参考链接:
	https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-kubernetes-api-datastore-50-nodes-or-less
	https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/calico.yaml
	https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/calico-typha.yaml


部署前要关注的几个配置(在资源清单搜索以下的"关键字"即可)
    CALICO_IPV4POOL_CIDR
        选用的Pod CIDR的网段地址，默认为"192.168.0.0/16"。
        
    CALICO_IPV4POOL_BLOCK_SIZE
    	指定子网掩码的长度，比如"24"，将来就会为每个节点分配一个24位的子网，如果不指定，则默认值为"26"。
    	默认是没有此配置项目的，需要手动添加，其和CALICO_IPV4POOL_CIDR在同一个环境变量级别。

    CALICO_IPV4POOL_IPIP|CALICO_IPV4POOL_VXLAN|CALICO_IPV6POOL_VXLAN
        选用的路由模式，有效值为: Always(总是使用),Never(始终不使用),Cross-Subnet(跨子网时才使用)。

    __CNI_MTU__
        默认由kube-system名称空间的"ConfigMap/calico-config"提供配置。

    autodetect
        BGP要使用的IP地址，如果物理服务器有多快网卡的话可以自行指定，"autodetect"表示会自动探测物理网卡地址。
        如果自动探测接口不对，则可能会导致BGP路由出现问题，因此我们可以考虑使用手动指定。

3.部署calicoctl工具

3.1 calicoctl概述

Calico本身是一个复杂的系统，复杂到它自己提供一个非常重要的Restful接口，结合calicoctl命令来管理自身的相关属性信息。

calicoctl可以直接与etcd进行操作，也可以通过kube-apiserver的方式与etcd来进行操作。

默认情况下，它与kube-apiserver通信的方式认证与kubectl的命令使用同一个context。但是我们还是推荐，使用手工定制的一个配置文件。

calicoctl是运行在集群之外的，用于管理集群功能的一个重要的组件。Calicoctl的安装方式很多，常见的方式有：单主机方式，kubectl命令插件方式，主机容器方式。

3.2 部署calicoctl最新版本工具【不推荐】

参考链接:
	https://docs.tigera.io/calico/latest/getting-started/kubernetes/hardway/the-calico-datastore#calicoctl
	https://docs.tigera.io/calico/latest/operations/calicoctl/install
	
	
实操案例:
	1.下载软件包
[root@master231 ~]# wget -O calicoctl https://github.com/projectcalico/calico/releases/latest/download/calicoctl-linux-amd64
	
	2.移动到PATH路径
[root@master231 ~]# mv calicoctl /usr/local/bin/
	
	3.添加执行权限
[root@master231 ~]# chmod +x /usr/local/bin/calicoctl 
[root@master231 ~]# 
[root@master231 ~]# ll /usr/local/bin/calicoctl 
-rwxr-xr-x 1 root root 70297012 Feb  6 03:00 /usr/local/bin/calicoctl*
[root@master231 ~]# 




温馨提示:
	如果calicoctl和calico版本不对应，则可能会出现如下的报错信息;
[root@master231 ~]# calicoctl get nodes
Failed to get resources: Version mismatch.
Client Version:   v3.29.2
Cluster Version:  3.25.2
Use --allow-version-mismatch to override.

[root@master231 ~]# 
[root@master231 ~]# calicoctl get nodes --allow-version-mismatch   # 尽管添加该选项可以出结果，但建议还是下载匹配的版本哟~
NAME        
master231   
worker232   
worker233   

[root@master231 ~]#

3.3部署calicoctl要和calico版本对应

参考链接:
	https://archive-os-3-25.netlify.app/calico/3.25/operations/calicoctl/install
	
	
实战案例:
	1.移除旧版本
[root@master231 ~]# rm -f /usr/local/bin/calicoctl 

	2.下载Calico软件包
[root@master231 ~]# curl -L https://github.com/projectcalico/calico/releases/download/v3.25.2/calicoctl-linux-amd64 -o calicoctl

	3.移动Calicoctl到PATH变量
[root@master231 ~]# mv calicoctl /usr/local/bin/
[root@master231 ~]# 

	4.添加执行权限
[root@master231 ~]# chmod +x /usr/local/bin/calicoctl 
[root@master231 ~]# 
[root@master231 ~]# ll /usr/local/bin/calicoctl
-rwxr-xr-x 1 root root 63805704 Feb 28 10:15 /usr/local/bin/calicoctl*
[root@master231 ~]# 

	5.查看节点信息
[root@master231 ~]# calicoctl get nodes
NAME        
master231   
worker232   
worker233   

[root@master231 ~]# 
[root@master231 ~]# calicoctl get nodes -o wide
NAME        ASN       IPV4            IPV6   
master231   (64512)   10.0.0.231/24          
worker232   (64512)   10.0.0.232/24          
worker233   (64512)   10.0.0.233/24          

[root@master231 ~]# 


	6.查看calicoctl的版本信息和集群信息是否一致
[root@master231 ~]# calicoctl version
Client Version:    v3.25.2
Git commit:        978a0e4bc
Cluster Version:   v3.25.2
Cluster Type:      typha,kdd,k8s,operator,bgp,kubeadm
[root@master231 ~]#

4.calicoctl工具基本使用

4.1 查看节点信息

	1.查看节点状态
[root@master231 ~]# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.232   | node-to-node mesh | up    | 01:11:13 | Established |
| 10.0.0.233   | node-to-node mesh | up    | 01:11:14 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

[root@master231 ~]# 


	2.检查系统信息
[root@master231 ~]# calicoctl node checksystem
Checking kernel version...
		5.15.0-119-generic  					OK
Checking kernel modules...
		xt_bpf              					OK
		xt_rpfilter         					OK
		xt_addrtype         					OK
		xt_multiport        					OK
		ip_tables           					OK
		ipt_rpfilter        					OK
		ipt_set             					OK
		xt_icmp             					OK
		xt_icmp6            					OK
		xt_u32              					OK
		ipt_ipvs            					OK
		ip6_tables          					OK
		ipt_REJECT          					OK
		nf_conntrack_netlink					OK
		xt_mark             					OK
		xt_set              					OK
		ip_set              					OK
		xt_conntrack        					OK
		vfio-pci            					OK
System meets minimum system requirements to run Calico!
[root@master231 ~]# 



	3.查看指定节点的详细信息
[root@master231 ~]# calicoctl get node -o wide
NAME        ASN       IPV4            IPV6   
master231   (64512)   10.0.0.231/24          
worker232   (64512)   10.0.0.232/24          
worker233   (64512)   10.0.0.233/24          

[root@master231 ~]# 
[root@master231 ~]# calicoctl get node worker233 -o yaml
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  annotations:
    projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"worker233","kubernetes.io/os":"linux"}'
  creationTimestamp: "2024-12-15T07:50:20Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: worker233
    kubernetes.io/os: linux
  name: worker233
  resourceVersion: "229064"
  uid: 6fbf740a-88ce-4575-98cf-1fc4255163b5
spec:
  addresses:
  - address: 10.0.0.233/24
    type: CalicoNodeIP
  - address: 10.0.0.233
    type: InternalIP
  bgp:
    ipv4Address: 10.0.0.233/24
  ipv4VXLANTunnelAddr: 10.100.140.64
  orchRefs:
  - nodeName: worker233
    orchestrator: k8s
status:
  podCIDRs:
  - 10.100.2.0/24
[root@master231 ~]#

4.3 查看Pod的IP地址信息

	1.查看Pod网段配置信息
[root@master231 ~]# calicoctl ipam show
+----------+---------------+-----------+------------+--------------+
| GROUPING |     CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+---------------+-----------+------------+--------------+
| IP Pool  | 10.100.0.0/16 |     65536 | 12 (0%)    | 65524 (100%) |
+----------+---------------+-----------+------------+--------------+
[root@master231 ~]# 


	2.查看IP地址是否被分配
[root@master231 ~]# kubectl get pods -o wide -n calico-apiserver 
NAME                                READY   STATUS    RESTARTS      AGE   IP               NODE        NOMINATED NODE   READINESS GATES
calico-apiserver-6b6d64dc57-p9rfm   1/1     Running   1 (74m ago)   71d   10.100.140.73    worker233   <none>           <none>
calico-apiserver-6b6d64dc57-qpt62   1/1     Running   1 (70d ago)   71d   10.100.203.133   worker232   <none>           <none>
[root@master231 ~]# 
[root@master231 ~]# calicoctl ipam show --ip=10.100.140.73
IP 10.100.140.73 is in use  # 此地址已经完成分配
Attributes:
  namespace: calico-apiserver
  node: worker233
  pod: calico-apiserver-6b6d64dc57-p9rfm
  timestamp: 2025-02-28 01:11:12.269848232 +0000 UTC
[root@master231 ~]# 
[root@master231 ~]# 
[root@master231 ~]# calicoctl ipam show --ip=10.100.140.77
10.100.140.77 is not assigned  # 该地址还未被分配
[root@master231 ~]#

4.3 calicoctl结合kubectl搭配使用

	1.将calicoctl更名为"kubectl-calico"
[root@master231 ~]# cp /usr/local/bin/calicoctl /usr/local/bin/kubectl-calico

	
	2.验证kubectl可使用命令
[root@master231 ~]# kubectl calico --help
Usage:
  kubectl-calico [options] <command> [<args>...]

    create       Create a resource by file, directory or stdin.
    replace      Replace a resource by file, directory or stdin.
    apply        Apply a resource by file, directory or stdin.  This creates a resource
                 if it does not exist, and replaces a resource if it does exists.
    patch        Patch a pre-exisiting resource in place.
    delete       Delete a resource identified by file, directory, stdin or resource type and
                 name.
    get          Get a resource identified by file, directory, stdin or resource type and
                 name.
    label        Add or update labels of resources.
    convert      Convert config files between different API versions.
    ipam         IP address management.
    node         Calico node management.
    version      Display the version of this binary.
    datastore    Calico datastore management.

Options:
  -h --help                    Show this screen.
  -l --log-level=<level>       Set the log level (one of panic, fatal, error,
                               warn, info, debug) [default: panic]
     --context=<context>       The name of the kubeconfig context to use.
     --allow-version-mismatch  Allow client and cluster versions mismatch.

Description:
  The calico kubectl plugin is used to manage Calico network and security
  policy, to view and manage endpoint configuration, and to manage a Calico
  node instance.

  See 'kubectl-calico <command> --help' to read about a specific subcommand.
[root@master231 ~]# 

	3.测试案例
[root@master231 ~]# kubectl calico node status
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.232   | node-to-node mesh | up    | 01:11:14 | Established |
| 10.0.0.233   | node-to-node mesh | up    | 01:11:15 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

[root@master231 ~]# 
[root@master231 ~]# kubectl calico ipam show
+----------+---------------+-----------+------------+--------------+
| GROUPING |     CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+---------------+-----------+------------+--------------+
| IP Pool  | 10.100.0.0/16 |     65536 | 12 (0%)    | 65524 (100%) |
+----------+---------------+-----------+------------+--------------+
[root@master231 ~]#

5.Calico网络验证

5.1 查看Calico相关配置

[root@master231 ~]# calicoctl get ippool -o yaml
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    creationTimestamp: "2024-12-15T08:08:07Z"
    name: default-ipv4-ippool
    resourceVersion: "2645"
    uid: 3ee88782-3764-4898-b5cf-f2fe09128e1b
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26  # 子网掩码
    cidr: 10.100.0.0/16  # Pod网段地址
    ipipMode: Never   # 此处并没有使用IPIP模式哟~
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: CrossSubnet  # 如果物理机跨网段则使用vxlan模式，若不使用，则直接使用宿主机网络。
kind: IPPoolList
metadata:
  resourceVersion: "229614"
[root@master231 ~]#

5.2 新增Pod宿主机会自动生成路由验证

	1.查看部署的Pod信息
[root@master231 ~]# kubectl get pods -o wide
NAME         READY   STATUS    RESTARTS     AGE   IP               NODE        NOMINATED NODE   READINESS GATES
xiuxian-v1   1/1     Running   1 (8h ago)   13d   10.100.203.151   worker232   <none>           <none>
xiuxian-v2   1/1     Running   1 (8h ago)   13d   10.100.140.86    worker233   <none>           <none>
[root@master231 ~]# 
[root@master231 ~]# kubectl exec xiuxian-v1 -- ip a
...
4: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether ee:2c:d7:ca:96:18 brd ff:ff:ff:ff:ff:ff
    inet 10.100.203.151/32 scope global eth0
       valid_lft forever preferred_lft forever
[root@master231 ~]# 
[root@master231 ~]# 
[root@master231 ~]# kubectl exec xiuxian-v2 -- ip a
...
4: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether b2:65:ec:10:46:9f brd ff:ff:ff:ff:ff:ff
    inet 10.100.140.86/32 scope global eth0
       valid_lft forever preferred_lft forever
[root@master231 ~]# 



	2.查看worker232节点
[root@worker232 ~]# ip a
...
13: caliaec18759349@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 4
[root@worker232 ~]# 
[root@worker232 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.254      0.0.0.0         UG    0      0        0 eth0
...
10.100.203.151  0.0.0.0         255.255.255.255 UH    0      0        0 caliaec18759349
[root@worker232 ~]# 


	3.查看worker233节点
[root@worker233 ~]# ip a
...
11: calid13af264431@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
[root@worker233 ~]# 
[root@worker233 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.254      0.0.0.0         UG    0      0        0 eth0
...
10.100.140.86   0.0.0.0         255.255.255.255 UH    0      0        0 calid13af264431
...
[root@worker233 ~]#

5.3 验证宿主机工作模式

	1.宿主机访问测试
[root@master231 ~]# curl 10.100.140.86 
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>
    <div>
      <img src="2.jpg">
    <div>
  </body>

</html>
[root@master231 ~]# 

	
	2.在worker233节点抓包测试
[root@worker233 ~]# tcpdump -i calid13af264431 -nn tcp port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on calid13af264431, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:41:00.750300 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [S], seq 5058614, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
00:41:00.750329 IP 10.100.140.86.80 > 10.0.0.231.57502: Flags [S.], seq 2526914944, ack 5058615, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
00:41:00.750911 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [.], ack 1, win 502, length 0
00:41:00.751127 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [P.], seq 1:78, ack 1, win 502, length 77: HTTP: GET / HTTP/1.1
00:41:00.751150 IP 10.100.140.86.80 > 10.0.0.231.57502: Flags [.], ack 78, win 507, length 0
00:41:00.751250 IP 10.100.140.86.80 > 10.0.0.231.57502: Flags [P.], seq 1:239, ack 78, win 507, length 238: HTTP: HTTP/1.1 200 OK
00:41:00.751351 IP 10.100.140.86.80 > 10.0.0.231.57502: Flags [P.], seq 239:594, ack 78, win 507, length 355: HTTP
00:41:00.752027 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [.], ack 239, win 501, length 0
00:41:00.752050 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [.], ack 594, win 501, length 0
00:41:00.752281 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [F.], seq 78, ack 594, win 501, length 0
00:41:00.752456 IP 10.100.140.86.80 > 10.0.0.231.57502: Flags [F.], seq 594, ack 79, win 507, length 0
00:41:00.752720 IP 10.0.0.231.57502 > 10.100.140.86.80: Flags [.], ack 595, win 501, length 0


	3.在eth0物理接口抓包验证【说明未经过隧道封装】
[root@worker233 ~]# tcpdump -i eth0 -nn tcp port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:44:47.387845 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [S], seq 2484663598, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
00:44:47.387948 IP 10.100.140.86.80 > 10.0.0.231.34638: Flags [S.], seq 1724452665, ack 2484663599, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
00:44:47.388229 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [.], ack 1, win 502, length 0
00:44:47.388279 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [P.], seq 1:78, ack 1, win 502, length 77: HTTP: GET / HTTP/1.1
00:44:47.388301 IP 10.100.140.86.80 > 10.0.0.231.34638: Flags [.], ack 78, win 507, length 0
00:44:47.388463 IP 10.100.140.86.80 > 10.0.0.231.34638: Flags [P.], seq 1:239, ack 78, win 507, length 238: HTTP: HTTP/1.1 200 OK
00:44:47.388557 IP 10.100.140.86.80 > 10.0.0.231.34638: Flags [P.], seq 239:594, ack 78, win 507, length 355: HTTP
00:44:47.388617 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [.], ack 239, win 501, length 0
00:44:47.388658 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [.], ack 594, win 501, length 0
00:44:47.389297 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [F.], seq 78, ack 594, win 501, length 0
00:44:47.389410 IP 10.100.140.86.80 > 10.0.0.231.34638: Flags [F.], seq 594, ack 79, win 507, length 0
00:44:47.389684 IP 10.0.0.231.34638 > 10.100.140.86.80: Flags [.], ack 595, win 501, length 0

5.4 验证IPIP工作模式

5.4.1 切换Calico的工作模式为IPIP

	1.导出资源清单
[root@master231 ~]# kubectl get ippools default-ipv4-ippool -o yaml > ippool.yaml

	
	2.修改资源清单内容
[root@master231 ~]# egrep "ipipMode|vxlanMode" ippool.yaml # 修改前内容
  ipipMode: Never
  vxlanMode: CrossSubnet
[root@master231 ~]# 
[root@master231 ~]# vim ippool.yaml
[root@master231 ~]# 
[root@master231 ~]# egrep "ipipMode|vxlanMode" ippool.yaml  # 修改后内容
  ipipMode: Always
  vxlanMode: Never
[root@master231 ~]# 

	3.应用配置
[root@master231 ~]# kubectl apply -f ippool.yaml


	4.检查所有worker节点，检查是否存在"tunl0"设备
[root@master231 ~]# ip link show
...
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
[root@master231 ~]# 


[root@worker232 ~]# ip link show
...
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
[root@worker232 ~]# 


[root@worker233 ~]# ip link show
...
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
[root@worker233 ~]#

5.4.2 验证Calico的IPIP工作模式

	1.宿主机访问测试
[root@master231 ~]# curl 10.100.140.86 
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>
    <div>
      <img src="2.jpg">
    <div>
  </body>

</html>
[root@master231 ~]# 

	
	2.在worker233节点抓包测试
[root@worker233 ~]# tcpdump -i calid13af264431 -nn tcp port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on calid13af264431, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:56:56.976227 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [S], seq 175422332, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
00:56:56.976249 IP 10.100.140.86.80 > 10.100.160.157.35950: Flags [S.], seq 4286128692, ack 175422333, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
00:56:56.976565 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [.], ack 1, win 507, length 0
00:56:56.976628 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [P.], seq 1:78, ack 1, win 507, length 77: HTTP: GET / HTTP/1.1
00:56:56.976633 IP 10.100.140.86.80 > 10.100.160.157.35950: Flags [.], ack 78, win 507, length 0
00:56:56.976756 IP 10.100.140.86.80 > 10.100.160.157.35950: Flags [P.], seq 1:239, ack 78, win 507, length 238: HTTP: HTTP/1.1 200 OK
00:56:56.976864 IP 10.100.140.86.80 > 10.100.160.157.35950: Flags [P.], seq 239:594, ack 78, win 507, length 355: HTTP
00:56:56.976959 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [.], ack 239, win 506, length 0
00:56:56.977115 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [.], ack 594, win 504, length 0
00:56:56.977481 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [F.], seq 78, ack 594, win 504, length 0
00:56:56.977555 IP 10.100.140.86.80 > 10.100.160.157.35950: Flags [F.], seq 594, ack 79, win 507, length 0
00:56:56.977873 IP 10.100.160.157.35950 > 10.100.140.86.80: Flags [.], ack 595, win 504, length 0


	3.在eth0接口抓包验证
[root@worker233 ~]# tcpdump -i eth0 -nn tcp port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
... # 很明显，发现无法抓到数据报文哟~【说明进入隧道模式进行数据报文封装啦~】


	4.抓取VXLAN相关的数据报文
[root@worker233 ~]# tcpdump -i eth0 -nn host 10.0.0.231
... # 不难发现，我们成功在节点上抓取到了IPIP相关的数据报文哟~
01:16:57.207903 IP 10.0.0.231 > 10.0.0.233: IP 10.100.160.159.60970 > 10.100.140.86.80: Flags [S], seq 869143365, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
01:16:57.208028 IP 10.0.0.233 > 10.0.0.231: IP 10.100.140.86.80 > 10.100.160.159.60970: Flags [S.], seq 1834752713, ack 869143366, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
01:16:57.208267 IP 10.0.0.231 > 10.0.0.233: IP 10.100.160.159.60970 > 10.100.140.86.80: Flags [.], ack 1, win 507, length 0
01:16:57.208346 IP 10.0.0.231 > 10.0.0.233: IP 10.100.160.159.60970 > 10.100.140.86.80: Flags [P.], seq 1:78, ack 1, win 507, length 77: HTTP: GET / HTTP/1.1
01:16:57.208423 IP 10.0.0.233 > 10.0.0.231: IP 10.100.140.86.80 > 10.100.160.159.60970: Flags [.], ack 78, win 507, length 0
01:16:57.208613 IP 10.0.0.233 > 10.0.0.231: IP 10.100.140.86.80 > 10.100.160.159.60970: Flags [P.], seq 1:239, ack 78, win 507, length 238: HTTP: HTTP/1.1 200 OK
01:16:57.208715 IP 10.0.0.233 > 10.0.0.231: IP 10.100.140.86.80 > 10.100.160.159.60970: Flags [P.], seq 239:594, ack 78, win 507, length 355: HTTP
01:16:57.208776 IP 10.0.0.231 > 10.0.0.233: IP 10.100.160.159.60970 > 10.100.140.86.80: Flags [.], ack 239, win 506, length 0
01:16:57.208968 IP 10.0.0.231 > 10.0.0.233: IP 10.100.160.159.60970 > 10.100.140.86.80: Flags [.], ack 594, win 504, length 0
01:16:57.209122 IP 10.0.0.231 > 10.0.0.233: IP 10.100.160.159.60970 > 10.100.140.86.80: Flags [F.], seq 78, ack 594, win 504, length 0
...

5.5 验证vxlan工作模式

5.5.1 切换Calico的工作模式为VXLAN

	1.导出资源清单
[root@master231 ~]# kubectl get ippools default-ipv4-ippool -o yaml > ippool.yaml

	
	2.修改资源清单内容
[root@master231 ~]# egrep "ipipMode|vxlanMode" ippool.yaml # 修改前内容
  ipipMode: Always
  vxlanMode: Never
[root@master231 ~]# 
[root@master231 ~]# vim ippool.yaml
[root@master231 ~]# 
[root@master231 ~]# egrep "ipipMode|vxlanMode" ippool.yaml  # 修改后内容
  ipipMode: Never
  vxlanMode: Always
[root@master231 ~]# 

	3.应用配置
[root@master231 ~]# kubectl apply -f ippool.yaml


	4.检查所有worker节点，检查路由信息
[root@master231 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
...
10.100.140.64   10.100.140.89   255.255.255.192 UG    0      0        0 vxlan.calico
10.100.203.128  10.100.203.153  255.255.255.192 UG    0      0        0 vxlan.calico
[root@master231 ~]# 


[root@worker232 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
...
10.100.140.64   10.100.140.89   255.255.255.192 UG    0      0        0 vxlan.calico
10.100.160.128  10.100.160.158  255.255.255.192 UG    0      0        0 vxlan.calico
[root@worker232 ~]# 


[root@worker233 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
...
10.100.160.128  10.100.160.158  255.255.255.192 UG    0      0        0 vxlan.calico
10.100.203.128  10.100.203.153  255.255.255.192 UG    0      0        0 vxlan.calico
[root@worker233 ~]#

5.5.2 验证Calico的vxlan工作模式

	1.宿主机访问测试
[root@master231 ~]# curl 10.100.140.86 
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>
    <div>
      <img src="2.jpg">
    <div>
  </body>

</html>
[root@master231 ~]# 

	
	2.在worker233节点抓包测试
[root@worker233 ~]# tcpdump -i calid13af264431 -nn tcp port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on calid13af264431, link-type EN10MB (Ethernet), snapshot length 262144 bytes
01:12:52.582357 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [S], seq 1978429517, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
01:12:52.582377 IP 10.100.140.86.80 > 10.100.160.158.45226: Flags [S.], seq 3093037311, ack 1978429518, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
01:12:52.582791 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [.], ack 1, win 507, length 0
01:12:52.582795 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [P.], seq 1:78, ack 1, win 507, length 77: HTTP: GET / HTTP/1.1
01:12:52.582828 IP 10.100.140.86.80 > 10.100.160.158.45226: Flags [.], ack 78, win 507, length 0
01:12:52.582952 IP 10.100.140.86.80 > 10.100.160.158.45226: Flags [P.], seq 1:239, ack 78, win 507, length 238: HTTP: HTTP/1.1 200 OK
01:12:52.583068 IP 10.100.140.86.80 > 10.100.160.158.45226: Flags [P.], seq 239:594, ack 78, win 507, length 355: HTTP
01:12:52.583514 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [.], ack 239, win 506, length 0
01:12:52.583518 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [.], ack 594, win 504, length 0
01:12:52.583603 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [F.], seq 78, ack 594, win 504, length 0
01:12:52.583768 IP 10.100.140.86.80 > 10.100.160.158.45226: Flags [F.], seq 594, ack 79, win 507, length 0
01:12:52.584591 IP 10.100.160.158.45226 > 10.100.140.86.80: Flags [.], ack 595, win 504, length 0



	3.在eth0接口抓包验证
[root@worker233 ~]# tcpdump -i eth0 -nn tcp port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
... # 很明显，发现无法抓到数据报文哟~【说明进入隧道模式进行数据报文封装啦~】


	4.抓取VXLAN相关的数据报文
[root@worker233 ~]# tcpdump -i eth0 -nn host 10.0.0.231
... # 不难发现，我们成功在节点上抓取到了VXLAN相关的数据报文哟~
01:14:11.390022 IP 10.0.0.231.54576 > 10.0.0.233.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.160.158.52696 > 10.100.140.86.80: Flags [S], seq 1849605526, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
01:14:11.390155 IP 10.0.0.233.40943 > 10.0.0.231.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.140.86.80 > 10.100.160.158.52696: Flags [S.], seq 3091381087, ack 1849605527, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
01:14:11.390517 IP 10.0.0.231.54576 > 10.0.0.233.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.160.158.52696 > 10.100.140.86.80: Flags [.], ack 1, win 507, length 0
01:14:11.390518 IP 10.0.0.231.54576 > 10.0.0.233.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.160.158.52696 > 10.100.140.86.80: Flags [P.], seq 1:78, ack 1, win 507, length 77: HTTP: GET / HTTP/1.1
01:14:11.390602 IP 10.0.0.233.40943 > 10.0.0.231.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.140.86.80 > 10.100.160.158.52696: Flags [.], ack 78, win 507, length 0
01:14:11.390838 IP 10.0.0.233.40943 > 10.0.0.231.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.140.86.80 > 10.100.160.158.52696: Flags [P.], seq 1:239, ack 78, win 507, length 238: HTTP: HTTP/1.1 200 OK
01:14:11.390938 IP 10.0.0.233.40943 > 10.0.0.231.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.140.86.80 > 10.100.160.158.52696: Flags [P.], seq 239:594, ack 78, win 507, length 355: HTTP
01:14:11.391012 IP 10.0.0.231.54576 > 10.0.0.233.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.160.158.52696 > 10.100.140.86.80: Flags [.], ack 239, win 506, length 0
01:14:11.391153 IP 10.0.0.231.54576 > 10.0.0.233.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.100.160.158.52696 > 10.100.140.86.80: Flags [.], ack 594, win 504, length 0
...
posted @ 2025-03-07 01:39  尹正杰  阅读(915)  评论(0)    收藏  举报