Cilium BGP Control Plane
Cilium BGP 宣告方案与 MatelLB BGP 方案类似
| Ubunt | Ubuntu 26.04.1 LTS |
|---|---|
| Kind | v0.33.0 go1.26.7 linux/amd64 |
| Containerlab | 0.59.0 |
| Kubernetes | v1.35.8 |
1. 创建集群
1.1. 创建 K8s 集群
#!/bin/bash
set -v
## 1.Prepare NoCNI kubernetes environment
## kindest/node:v1.35.8
cat << EOF | kind create cluster --name=cilium-bgp --image=kindest/node:v1.35.8 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true
podSubnet: "10.98.0.0/16"
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.5.10
node-labels: "rack=rack0"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.5.11
node-labels: "rack=rack0"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.8.10
node-labels: "rack=rack1"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.8.11
node-labels: "rack=rack1"
EOF
禁用 IPV6 时创建 Kind 网络出现的问题
## 当前主机内核层面 IPv6 是关闭状态
## Kind 的 nerdctl provider 创建的 kind 网络是双栈的,除了 10.4.x 的 v4 子网还自动分配了一个 ipv6 子网
## 报错信息提示: CNI bridge 插件给网桥 br-0b27c158feb3 添加 fc00:f853:ccd:e793::1/64 被内核拒绝了
using nerdctl due to KIND_EXPERIMENTAL_PROVIDER
Creating cluster "cilium-bgp" ...
✓ Ensuring node image (kindest/node:v1.35.8) 🖼️
✗ Preparing nodes 📦 📦 📦 📦
ERROR: failed to create cluster: command "nerdctl run --name cilium-bgp-control-plane --hostname cilium-bgp-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs tmp --tmpfs run --volume var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=cilium-bgp --net kind --restart=on-failure:1 --init=false --publish=127.0.0.1:36443:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.35.8" failed with error: exit status 1
Command Output: time="xxxx-xx-xxT08:09:59+08:00" level=fatal msg="failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createRuntime hook #0: exit status 1, stdout: , stderr: time=\"2026-09-13T08:09:57+08:00\" level=warning msg=\"Container failed starting. Removing allocated network configuration.\"\ntime=\"2026-09-13T08:09:57+08:00\" level=fatal msg=\"failed to call cni.Setup: plugin type=\\\"bridge\\\" failed (add): failed to set bridge addr: could not add IP address fc00:f853:ccd:e793::1/64 to \\\"br-0b27c158feb3\\\": permission denied\""
## 解决方式
## http://github.com/containers/netavark/issues/439
## https://github.com/kubernetes-sigs/kind/issues/4156
## 目前通过预创建一个名为 kind 的纯 IPv4 网络可以作为一种变通方法
nerdctl network create kind --subnet x.x.x.x/xx
1.2. 创建 交换机/路由器 环境
用于模拟 K8s 节点处于不同网段
flowchart TB
sp0["spine0<br/>eth1: 10.1.10.2（连 leaf0）<br/>eth2: 10.1.34.2（连 leaf1）"]
sp1["spine1<br/>eth1: 10.1.12.2（连 leaf0）<br/>eth2: 10.1.11.2（连 leaf1）"]
leaf0["leaf0<br/>eth3: 10.1.5.1<br/>（rack0 网关）<br/>eth1: 10.1.10.1<br/>eth2: 10.1.12.1"]
leaf1["leaf1<br/>eth3: 10.1.8.1<br/>（rack1 网关）<br/>eth1: 10.1.34.1<br/>eth2: 10.1.11.1"]
br0["br-leaf0<br/>10.1.5.0/24"]
br1["br-leaf1<br/>10.1.8.0/24"]
subgraph rack0["rack0"]
cp["cilium-bgp-control-plane<br/>net0: 10.1.5.10"]
wk1["cilium-bgp-worker <br/>net0: 10.1.5.11"]
end
subgraph rack1["rack1"]
wk2["cilium-bgp-worker2 <br/>net0: 10.1.8.10"]
wk3["cilium-bgp-worker3 <br/>net0: 10.1.8.11"]
end
sp0 --- leaf0
sp0 --- leaf1
sp1 --- leaf0
sp1 --- leaf1
leaf0 --- br0
leaf1 --- br1
br0 --- cp
br0 --- wk1
br1 --- wk2
br1 --- wk3
#!/bin/bash
## burlyluo/vyos:1.4.9
## burlyluo/nettool:latest
set -v
# containerlab 的 kind: bridge 节点要求网桥预先存在
BRIDGES=(br-leaf0 br-leaf1)
# 清理旧网桥（存在才删，避免报错）
for br in "${BRIDGES[@]}"; do
if ip link show "$br" &>/dev/null; then
ip link del "$br"
fi
done
# 创建并启用
for br in "${BRIDGES[@]}"; do
ip link add "$br" type bridge
ip link set "$br" up
done
cat << EOF > clab.yaml | containerlab deploy -t clab.yaml -
name: cilium-bgp
topology:
nodes:
spine0:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/spine0-boot.cfg:/opt/vyatta/etc/config/config.boot
spine1:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/spine1-boot.cfg:/opt/vyatta/etc/config/config.boot
leaf0:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/leaf0-boot.cfg:/opt/vyatta/etc/config/config.boot
leaf1:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/leaf1-boot.cfg:/opt/vyatta/etc/config/config.boot
br-leaf0:
kind: bridge
br-leaf1:
kind: bridge
server1:
kind: linux
image: burlyluo/nettool:latest
network-mode: container:cilium-bgp-control-plane
exec:
- ip addr add 10.1.5.10/24 dev net0
- ip route replace default via 10.1.5.1
server2:
kind: linux
image: burlyluo/nettool:latest
network-mode: container:cilium-bgp-worker
exec:
- ip addr add 10.1.5.11/24 dev net0
- ip route replace default via 10.1.5.1
server3:
kind: linux
image: burlyluo/nettool:latest
network-mode: container:cilium-bgp-worker2
exec:
- ip addr add 10.1.8.10/24 dev net0
- ip route replace default via 10.1.8.1
server4:
kind: linux
image: burlyluo/nettool:latest
network-mode: container:cilium-bgp-worker3
exec:
- ip addr add 10.1.8.11/24 dev net0
- ip route replace default via 10.1.8.1
links:
- endpoints: ["br-leaf0:br-leaf0-net0", "server1:net0"]
mtu: 1500
- endpoints: ["br-leaf0:br-leaf0-net1", "server2:net0"]
mtu: 1500
- endpoints: ["br-leaf1:br-leaf1-net0", "server3:net0"]
mtu: 1500
- endpoints: ["br-leaf1:br-leaf1-net1", "server4:net0"]
mtu: 1500
- endpoints: ["leaf0:eth1", "spine0:eth1"]
mtu: 1500
- endpoints: ["leaf0:eth2", "spine1:eth1"]
mtu: 1500
- endpoints: ["leaf0:eth3", "br-leaf0:br-leaf0-net2"]
mtu: 1500
- endpoints: ["leaf1:eth1", "spine0:eth2"]
mtu: 1500
- endpoints: ["leaf1:eth2", "spine1:eth2"]
mtu: 1500
- endpoints: ["leaf1:eth3", "br-leaf1:br-leaf1-net2"]
mtu: 1500
EOF
1.2.1. Leaf 配置
既做二层交换（本地网段内按 MAC 转发，比如 rack0 两台节点在 br-leaf0 上互访），又做三层路由（当网关、和 spine 跑 BGP）。市面上这类产品的名字一般叫 "三层交换机" 或者 "多层交换机"，简单来讲就是现在产品经过这么久迭代，能力已经很强了，交换机也具备路由器能力。
interfaces {
ethernet eth1 {
address "10.1.10.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth2 {
address "10.1.12.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth3 {
address "10.1.5.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
loopback lo {
}
}
nat {
source {
rule 100 {
outbound-interface {
name "eth0"
}
source {
address "10.1.0.0/16"
}
translation {
address "masquerade"
}
}
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.5.0/24 {
}
network 10.1.10.0/24 {
}
network 10.1.12.0/24 {
}
}
}
neighbor 10.1.5.10 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65005"
}
neighbor 10.1.5.11 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65005"
}
neighbor 10.1.10.2 {
address-family {
ipv4-unicast
}
remote-as "500"
}
neighbor 10.1.12.2 {
address-family {
ipv4-unicast
}
remote-as "800"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id "10.1.5.1"
}
system-as "65005"
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "leaf0"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "Asia/Shanghai"
}
interfaces {
ethernet eth1 {
address "10.1.34.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth2 {
address "10.1.11.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth3 {
address "10.1.8.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
loopback lo {
}
}
nat {
source {
rule 100 {
outbound-interface {
name "eth0"
}
source {
address "10.1.0.0/16"
}
translation {
address "masquerade"
}
}
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.8.0/24 {
}
network 10.1.10.0/24 {
}
network 10.1.12.0/24 {
}
}
}
neighbor 10.1.8.10 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65008"
}
neighbor 10.1.8.11 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65008"
}
neighbor 10.1.11.2 {
address-family {
ipv4-unicast
}
remote-as "800"
}
neighbor 10.1.34.2 {
address-family {
ipv4-unicast
}
remote-as "500"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id "10.1.8.1"
}
system-as "65008"
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "leaf1"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "Asia/Shanghai"
}
1.2.2. Spine 配置
这个就是传统意义上的路由器，纯路由转发。
interfaces {
ethernet eth1 {
address "10.1.10.2/24"
duplex "auto"
speed "auto"
}
ethernet eth2 {
address "10.1.34.2/24"
duplex "auto"
speed "auto"
}
loopback lo {
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.10.0/24 {
}
network 10.1.34.0/24 {
}
}
}
neighbor 10.1.10.1 {
address-family {
ipv4-unicast
}
remote-as "65005"
}
neighbor 10.1.34.1 {
address-family {
ipv4-unicast
}
remote-as "65008"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
}
system-as "500"
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "spine0"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "Asia/Shanghai"
}
interfaces {
ethernet eth1 {
address "10.1.12.2/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth2 {
address "10.1.11.2/24"
duplex "auto"
mtu "9000"
speed "auto"
}
loopback lo {
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.11.0/24 {
}
network 10.1.12.0/24 {
}
}
}
neighbor 10.1.11.1 {
address-family {
ipv4-unicast
}
remote-as "65008"
}
neighbor 10.1.12.1 {
address-family {
ipv4-unicast
}
remote-as "65005"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id "10.1.8.1"
}
system-as "800"
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "spine1"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "Asia/Shanghai"
}
1.3. 创建 Cilium 网络
#!/bin/bash
set -v
# 1.Remove kubernetes node taints
kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule-
# 2.Install Cilium[Cilium v1.15.17]
{ helm repo add cilium https://helm.cilium.io ; helm repo update; } > /dev/null 2>&1
controller_node_ip=`docker exec $(docker ps -a --format "table {{.Names}}" | grep control) ip a s eth0 | awk -F "inet " '{print $2}' | grep 172.18.0. | awk -F "/16" '{print $1}'`
## 具体参数作用看官网注释：
## https://github.com/cilium/cilium/blob/v1.19/install/kubernetes/cilium/values.yaml
helm install cilium cilium/cilium --version v1.19.7 \
--namespace kube-system \
--set image.pullPolicy=IfNotPresent \
--set k8sServiceHost=$controller_node_ip \
--set k8sServicePort=6443 \
--set debug.enabled=true \
--set debug.verbose="datapath flow kvstore envoy policy" \
--set bpf.monitorAggregation=none \
--set monitor.enabled=true \
--set ipam.mode=kubernetes \
--set routingMode=native \
--set ipv4NativeRoutingCIDR=10.0.0.0/8 \
--set bgpControlPlane.enabled=true \
--set k8s.requireIPv4PodCIDR=true
1.4. 创建 Pod/Service
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: wluo
name: wluo
spec:
selector:
matchLabels:
app: wluo
template:
metadata:
labels:
app: wluo
spec:
containers:
- image: burlyluo/nettool:latest
name: nettoolbox
env:
- name: NETTOOL_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
privileged: true
---
apiVersion: v1
kind: Service
metadata:
name: wluo
spec:
type: NodePort
selector:
app: wluo
ports:
- name: wluo
port: 80
targetPort: 80
nodePort: 32000
1.5. 查看环境
1.5.1. 查看 Docker 容器信息
# docker ps
CONTAINER ID COMMAND CREATED STATUS PORTS NAMES
62908d76835c "/sbin/tini -g -- /e…" 6 hours ago Up 6 hours clab-cilium-bgp-server2
15dc82066e6d "/sbin/tini -g -- /e…" 6 hours ago Up 6 hours clab-cilium-bgp-server1
f20b9cbda683 "/sbin/tini -g -- /e…" 6 hours ago Up 6 hours clab-cilium-bgp-server3
4d0eafbf5b99 "/sbin/tini -g -- /e…" 6 hours ago Up 6 hours clab-cilium-bgp-server4
34f3d3ec7cad "/sbin/init" 6 hours ago Up 6 hours clab-cilium-bgp-leaf0
3f11ab2545b3 "/sbin/init" 6 hours ago Up 6 hours clab-cilium-bgp-leaf1
b7063813bc6f "/sbin/init" 6 hours ago Up 6 hours clab-cilium-bgp-spine1
24a65483aabc "/sbin/init" 6 hours ago Up 6 hours clab-cilium-bgp-spine0
f4467be78e72 "/usr/local/bin/entr…" 6 hours ago Up 6 hours cilium-bgp-worker2
a782a2d31c2b "/usr/local/bin/entr…" 6 hours ago Up 6 hours cilium-bgp-worker3
61bba62f759b "/usr/local/bin/entr…" 6 hours ago Up 6 hours cilium-bgp-worker
eef4d1eb9288 "/usr/local/bin/entr…" 6 hours ago Up 6 hours 127.0.0.1:37457->6443/tcp cilium-bgp-control-plane
1.5.2. 查看 K8s 集群信息
# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cilium-bgp-control-plane Ready control-plane 5h58m v1.35.8 10.1.5.10 <none> Debian GNU/Linux 13 (trixie) 7.0.0-30-generic containerd://2.3.4
cilium-bgp-worker Ready <none> 5h58m v1.35.8 10.1.5.11 <none> Debian GNU/Linux 13 (trixie) 7.0.0-30-generic containerd://2.3.4
cilium-bgp-worker2 Ready <none> 5h58m v1.35.8 10.1.8.10 <none> Debian GNU/Linux 13 (trixie) 7.0.0-30-generic containerd://2.3.4
cilium-bgp-worker3 Ready <none> 5h58m v1.35.8 10.1.8.11 <none> Debian GNU/Linux 13 (trixie) 7.0.0-30-generic containerd://2.3.4
# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default wluo-bh5rs 1/1 Running 0 3h4m 10.98.0.12 cilium-bgp-control-plane
default wluo-n2hcd 1/1 Running 0 3h4m 10.98.1.162 cilium-bgp-worker
default wluo-x6pvs 1/1 Running 0 3h4m 10.98.3.154 cilium-bgp-worker3
default wluo-xz7g5 1/1 Running 0 3h4m 10.98.2.36 cilium-bgp-worker2
kube-system cilium-8djvz 2/2 Running 0 5h51m 10.1.8.11 cilium-bgp-worker3
kube-system cilium-c8t5q 2/2 Running 0 5h51m 10.1.8.10 cilium-bgp-worker2
kube-system cilium-envoy-gppjt 1/1 Running 0 5h51m 10.1.8.10 cilium-bgp-worker2
kube-system cilium-envoy-jmmcr 1/1 Running 0 5h51m 10.1.8.11 cilium-bgp-worker3
kube-system cilium-envoy-mjzcj 1/1 Running 0 5h51m 10.1.5.10 cilium-bgp-control-plane
kube-system cilium-envoy-rrm4h 1/1 Running 0 5h51m 10.1.5.11 cilium-bgp-worker
kube-system cilium-fjprr 2/2 Running 0 5h51m 10.1.5.11 cilium-bgp-worker
kube-system cilium-operator-9f4ffd979-gn5kq 1/1 Running 0 5h51m 10.1.8.11 cilium-bgp-worker3
kube-system cilium-operator-9f4ffd979-jzcm9 1/1 Running 0 5h51m 10.1.5.11 cilium-bgp-worker
kube-system cilium-rsl97 2/2 Running 0 5h51m 10.1.5.10 cilium-bgp-control-plane
kube-system coredns-7d764666f9-tm6x5 1/1 Running 0 5h59m 10.98.3.177 cilium-bgp-worker3
kube-system coredns-7d764666f9-wmz69 1/1 Running 0 5h59m 10.98.3.215 cilium-bgp-worker3
kube-system etcd-cilium-bgp-control-plane 1/1 Running 0 5h59m 10.1.5.10 cilium-bgp-control-plane
kube-system kube-apiserver-cilium-bgp-control-plane 1/1 Running 0 5h59m 10.1.5.10 cilium-bgp-control-plane
kube-system kube-controller-manager-cilium-bgp-control-plane 1/1 Running 0 5h59m 10.1.5.10 cilium-bgp-control-plane
kube-system kube-proxy-52k96 1/1 Running 0 5h59m 10.1.5.11 cilium-bgp-worker
kube-system kube-proxy-7kgnz 1/1 Running 0 5h59m 10.1.8.11 cilium-bgp-worker3
kube-system kube-proxy-dm6sr 1/1 Running 0 5h59m 10.1.8.10 cilium-bgp-worker2
kube-system kube-proxy-pwzcr 1/1 Running 0 5h59m 10.1.5.10 cilium-bgp-control-plane
kube-system kube-scheduler-cilium-bgp-control-plane 1/1 Running 0 5h59m 10.1.5.10 cilium-bgp-control-plane
local-path-storage local-path-provisioner-67b778fdf6-57bwm 1/1 Running 0 5h59m 10.98.3.71 cilium-bgp-worker3
2. 路由宣告
2.1. Pod 网段宣告
宣告的不是 Pod 本身，而是每个节点分到的 Pod 网段（CIDR）
2.1.1. BGP 路由宣告前访问 Pod
- 两个节点的 net0 都在 10.1.5.0/24 里（同挂 leaf0 三层路由器），但两个 Pod 分属 10.98.0.0/24 和 10.98.1.0/24 不同网段。所以 Pod 互访是跨网段路由，不是二层直通，中间必须有对端网段路由。
- Pod 里只有一条走 10.98.0.53 的默认路由。抓包发现 Pod 先 ARP 问了一圈谁是 10.98.0.53，被回复的 MAC 是 lxc544538ebab54（Pod eth0 的 veth 在节点侧），而不是 cilium_host 10.98.0.53 对应的 MAC 地址。所以 cilium_host 只是把网关 IP 挂在 Pod 身上，让 ARP 有人应答，流量本身从 veth 直接进节点，不经过它。
- 包进 lxc 口后，先被 Cilium 挂在这个口的 eBPF 程序处理（策略检查、打身份标签等...），然后才查路由表。节点上只有自己的 10.98.0.0/24，没有去 10.98.1.0/24 的路由，只能发给默认路由，包从 net0 发给 10.1.5.1（leaf0 三层网关）。
- leaf0 查表也没有 10.98.1.0/24，就又落在默认路由（K>* 0.0.0.0/0 via 172.20.20.1, eth0）上，把包扔进 containerlab 管理网。之后两种下场，反正没人应答，curl 就超时了：
- 被宿主机 rp_filter 当成来路不明的包丢弃（源 IP 10.98.0.12 本来就不该从管理口进来）
- 顺着宿主机的默认路由继续往外漂
# kubectl exec -it wluo-bh5rs -- curl -m3 10.98.1.162
curl: (28) Connection timed out after 3000 milliseconds
command terminated with exit code 28
查看 Pod/Node 路由、网卡信息
## 查看 Pod 内路由
## 默认路由发往 10.98.0.53,此 IP 为 Pod 节点中 cilium_host@cilium_net 设备 IP
# kubectl exec -it wluo-bh5rs -- ip route show
default via 10.98.0.53 dev eth0 mtu 1500
10.98.0.53 dev eth0 scope link
## 查看 Pod 节点路由信息，同样没有去往 10.98.1.162 的路由信息，丢给默认路由。此时发往 leaf0 路由器
# docker exec cilium-bgp-control-plane ip route show
default via 10.1.5.1 dev net0
10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10
10.98.0.0/24 via 10.98.0.53 dev cilium_host proto kernel src 10.98.0.53
10.98.0.53 dev cilium_host proto kernel scope link
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.5
## 查看 Pod 内网卡
# kubectl exec -it wluo-bh5rs -- ip address show eth0
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ea:fb:71:0e:de:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
altname cilium_cni:eth0
inet 10.98.0.12/32 scope global eth0
valid_lft forever preferred_lft forever
## 查看 Pod 内 eth0 接口 veth 虚拟网线对端接口(在 Pod 节点上)
# docker exec cilium-bgp-control-plane ip addr show lxc544538ebab54
8: lxc544538ebab54@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether f2:0c:b9:36:82:d8 brd ff:ff:ff:ff:ff:ff link-netns cni-6ecea9f3-3225-a0af-f982-b9ab334175b4
## 查看 Pod 网关信息
# docker exec cilium-bgp-control-plane ip addr show cilium_host
4: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 36:ba:35:81:22:14 brd ff:ff:ff:ff:ff:ff
inet 10.98.0.53/32 scope global cilium_host
valid_lft forever preferred_lft forever
查询 leaf 路由表信息
## 路由器同样没有去往 10.98.1.162 的路由信息
# docker exec -it clab-cilium-bgp-leaf0 bash
root@leaf0:/# show ip route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 03:41:21
C>* 10.1.5.0/24 is directly connected, eth3, 03:41:18
B>* 10.1.8.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 03:41:13
* via 10.1.12.2, eth2, weight 1, 03:41:13
C>* 10.1.10.0/24 is directly connected, eth1, 03:41:18
B>* 10.1.11.0/24 [20/0] via 10.1.12.2, eth2, weight 1, 03:41:14
C>* 10.1.12.0/24 is directly connected, eth2, 03:41:18
B>* 10.1.34.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 03:41:14
C>* 172.20.20.0/24 is directly connected, eth0, 03:41:21
在 Pod 中请求其他 Pod IP 时，多开一个终端在 Client Pod 出抓包
## 从抓包地址可以看出，ARP 响应的 MAC 地址信息为 veth 对端 lxc 地址，而不是 Pod 网关 cilium_host 的 MAC 地址
# kubectl exec -it wluo-bh5rs -- tcpdump -nn -vvv -i eth0 -e
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
07:24:33.907733 ea:fb:71:0e:de:a5 > f2:0c:b9:36:82:d8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 21187, offset 0, flags [DF], proto TCP (6), length 60)
10.98.0.12.51398 > 10.98.1.162.80: Flags [S], cksum 0x16a0 (incorrect -> 0xfa5a), seq 397012121, win 64240, options [mss 1460,sackOK,TS val 2200300480 ecr 0,nop,wscale 10], length 0
07:24:34.922274 ea:fb:71:0e:de:a5 > f2:0c:b9:36:82:d8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 21188, offset 0, flags [DF], proto TCP (6), length 60)
10.98.0.12.51398 > 10.98.1.162.80: Flags [S], cksum 0x16a0 (incorrect -> 0xf663), seq 397012121, win 64240, options [mss 1460,sackOK,TS val 2200301495 ecr 0,nop,wscale 10], length 0
07:24:35.945286 ea:fb:71:0e:de:a5 > f2:0c:b9:36:82:d8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 21189, offset 0, flags [DF], proto TCP (6), length 60)
10.98.0.12.51398 > 10.98.1.162.80: Flags [S], cksum 0x16a0 (incorrect -> 0xf264), seq 397012121, win 64240, options [mss 1460,sackOK,TS val 2200302518 ecr 0,nop,wscale 10], length 0
07:24:39.017276 ea:fb:71:0e:de:a5 > f2:0c:b9:36:82:d8, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.98.0.53 tell 10.98.0.12, length 28
07:24:39.017332 f2:0c:b9:36:82:d8 > ea:fb:71:0e:de:a5, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.98.0.53 is-at f2:0c:b9:36:82:d8, length 28
2.1.2. 创建 BGP 路由通告
apiVersion: cilium.io/v2
## 两个 rack 的 peers 都引用它
kind: CiliumBGPPeerConfig
metadata:
name: cilium-bgp-peer-config
spec:
## BGP 善后机制
gracefulRestart:
## 会话闪断（cilium agent 重启、节点升级）时，路由器不会立刻删掉之前学到的路由，而是保留一段时间等对端恢复
enabled: true
## 保留 120 秒
restartTimeSeconds: 120
families:
## 这个会话交换哪种路由
- afi: ipv4
## ipv4 + unicast = 普通 IPv4 单播前缀
safi: unicast
## 订阅哪些 CiliumBGPAdvertisement
advertisements:
matchLabels:
advertise: bgp
---
apiVersion: cilium.io/v2
## 具体配置
kind: CiliumBGPAdvertisement
metadata:
name: bgp-advertisements
## 被 CiliumBGPPeerConfig 的 advertisements 匹配
labels:
advertise: bgp
spec:
advertisements:
## 播报类型：把本节点的 podCIDR 宣告出去：
## - control-plane：10.98.0.0/24
## - worker3：10.98.3.0/24
## - ...
- advertisementType: "PodCIDR"
---
apiVersion: cilium.io/v2
## 和谁建立 BGP 会话
kind: CiliumBGPClusterConfig
metadata:
name: cilium-bgp-rack0
spec:
## 匹配哪些节点
nodeSelector:
matchLabels:
rack: rack0
bgpInstances:
## BGP 实例（虚拟路由器）的名字,全局唯一即可
- name: "instance-65005"
## 本端的 AS 自治系统号
localASN: 65005
## 对端信息
peers:
## 邻居名（标识）
- name: "peer-leaf0"
## 对端的 AS 自治系统号
peerASN: 65005
## 邻居建立会话的地址
peerAddress: "10.1.5.1"
## 当前配置连接到哪份 CiliumBGPPeerConfig 资源
peerConfigRef:
name: cilium-bgp-peer-config
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
name: cilium-bgp-rack1
spec:
nodeSelector:
matchLabels:
rack: rack1
bgpInstances:
- name: "instance-65008"
localASN: 65008
peers:
- name: "peer-leaf1"
peerASN: 65008
peerAddress: "10.1.8.1"
peerConfigRef:
name: cilium-bgp-peer-config
验证 BGP 路由通告
请求后，前三步的流量走向没有变化，最终进入 leaf 时，查询到了去往 10.98.1.162 的路由信息
查询 Cilium Daemonset 容器中识别的 BGP 信息
# kubectl exec -n kube-system cilium-rsl97 -- cilium-dbg bgp peers
Local AS Peer AS Peer Address Session Uptime Family Received Advertised
65005 65005 10.1.5.1:179 established 48m31s ipv4/unicast 10 1
# kubectl exec -n kube-system cilium-fjprr -- cilium-dbg bgp peers
Local AS Peer AS Peer Address Session Uptime Family Received Advertised
65005 65005 10.1.5.1:179 established 48m36s ipv4/unicast 10 1
# kubectl exec -n kube-system cilium-c8t5q -- cilium-dbg bgp peers
Local AS Peer AS Peer Address Session Uptime Family Received Advertised
65008 65008 10.1.8.1:179 established 48m47s ipv4/unicast 10 1
# kubectl exec -n kube-system cilium-8djvz -- cilium-dbg bgp peers
Local AS Peer AS Peer Address Session Uptime Family Received Advertised
65008 65008 10.1.8.1:179 established 48m57s ipv4/unicast 10 1
# kubectl exec -n kube-system cilium-8djvz -- cilium-dbg bgp routes
VRouter Prefix NextHop Age Attrs
65008 10.98.3.0/24 0.0.0.0 47m19s [{Origin: i} {Nexthop: 0.0.0.0}]
# kubectl exec -n kube-system cilium-c8t5q -- cilium-dbg bgp routes
VRouter Prefix NextHop Age Attrs
65008 10.98.2.0/24 0.0.0.0 47m26s [{Origin: i} {Nexthop: 0.0.0.0}]
# kubectl exec -n kube-system cilium-fjprr -- cilium-dbg bgp routes
VRouter Prefix NextHop Age Attrs
65005 10.98.1.0/24 0.0.0.0 47m33s [{Origin: i} {Nexthop: 0.0.0.0}]
# kubectl exec -n kube-system cilium-rsl97 -- cilium-dbg bgp routes
VRouter Prefix NextHop Age Attrs
65005 10.98.0.0/24 0.0.0.0 47m39s [{Origin: i} {Nexthop: 0.0.0.0}]
查询 leaf 路由器识别的 BGP 信息
# docker exec -it clab-cilium-bgp-leaf0 bash
root@leaf0:/# show ip route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 04:06:17
C>* 10.1.5.0/24 is directly connected, eth3, 04:06:14
B>* 10.1.8.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 04:06:09
* via 10.1.12.2, eth2, weight 1, 04:06:09
C>* 10.1.10.0/24 is directly connected, eth1, 04:06:14
B>* 10.1.11.0/24 [20/0] via 10.1.12.2, eth2, weight 1, 04:06:10
C>* 10.1.12.0/24 is directly connected, eth2, 04:06:14
B>* 10.1.34.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 04:06:10
## control-plane 的 Pod 网段,下一跳就是他自己
B>* 10.98.0.0/24 [200/0] via 10.1.5.10, eth3, weight 1, 00:02:37
## worker 的 Pod 网段
B>* 10.98.1.0/24 [200/0] via 10.1.5.11, eth3, weight 1, 00:02:37
## worker2 的 Pod 网段
B>* 10.98.2.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 00:02:38
* via 10.1.12.2, eth2, weight 1, 00:02:38
## worker3 的 Pod 网段
B>* 10.98.3.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 00:02:37
* via 10.1.12.2, eth2, weight 1, 00:02:37
C>* 172.20.20.0/24 is directly connected, eth0, 04:06:17
Pod 请求成功
# kubectl exec -it wluo-bh5rs -- curl -m3 10.98.1.162
PodName: wluo-n2hcd | PodIP: eth0 10.98.1.162/32
2.2. Service 宣告
在 Pod CIDR 宣告的基础上，通过 serviceSelector 额外宣告 Service 地址段
2.2.1. 通过 Cilium IPAM 分配 Service LoadBalancer
配置格式参考官网文档，不同版本配置信息会有些许差异。操作前先把上一步创建的资源都删了。
apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
name: "red-pool"
spec:
blocks:
## 形式一：整段 CIDR
- cidr: "40.0.10.0/24"
## 形式二：起止 IP 区间（和 CIDR 二选一，不能同时写）
# - start: "10.1.5.100"
# stop: "10.1.5.200"
## 可选：只把池中 IP 分给匹配的 Service 资源
serviceSelector:
matchLabels:
color: red
---
apiVersion: v1
kind: Service
metadata:
name: service-lb-ipam
labels:
color: red
spec:
type: LoadBalancer
selector:
color: red
ports:
- port: 80
---
apiVersion: v1
kind: Service
metadata:
name: nodeport
spec:
type: NodePort
selector:
color: red
ports:
- name: wluo
port: 80
targetPort: 80
nodePort: 32000
---
apiVersion: v1
kind: Service
metadata:
name: clusterip
spec:
type: ClusterIP
selector:
color: red
ports:
- name: wluo
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
color: red
name: lb-ipam
spec:
selector:
matchLabels:
color: red
template:
metadata:
labels:
color: red
spec:
containers:
- image: burlyluo/nettool:latest
name: nettoolbox
securityContext:
privileged: true
查看创建的 Pod/Service 资源信息
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
clusterip ClusterIP 10.96.195.136 <none> 80/TCP 3s
nodeport NodePort 10.96.25.197 <none> 80:32000/TCP 3s
service-lb-ipam LoadBalancer 10.96.78.63 40.0.10.0 80:30910/TCP 140m
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
lb-ipam-bgx96 1/1 Running 0 147m 10.98.0.201 cilium-bgp-control-plan
lb-ipam-f6gdv 1/1 Running 0 147m 10.98.1.50 cilium-bgp-worker
lb-ipam-wmm6s 1/1 Running 0 147m 10.98.2.123 cilium-bgp-worker2
lb-ipam-zw6p8 1/1 Running 0 147m 10.98.3.47 cilium-bgp-worker3
2.2.2. BGP 路由宣告前访问 Service
与 "BGP 路由宣告前访问 Pod" 情况基本一致，宣告前是没有 Service LoadBalancer 地址信息的，测试结果如下：
需要说明的是，本次创建 Service 资源使用了 ClusterIP、NodePort、LoadBalancer三种类型，匹配相同的分布在 4 个节点上的 4 个不同的 Pod 宣告前请求效果如下：
- ClusterIP：在 Cilium kube-proxy replacement 模式下，对 Service 的负载均衡通过 eBPF DNAT 实现的（不依赖 iptables 规则）。客户端所在节点直接把目的地址从 ClusterIP 改写成 Backend Pod IP，之后按普通 Pod 流量转发，ClusterIP 本身不离开源节点。改写发生在哪个 eBPF 程序上，取决于请求从哪来：
- 宿主机进程（本次 curl 场景）：改写发生在 Socket 层。Cilium 通过 socket-LB 在 cgroup 的 connect4 钩子挂载 BPF 程序，connect() 系统调用时就把目的地址改写完毕。包还没进入网络设备就已经是 Pod IP 了，所以 tcpdump 抓包时的目的地址是 Pod IP，而不是 ClusterIP。
- 集群内 Pod：改写发生在 Pod veth 上的 bpf_lxc 程序（流量出 Pod 到对端 lxc 设备时查服务表）
- 外部流量（NodePort、LoadBalancer 路径）：改写发生在物理网卡上的 bpf_host 程序
查询 leaf 路由器识别的 BGP 信息
# docker exec -it clab-cilium-bgp-leaf0 bash
root@leaf0:/# show ip route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 21:23:10
C>* 10.1.5.0/24 is directly connected, eth3, 21:23:07
B>* 10.1.8.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 21:23:02
* via 10.1.12.2, eth2, weight 1, 21:23:02
C>* 10.1.10.0/24 is directly connected, eth1, 21:23:07
B>* 10.1.11.0/24 [20/0] via 10.1.12.2, eth2, weight 1, 21:23:03
C>* 10.1.12.0/24 is directly connected, eth2, 21:23:07
B>* 10.1.34.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 21:23:03
C>* 172.20.20.0/24 is directly connected, eth0, 21:23:10
## 查询路由表信息
root@leaf0:/# ip route show
default via 172.20.20.1 dev eth0
10.1.5.0/24 dev eth3 proto kernel scope link src 10.1.5.1
10.1.8.0/24 nhid 30 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.1.10.0/24 dev eth1 proto kernel scope link src 10.1.10.1
10.1.11.0/24 nhid 26 via 10.1.12.2 dev eth2 proto bgp metric 20
10.1.12.0/24 dev eth2 proto kernel scope link src 10.1.12.1
10.1.34.0/24 nhid 25 via 10.1.10.2 dev eth1 proto bgp metric 20
172.20.20.0/24 dev eth0 proto kernel scope link src 172.20.20.2
查询 Cilium 监听信息
# kubectl -n kube-system exec ds/cilium -- cilium-dbg service list
ID Frontend Service Type Backend
0 40.0.10.0:80/TCP LoadBalancer 1 => 10.98.0.201:80/TCP (active)
2 => 10.98.1.50:80/TCP (active)
3 => 10.98.2.123:80/TCP (active)
4 => 10.98.3.47:80/TCP (active)
1 10.96.0.10:53/TCP ClusterIP 1 => 10.98.3.177:53/TCP (active)
2 => 10.98.3.215:53/TCP (active)
2 10.96.0.10:53/UDP ClusterIP 1 => 10.98.3.177:53/UDP (active)
2 => 10.98.3.215:53/UDP (active)
3 10.96.0.10:9153/TCP ClusterIP 1 => 10.98.3.177:9153/TCP (active)
2 => 10.98.3.215:9153/TCP (active)
4 10.96.0.1:443/TCP ClusterIP 1 => 172.18.0.5:6443/TCP (active)
5 10.96.64.225:443/TCP ClusterIP 1 => 10.1.8.11:4244/TCP (active)
9 10.96.78.63:80/TCP ClusterIP 1 => 10.98.0.201:80/TCP (active)
2 => 10.98.1.50:80/TCP (active)
3 => 10.98.2.123:80/TCP (active)
4 => 10.98.3.47:80/TCP (active)
12 10.96.25.197:80/TCP ClusterIP 1 => 10.98.0.201:80/TCP (active)
2 => 10.98.1.50:80/TCP (active)
3 => 10.98.2.123:80/TCP (active)
4 => 10.98.3.47:80/TCP (active)
13 10.96.195.136:80/TCP ClusterIP 1 => 10.98.0.201:80/TCP (active)
2 => 10.98.1.50:80/TCP (active)
3 => 10.98.2.123:80/TCP (active)
4 => 10.98.3.47:80/TCP (active)
# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list
SERVICE ADDRESS BACKEND ADDRESS (REVNAT_ID) (SLOT)
10.96.195.136:80/TCP (1) 10.98.0.201:80/TCP (13) (1)
10.96.195.136:80/TCP (4) 10.98.3.47:80/TCP (13) (4)
10.96.64.225:0/ANY (0) 0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
10.96.0.1:443/TCP (0) 0.0.0.0:0 (4) (0) [ClusterIP, non-routable]
10.96.78.63:80/TCP (4) 10.98.3.47:80/TCP (9) (4)
10.96.78.63:80/TCP (1) 10.98.0.201:80/TCP (9) (1)
10.96.0.10:9153/TCP (2) 10.98.3.215:9153/TCP (3) (2)
10.96.0.10:53/TCP (0) 0.0.0.0:0 (1) (0) [ClusterIP, non-routable]
10.96.195.136:80/TCP (2) 10.98.1.50:80/TCP (13) (2)
10.96.25.197:80/TCP (2) 10.98.1.50:80/TCP (12) (2)
10.96.64.225:443/TCP (1) 10.1.8.11:4244/TCP (5) (1)
10.96.25.197:80/TCP (4) 10.98.3.47:80/TCP (12) (4)
10.96.25.197:0/ANY (0) 0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
10.96.78.63:80/TCP (0) 0.0.0.0:0 (9) (0) [ClusterIP, non-routable]
10.96.78.63:0/ANY (0) 0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
10.96.0.1:0/ANY (0) 0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
10.96.0.10:9153/TCP (1) 10.98.3.177:9153/TCP (3) (1)
10.96.0.10:53/UDP (0) 0.0.0.0:0 (2) (0) [ClusterIP, non-routable]
10.96.0.10:53/UDP (1) 10.98.3.177:53/UDP (2) (1)
10.96.78.63:80/TCP (3) 10.98.2.123:80/TCP (9) (3)
10.96.195.136:80/TCP (0) 0.0.0.0:0 (13) (0) [ClusterIP, non-routable]
10.96.0.10:53/UDP (2) 10.98.3.215:53/UDP (2) (2)
10.96.195.136:0/ANY (0) 0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
10.96.0.1:443/TCP (1) 172.18.0.5:6443/TCP (4) (1)
10.96.0.10:53/TCP (2) 10.98.3.215:53/TCP (1) (2)
10.96.0.10:53/TCP (1) 10.98.3.177:53/TCP (1) (1)
10.96.0.10:9153/TCP (0) 0.0.0.0:0 (3) (0) [ClusterIP, non-routable]
10.96.25.197:80/TCP (0) 0.0.0.0:0 (12) (0) [ClusterIP, non-routable]
10.96.0.10:0/ANY (0) 0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
10.96.64.225:443/TCP (0) 0.0.0.0:0 (5) (0) [ClusterIP, InternalLocal, non-routable]
10.96.25.197:80/TCP (3) 10.98.2.123:80/TCP (12) (3)
10.96.78.63:80/TCP (2) 10.98.1.50:80/TCP (9) (2)
10.96.195.136:80/TCP (3) 10.98.2.123:80/TCP (13) (3)
10.96.25.197:80/TCP (1) 10.98.0.201:80/TCP (12) (1)
请求 ClusterIP 结果
## 目前没有进行路由宣告，不知道怎么把请求转给其他 pod ip，所以请求失败
## 请求 3 次后，负载将 svc ip 改写为本节点 pod ip，所以请求成功了
# docker exec cilium-bgp-control-plane curl -m2 10.96.195.136
curl: (28) Connection timed out after 2001 milliseconds
# docker exec cilium-bgp-control-plane curl -m2 10.96.195.136
curl: (28) Connection timed out after 2001 milliseconds
# docker exec cilium-bgp-control-plane curl -m2 10.96.195.136
PodName: lb-ipam-bgx96 | PodIP: eth0 10.98.0.201/32
在 cilium-bgp-control-plane 节点通过 tcpdump 抓包查看效果（ClusterIP）
root@cilium-bgp-control-plane:/# tcpdump -nn -vvv -i any dst port 80
08:42:00.659301 net0 Out IP (tos 0x0, ttl 64, id 10240, offset 0, flags [DF], proto TCP (6), length 60)
10.1.5.10.1037 > 10.98.3.47.80: Flags [S], cksum 0xcebf (correct), seq 2633282226, win 64240, options [mss 1460,sackOK,TS val 4156233201 ecr 0,nop,wscale 10], length 0
08:42:01.705292 net0 Out IP (tos 0x0, ttl 64, id 10241, offset 0, flags [DF], proto TCP (6), length 60)
10.1.5.10.1037 > 10.98.3.47.80: Flags [S], cksum 0xcaa9 (correct), seq 2633282226, win 64240, options [mss 1460,sackOK,TS val 4156234247 ecr 0,nop,wscale 10], length 0
08:42:09.097702 net0 Out IP (tos 0x0, ttl 64, id 30290, offset 0, flags [DF], proto TCP (6), length 60)
10.1.5.10.54289 > 10.98.2.123.80: Flags [S], cksum 0x0917 (correct), seq 1322803575, win 64240, options [mss 1460,sackOK,TS val 2329352325 ecr 0,nop,wscale 10], length 0
08:42:10.153311 net0 Out IP (tos 0x0, ttl 64, id 30291, offset 0, flags [DF], proto TCP (6), length 60)
10.1.5.10.54289 > 10.98.2.123.80: Flags [S], cksum 0x04f7 (correct), seq 1322803575, win 64240, options [mss 1460,sackOK,TS val 2329353381 ecr 0,nop,wscale 10], length 0
08:42:18.092077 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47104, offset 0, flags [DF], proto TCP (6), length 60)
10.98.0.53.23267 > 10.98.0.201.80: Flags [S], cksum 0x15f0 (incorrect -> 0x1bd2), seq 2402818277, win 64240, options [mss 1460,sackOK,TS val 3786094204 ecr 0,nop,wscale 10], length 0
08:42:18.092143 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47105, offset 0, flags [DF], proto TCP (6), length 52)
10.98.0.53.23267 > 10.98.0.201.80: Flags [.], cksum 0x15e8 (incorrect -> 0x1fcf), seq 2402818278, ack 3172350725, win 63, options [nop,nop,TS val 3786094204 ecr 3115938718], length 0
08:42:18.092216 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47106, offset 0, flags [DF], proto TCP (6), length 129)
10.98.0.53.23267 > 10.98.0.201.80: Flags [P.], cksum 0x1635 (incorrect -> 0x8b5e), seq 0:77, ack 1, win 63, options [nop,nop,TS val 3786094204 ecr 3115938718], length 77: HTTP, length: 77
GET / HTTP/1.1
Host: 10.96.195.136
User-Agent: curl/8.14.1
Accept: */*
08:42:18.092381 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47107, offset 0, flags [DF], proto TCP (6), length 52)
10.98.0.53.23267 > 10.98.0.201.80: Flags [.], cksum 0x15e8 (incorrect -> 0x1e94), seq 77, ack 237, win 63, options [nop,nop,TS val 3786094205 ecr 3115938719], length 0
08:42:18.092428 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47108, offset 0, flags [DF], proto TCP (6), length 52)
10.98.0.53.23267 > 10.98.0.201.80: Flags [.], cksum 0x15e8 (incorrect -> 0x1e60), seq 77, ack 289, win 63, options [nop,nop,TS val 3786094205 ecr 3115938719], length 0
08:42:18.092649 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47109, offset 0, flags [DF], proto TCP (6), length 52)
10.98.0.53.23267 > 10.98.0.201.80: Flags [F.], cksum 0x15e8 (incorrect -> 0x1e5f), seq 77, ack 289, win 63, options [nop,nop,TS val 3786094205 ecr 3115938719], length 0
08:42:18.092773 lxc4d2da8b981aa Out IP (tos 0x0, ttl 63, id 47110, offset 0, flags [DF], proto TCP (6), length 52)
10.98.0.53.23267 > 10.98.0.201.80: Flags [.], cksum 0x15e8 (incorrect -> 0x1e5e), seq 78, ack 290, win 63, options [nop,nop,TS val 3786094205 ecr 3115938719], length 0
- NodePort：现象、原因与 ClusterIP 一致
# docker exec cilium-bgp-control-plane curl -m2 10.1.5.11:32000
curl: (28) Connection timed out after 2002 milliseconds
# docker exec cilium-bgp-control-plane curl -m2 10.1.5.11:32000
curl: (28) Connection timed out after 2001 milliseconds
# docker exec cilium-bgp-control-plane curl -m2 10.1.5.11:32000
PodName: lb-ipam-f6gdv | PodIP: eth0 10.98.1.50/32
- LoadBalancer：请求 CLUSTER-IP 时现象、原因与 ClusterIP 一致，EXTERNAL-IP 没有路由宣告自然也是不通
## 没有进行路由宣告，leaf spine 路由器不知道如何转发改地址，所以请求失败了
# curl -m2 40.0.10.0
curl: (28) Connection timed out after 2000 milliseconds
2.2.3. 创建 Service/Pod 路由通告
进行路由通告后，路由器已识别各 Pod/Service 网段
BGP 路由通告配置
## 两个 rack 的 peers 都引用它
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
name: cilium-bgp-peer-config
spec:
gracefulRestart:
enabled: true
restartTimeSeconds: 120
## 只协商 IPv4 unicast
## 不写 families 则默认同时协商 IPv4/6
families:
- afi: ipv4
safi: unicast
## 订阅哪些 CiliumBGPAdvertisement
advertisements:
matchLabels:
advertise: bgp
---
## 宣告内容：PodCIDR + 全部 LoadBalancer Service VIP
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
name: bgp-advertisements
## 被 CiliumBGPPeerConfig 匹配
labels:
advertise: bgp
spec:
advertisements:
## 宣告所有 Pod CIDR
- advertisementType: "PodCIDR"
## 宣告特定类型的 Service
- advertisementType: "Service"
service:
addresses:
- LoadBalancerIP
selector:
matchExpressions:
- {key: wluo, operator: NotIn, values: ["wluo"]}
---
## rack0：本端 ASN 65005，对端 10.1.5.1
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
name: cilium-bgp-rack0
spec:
nodeSelector:
matchLabels:
rack: rack0
bgpInstances:
- name: "instance-65005"
localASN: 65005
peers:
- name: "peer-leaf0"
peerASN: 65005
peerAddress: "10.1.5.1"
peerConfigRef:
name: cilium-bgp-peer-config
---
## rack1：本端 ASN 65008，对端 10.1.8.1
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
name: cilium-bgp-rack1
spec:
nodeSelector:
matchLabels:
rack: rack1
bgpInstances:
- name: "instance-65008"
localASN: 65008
peers:
- name: "peer-leaf1"
peerASN: 65008
peerAddress: "10.1.8.1"
peerConfigRef:
name: cilium-bgp-peer-config
2.2.4. 验证 BGP 路由通告
查询 leaf 识别的 BGP 信息
# docker exec -it clab-cilium-bgp-leaf0 bash
root@leaf0:/# show ip route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 1d06h55m
C>* 10.1.5.0/24 is directly connected, eth3, 1d06h55m
B>* 10.1.8.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 1d06h55m
* via 10.1.12.2, eth2, weight 1, 1d06h55m
C>* 10.1.10.0/24 is directly connected, eth1, 1d06h55m
B>* 10.1.11.0/24 [20/0] via 10.1.12.2, eth2, weight 1, 1d06h55m
C>* 10.1.12.0/24 is directly connected, eth2, 1d06h55m
B>* 10.1.34.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 1d06h55m
B>* 10.98.0.0/24 [200/0] via 10.1.5.10, eth3, weight 1, 00:02:33
B>* 10.98.1.0/24 [200/0] via 10.1.5.11, eth3, weight 1, 00:02:33
B>* 10.98.2.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 00:02:33
* via 10.1.12.2, eth2, weight 1, 00:02:33
B>* 10.98.3.0/24 [20/0] via 10.1.10.2, eth1, weight 1, 00:02:33
* via 10.1.12.2, eth2, weight 1, 00:02:33
B>* 40.0.10.0/32 [200/0] via 10.1.5.10, eth3, weight 1, 00:02:33
* via 10.1.5.11, eth3, weight 1, 00:02:33
C>* 172.20.20.0/24 is directly connected, eth0, 1d06h55m
root@leaf0:/# ip route show
default via 172.20.20.1 dev eth0
10.1.5.0/24 dev eth3 proto kernel scope link src 10.1.5.1
10.1.8.0/24 nhid 30 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.1.10.0/24 dev eth1 proto kernel scope link src 10.1.10.1
10.1.11.0/24 nhid 26 via 10.1.12.2 dev eth2 proto bgp metric 20
10.1.12.0/24 dev eth2 proto kernel scope link src 10.1.12.1
10.1.34.0/24 nhid 25 via 10.1.10.2 dev eth1 proto bgp metric 20
10.98.0.0/24 nhid 49 via 10.1.5.10 dev eth3 proto bgp metric 20
10.98.1.0/24 nhid 47 via 10.1.5.11 dev eth3 proto bgp metric 20
10.98.2.0/24 nhid 30 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.98.3.0/24 nhid 30 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
40.0.10.0 nhid 48 proto bgp metric 20
nexthop via 10.1.5.11 dev eth3 weight 1
nexthop via 10.1.5.10 dev eth3 weight 1
172.20.20.0/24 dev eth0 proto kernel scope link src 172.20.20.2
请求 ClusterIP 后端 4 个节点上的 Pod 均有响应
# docker exec cilium-bgp-control-plane curl -s 10.96.195.136
PodName: lb-ipam-f6gdv | PodIP: eth0 10.98.1.50/32
# docker exec cilium-bgp-control-plane curl -s 10.96.195.136
PodName: lb-ipam-wmm6s | PodIP: eth0 10.98.2.123/32
# docker exec cilium-bgp-control-plane curl -s 10.96.195.136
PodName: lb-ipam-zw6p8 | PodIP: eth0 10.98.3.47/32
# docker exec cilium-bgp-control-plane curl -s 10.96.195.136
PodName: lb-ipam-bgx96 | PodIP: eth0 10.98.0.201/32
请求 LoadBalancer 后端 4 个节点上的 Pod 均有响应
# docker exec cilium-bgp-control-plane curl -s 40.0.10.0
PodName: lb-ipam-f6gdv | PodIP: eth0 10.98.1.50/32
# docker exec cilium-bgp-control-plane curl -s 40.0.10.0
PodName: lb-ipam-wmm6s | PodIP: eth0 10.98.2.123/32
# docker exec cilium-bgp-control-plane curl -s 40.0.10.0
PodName: lb-ipam-zw6p8 | PodIP: eth0 10.98.3.47/32
# docker exec cilium-bgp-control-plane curl -s 40.0.10.0
PodName: lb-ipam-bgx96 | PodIP: eth0 10.98.0.201/32
浙公网安备 33010602011771号