浅谈 Docker 网络：单节点多容器
1.同网段多容器访问
这一节将对 Docker 多容器网络进行讨论，构建容器网络示意图如下：
创建容器 demo0 和 demo1:
[root@lianhua ~]$ docker run -it --name demo1 httpd [root@lianhua ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 39a303a4f993 httpd "/bin/bash" About a minute ago Up 2 seconds demo1 7be09e54b24c httpd "/bin/bash" 13 hours ago Up 13 hours demo0
查看容器的网络配置信息：
[root@lianhua ~]$ docker inspect bridge [ { "Name": "bridge", "Id": "1a779d0e62d5a309e1e942862b76d69d4ba9ed9be9c7bcdc051e8de89b0cc3ee", "Created": "2020-08-26T00:06:03.910196776+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Containers": { "39a303a4f99308e60f7d53a7d471276ce249cc7cf88194d72e2e717cd7533354": { "Name": "demo1", "EndpointID": "c8c8eeccba33d7fedc09bfca26c0abedcbe53d4ee941fdef29c2e08498f67237", "MacAddress": "02:42:ac:11:00:03", "IPv4Address": "172.17.0.3/16", "IPv6Address": "" }, "7be09e54b24c45100769e131b46259c519710785ccfb68afaa904a1114add9a1": { "Name": "demo0", "EndpointID": "98399b3c0560aac4ca63de9f79659176562406ac02d917c667852b9a863296bb", "MacAddress": "02:42:ac:11:00:02", "IPv4Address": "172.17.0.2/16", "IPv6Address": "" } }, } ]
容器间互相访问：
[root@lianhua ~]$ docker exec -it demo0 /bin/bash bash-4.2$ ping 172.17.0.3 -c 3 PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data. 64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.063 ms 64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.047 ms 64 bytes from 172.17.0.3: icmp_seq=3 ttl=64 time=0.044 ms --- 172.17.0.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.044/0.051/0.063/0.010 ms [root@lianhua ~]$ tcpdump -i docker0 -n icmp -vv tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:00:04.074479 IP (tos 0x0, ttl 64, id 44606, offset 0, flags [DF], proto ICMP (1), length 84) 172.17.0.2 > 172.17.0.3: ICMP echo request, id 22, seq 1, length 64 11:00:04.074518 IP (tos 0x0, ttl 64, id 9717, offset 0, flags [none], proto ICMP (1), length 84) 172.17.0.3 > 172.17.0.2: ICMP echo reply, id 22, seq 1, length 64 11:00:05.074188 IP (tos 0x0, ttl 64, id 45567, offset 0, flags [DF], proto ICMP (1), length 84) 172.17.0.2 > 172.17.0.3: ICMP echo request, id 22, seq 2, length 64
容器 demo0 可以访问到 demo1，在网桥 docker0 上抓到 ICMP 包。
进一步查看路由表我们发现这样一条规则：
[root@lianhua ~]$ iptables -S FORWARD
-A FORWARD -i docker0 -o docker0 -j ACCEPT
该规则定义了连在 docker0 网桥上的容器可以相互访问。那么，如果将 ACCEPT 改成 DROP 会不会连在 docker0 上的容器相互间无法访问呢？我们改写规则如下：
[root@lianhua ~]$ iptables -A FORWARD -i docker0 -o docker0 -j DROP [root@lianhua ~]$ iptables -D FORWARD 19 # 19 是 -A FORWARD -i docker0 -o docker0 -j ACCEPT 这条规则 [root@lianhua ~]$ iptables -S FORWARD -A FORWARD -i docker0 -o docker0 -j DROP
规则改写好了，相互 ping 看能否访问：
[root@lianhua ~]$ docker exec -it demo0 /bin/bash bash-4.2$ ping 172.17.0.3 -c 3 --- 172.17.0.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.044/0.051/0.063/0.010 ms bash-4.2$ ping 172.17.0.3 -c 3 PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data. --- 172.17.0.3 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms [root@lianhua ~]$ tcpdump -i docker0 -n icmp -vv tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:07:44.129449 IP (tos 0x0, ttl 64, id 44013, offset 0, flags [DF], proto ICMP (1), length 84) 172.17.0.2 > 172.17.0.3: ICMP echo request, id 23, seq 1, length 64 11:07:45.129195 IP (tos 0x0, ttl 64, id 44650, offset 0, flags [DF], proto ICMP (1), length 84) 172.17.0.2 > 172.17.0.3: ICMP echo request, id 23, seq 2, length 64 11:07:46.129186 IP (tos 0x0, ttl 64, id 45023, offset 0, flags [DF], proto ICMP (1), length 84) 172.17.0.2 > 172.17.0.3: ICMP echo request, id 23, seq 3, length 64
和预想的一样，容器 demo0 无法访问 demo1，ICMP 包只有去往 demo1 的，没有返回的 ICMP 包。
（当 DROP 掉容器在网桥之间互通时，可通过在容器间建立 link 通信实现容器互通）
2.不同网段多容器访问
那么单节点上不同网段的多容器互相之间能否访问呢？基于此设想构建多容器示意图如下：
bridge1 和 bridge2 是自建的 docker 网络，它使用的驱动类型是网桥驱动。test1 和 test2 是连到 bridge1 和 bridge2 的容器。
创建 bridge1 和 bridge2 网络：
[root@lianhua ~]$ docker network ls NETWORK ID NAME DRIVER SCOPE 1a779d0e62d5 bridge bridge local a7669a03d994 bridge1 bridge local bafee5b9a6d8 bridge2 bridge local [root@lianhua ~]$ docker inspect bridge1 [ { "Name": "bridge1", "Id": "a7669a03d99463b6bdbca3852a0c9de992e3d1bc66ecfba7e87aa356ca9ed2c5", "Created": "2021-01-03T15:59:48.979432719+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "172.21.0.0/16", "Gateway": "172.21.0.1" } ] }, ... } ]
[root@lianhua ~]$ docker inspect bridge2 [ { "Name": "bridge2", "Id": "bafee5b9a6d890a2924081146d43fb3a378548e417f595a3a8f5239941178343", "Created": "2021-01-03T16:30:47.6901238+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "172.22.0.0/16", "Gateway": "172.22.0.1" } ] }, ... } ]
show bridge1 和 bridge2 对应的网桥分别为 br-a7669a03d994 和 br-bafee5b9a6d8。
创建容器 test1 和 test2：
[root@lianhua ~]$ docker run -it --name test1 --network bridge1 httpd [root@lianhua ~]$ docker run -it --name test2 --network bridge2 httpd [root@lianhua ~]$ docker inspect bridge1 [ ... { "Containers": { "861089a681834b16148c36497f689aaaf58fff0937f03bf232cad5bac10bc599": { "Name": "test1", "EndpointID": "11116435d4a393a8c6673e0ecd1f021729b2d7f652ec1dad088710c7071ce1d9", "MacAddress": "02:42:ac:15:00:02", "IPv4Address": "172.21.0.2/16", "IPv6Address": "" } }, } ] [root@lianhua ~]$ docker inspect bridge2 [ ... { "Containers": { "8fd92d473e7b6d30144229bb348f877c7a4dab2fd6285c1c633305d4fe058b35": { "Name": "test2", "EndpointID": "4bf82b73159c0f86c536d23f7d456dc10bcaee7ec079baf5039966129a0ab816", "MacAddress": "02:42:ac:16:00:02", "IPv4Address": "172.22.0.2/16", "IPv6Address": "" } }, } ]
网络 bridge1 和 bridge2 的子网分别是 172.21.0.0/16 和 172.22.0.0/16，容器 test1 和 test2 分到的 ip 地址是 172.21.0.2 和 172.22.0.2，它们位于不同网段。
检查路由和 ip_forwarding：
[root@lianhua ~]$ ip route default via 192.168.0.1 dev eth0 169.254.169.254 via 192.168.0.1 dev eth0 proto static 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.21.0.0/16 dev br-a7669a03d994 proto kernel scope link src 172.21.0.1 172.22.0.0/16 dev br-bafee5b9a6d8 proto kernel scope link src 172.22.0.1 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.69 [root@lianhua ~]$ cat /proc/sys/net/ipv4/ip_forward 1
路由和 ip_forwarding 是打开的，进到容器 test1 访问 test2 看看结果:
bash-4.2$ ping 172.22.0.1 PING 172.22.0.1 (172.22.0.1) 56(84) bytes of data. 64 bytes from 172.22.0.1: icmp_seq=1 ttl=64 time=0.063 ms 64 bytes from 172.22.0.1: icmp_seq=2 ttl=64 time=0.039 ms 64 bytes from 172.22.0.1: icmp_seq=3 ttl=64 time=0.036 ms bash-4.2$ ping 172.22.0.2 PING 172.22.0.2 (172.22.0.2) 56(84) bytes of data. --- 172.22.0.2 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 3999ms [root@lianhua ~]$ tcpdump -i br-a7669a03d994 -n icmp -vv tcpdump: listening on br-a7669a03d994, link-type EN10MB (Ethernet), capture size 262144 bytes 11:42:39.537025 IP (tos 0x0, ttl 64, id 23242, offset 0, flags [DF], proto ICMP (1), length 84) 172.21.0.2 > 172.22.0.1: ICMP echo request, id 23, seq 1, length 64 11:42:39.537063 IP (tos 0x0, ttl 64, id 7129, offset 0, flags [none], proto ICMP (1), length 84) 172.22.0.1 > 172.21.0.2: ICMP echo reply, id 23, seq 1, length 64 11:42:40.536198 IP (tos 0x0, ttl 64, id 23603, offset 0, flags [DF], proto ICMP (1), length 84) 172.21.0.2 > 172.22.0.1: ICMP echo request, id 23, seq 2, length 64 11:42:40.536220 IP (tos 0x0, ttl 64, id 8032, offset 0, flags [none], proto ICMP (1), length 84) [root@lianhua ~]$ tcpdump -i br-a7669a03d994 -n icmp -vv tcpdump: listening on br-a7669a03d994, link-type EN10MB (Ethernet), capture size 262144 bytes 11:42:54.974063 IP (tos 0x0, ttl 64, id 58344, offset 0, flags [DF], proto ICMP (1), length 84) 172.21.0.2 > 172.22.0.2: ICMP echo request, id 24, seq 1, length 64 11:42:55.973223 IP (tos 0x0, ttl 64, id 58388, offset 0, flags [DF], proto ICMP (1), length 84) 172.21.0.2 > 172.22.0.2: ICMP echo request, id 24, seq 2, length 64 11:42:56.973230 IP (tos 0x0, ttl 64, id 58667, offset 0, flags [DF], proto ICMP (1), length 84) 172.21.0.2 > 172.22.0.2: ICMP echo request, id 24, seq 3, length 64 11:42:57.973190 IP (tos 0x0, ttl 64, id 58989, offset 0, flags [DF], proto ICMP (1), length 84)
奇怪，容器 test1 可以访问到 test2 的网关，但是却访问不了 test2。查看路由表，我们找到了这条规则：
-A DOCKER-ISOLATION-STAGE-1 -i br-9c9714b1da04 ! -o br-9c9714b1da04 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -i br-c925ef6b9064 ! -o br-c925ef6b9064 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-2 -o br-9c9714b1da04 -j DROP -A DOCKER-ISOLATION-STAGE-2 -o br-c925ef6b9064 -j DROP
从网桥 br-9c9714b1da04 发出的包，如果目的接口不是 br-9c9714b1da04， 则进入规则 DOCKER-ISOLATION-STAGE-2。 而规则 DOCKER-ISOLATION-STAGE-2 的作用是将到网桥 br-9c9714b1da04 和 br-c925ef6b9064 的包丢掉。看到这里我们怀疑 test1 到 test2 的包是因为进入了规则 DOCKER-ISOLATION-STAGE-2 而被丢掉了。
改写 DOCKER-ISOLATION-STAGE-2 规则如下：
[root@lianhua ~]$ iptables -S DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-2 -o br-bafee5b9a6d8 -j ACCEPT -A DOCKER-ISOLATION-STAGE-2 -o br-a7669a03d994 -j ACCEPT
再一次从 test1 ping test2：
bash-4.2$ ping 172.22.0.2 PING 172.22.0.2 (172.22.0.2) 56(84) bytes of data. 64 bytes from 172.22.0.2: icmp_seq=1 ttl=63 time=0.091 ms 64 bytes from 172.22.0.2: icmp_seq=2 ttl=63 time=0.034 ms ^C --- 172.22.0.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.034/0.062/0.091/0.029 ms
修改规则之后 test1 和 test2 可以互相访问，验证了我们的猜测。
（那么问题来了，未修改规则之前，为什么访问 test2 的网关却是可以通的呢？通过上面的路由表规则我们知道访问网关并没有进入规则 DOCKER-ISOLATION-STAGE-2（不然包会被丢掉），所以我们合理怀疑网关并没有建在网桥上。（有知道的童鞋欢迎指出来这个点！））
2.1 多网口容器
除了直接修改路由表实现不同网段容器相互访问的方式之外，也可以通过在容器中添加另一张网络使得容器成为多网口容器。示意图如下：
将 test1 接入到网络 bridge2 上：
[root@lianhua ~]$ docker network connect bridge2 test1 [root@lianhua ~]$ docker inspect bridge2 [ { "Name": "bridge2", ... "Containers": { "459df1132c4b82d8bbde24ecd253d27bb514e3befd9d0c0843aa57257ea08c01": { "Name": "test2", "EndpointID": "3cd99fae65fbbece78935d5369d3972a13300dd00439ba284793de13b185e530", "MacAddress": "02:42:ac:1a:00:02", "IPv4Address": "172.26.0.2/16", "IPv6Address": "" }, "94a3abcf7e5454b928a562fcf6ebfd33f352d65f1971aee4fb237de0526d16db": { "Name": "test1", "EndpointID": "8c919123ef44c0b86d2843ab291649ae9283639ec4f02f869efdd1ef73becf18", "MacAddress": "02:42:ac:1a:00:03", "IPv4Address": "172.26.0.3/16", "IPv6Address": "" } }, } ]
可以看到 test1 中有两个网口，分别配了 bridge1 和 bridge2 的 ip。进入 test1 访问容器 test2：
bash-4.2$ ping 172.26.0.2 PING 172.26.0.2 (172.26.0.2) 56(84) bytes of data. 64 bytes from 172.26.0.2: icmp_seq=1 ttl=64 time=0.057 ms 64 bytes from 172.26.0.2: icmp_seq=2 ttl=64 time=0.035 ms 64 bytes from 172.26.0.2: icmp_seq=3 ttl=64 time=0.033 ms 64 bytes from 172.26.0.2: icmp_seq=4 ttl=64 time=0.029 ms 64 bytes from 172.26.0.2: icmp_seq=5 ttl=64 time=0.027 ms 64 bytes from 172.26.0.2: icmp_seq=6 ttl=64 time=0.025 ms [root@lianhua ~]$ tcpdump -i br-9c9714b1da04 -n icmp -vv tcpdump: listening on br-9c9714b1da04, link-type EN10MB (Ethernet), capture size 262144 bytes 23:26:36.616194 IP (tos 0x0, ttl 64, id 4328, offset 0, flags [DF], proto ICMP (1), length 84) 172.26.0.3 > 172.26.0.2: ICMP echo request, id 34, seq 11, length 64 23:26:36.616221 IP (tos 0x0, ttl 64, id 29472, offset 0, flags [none], proto ICMP (1), length 84) 172.26.0.2 > 172.26.0.3: ICMP echo reply, id 34, seq 11, length 64 23:26:37.616189 IP (tos 0x0, ttl 64, id 4964, offset 0, flags [DF], proto ICMP (1), length 84) 172.26.0.3 > 172.26.0.2: ICMP echo request, id 34, seq 12, length 64 23:26:37.616213 IP (tos 0x0, ttl 64, id 29627, offset 0, flags [none], proto ICMP (1), length 84)
test1 通过 172.26.0.3 的网口 ip test2，实现了不同网段多容器的访问。
芝兰生于空谷，不以无人而不芳。
