kube-proxy ipvs calico
1. 关闭防火墙
#/etc/selinux/config SELINUX=disabled
#sestatus
SELinux status: disabled
#systemctl stop firewalld
#systemctl disable firewalld
2. 修改系统参数
# /etc/sysctl.conf net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 # sysctl -p
3. 安装Ipvsadm
yum -y install ipvsadm ipset # lsmod | grep ip_vs ip_vs_sh 12688 0 ip_vs_wrr 12697 0 ip_vs_rr 12600 27 ip_vs 141092 33 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 133387 9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6 libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
安装所需模块
modprobe -a ip_set ip_tables ip6_tables ipt_REJECT ipt_rpfilter ipt_set nf_conntrack_netlink nf_conntrack_proto_sctp sctp xt_addrtype xt_comment xt_conntrack xt_icmp xt_icmp6 xt_ipvs xt_mark xt_multiport xt_rpfilter xt_sctp xt_set xt_u32 ipip
其中有几个不存在
modprobe: WARNING: Module nf_conntrack_proto_sctp not found.
modprobe: WARNING: Module xt_icmp not found.
modprobe: WARNING: Module xt_icmp6 not found.
modprobe: WARNING: Module xt_rpfilter not found.
当前环境使用缺失以上mod可以运行，先不管了
如果使用ipip 还要确认ipip模块
modprobe ipip
ip link set dev tunl0 up
启动calico后查看tunl0网卡的ip地址
如果没有分配地址，需要重新删除calico,清理calico存储尤其是ippool
直到分配地址
tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
inet 172.18.39.0/32 scope global tunl0
valid_lft forever preferred_lft forever
如果模块没有加载
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF # chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4 上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。
4. 修改kubelet配置
(新版已经移除了相关参数，使用默认的kube-proxy镜像提交daemonset即可)
KUBE_PROXY_ARGS="--bind-address=10.10.1.8 \
--hostname-override=docker4.node \
--masquerade-all \
--feature-gates=SupportIPVSProxyMode=true \
--proxy-mode=ipvs \
--ipvs-min-sync-period=5s \
--ipvs-sync-period=5s \
--ipvs-scheduler=rr \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
--cluster-cidr=10.254.0.0/16"
5. calico 配置 port range (没有指定的可以跳过)
calico.yaml
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "172.18.0.0/16"
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
# Set Felix endpoint to host default action to ACCEPT.
- name: FELIX_DEFAULTENDPOINTTOHOSTACTION
value: "ACCEPT"
# Disable IPv6 on Kubernetes.
- name: FELIX_IPV6SUPPORT
value: "false"
# Set Felix logging to "info"
- name: FELIX_LOGSEVERITYSCREEN
value: "info"
- name: FELIX_HEALTHENABLED
value: "true"
# Node port range while kube-proxy in ipvs mode
- name: FELIX_KUBENODEPORTRANGES
value: "10000:11000"
注意事项 ：
1.calico会自动选择网卡，这里要看calico-node的日志确定是否是需要的网卡。
2.calico的 CIDR不要与clusterip和hostip重叠，否则网络不通。
3. namespaceSelector,podSelector 与ingress-nginx 的namespace不可用的情况，考虑是不是用了hostNetwork
https://superuser.com/questions/1481901/networkpolicy-has-no-effect-on-nginx-ingress-namespace
参考资料
https://docs.projectcalico.org/networking/use-ipvs
https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises
https://blog.51cto.com/1000682/2362853
https://www.kubernetes.org.cn/3025.html
