kubernetes的没有metallb服务 基于haproxy Ingress创建管理 及访问流程

 

 

 

没有metallb服务 自己部署一个外层负载均衡器

 

51f720e1e55c62e2c4ea9ad27279e403

 

[root@master1 ~]# kubectl get node
NAME          STATUS   ROLES           AGE   VERSION
master1.org   Ready    control-plane   16d   v1.35.4
node1.org     Ready    <none>          16d   v1.35.4
node2.org     Ready    <none>          16d   v1.35.4
node3.org     Ready    <none>          16d   v1.35.4

 

 

kubectl create deployment pod-test1 --image=registry.cn-beijing.aliyuncs.com/wangxiaochun/pod-test:v0.1 --replicas=3

kubectl create service clusterip pod-test1 --tcp=80:80

kubectl create deployment pod-test2 --image=registry.cn-beijing.aliyuncs.com/wangxiaochun/pod-test:v0.2 --replicas=3

kubectl create service clusterip pod-test2 --tcp=80:80

[root@master1 workload]# kubectl get pod
NAME                         READY   STATUS    RESTARTS   AGE
pod-test1-5d64bc66d7-cxkc6   1/1     Running   0          80s
pod-test1-5d64bc66d7-hxk9j   1/1     Running   0          80s
pod-test1-5d64bc66d7-wnkwd   1/1     Running   0          80s
pod-test2-5bdb495689-5s5bt   1/1     Running   0          5s
pod-test2-5bdb495689-cb2vc   1/1     Running   0          5s
pod-test2-5bdb495689-nbgnt   1/1     Running   0          5s

 

[root@master1 workload]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   15d
pod-test1    ClusterIP   10.105.58.246   <none>        80/TCP    3m30s
pod-test2    ClusterIP   10.106.88.86    <none>        80/TCP    4s

 

[root@master1 workload]# kubectl get endpoints
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME         ENDPOINTS                                         AGE
kubernetes   192.168.3.60:6443                                 15d
pod-test1    10.244.1.215:80,10.244.2.212:80,10.244.2.213:80   3m54s
pod-test2    10.244.1.216:80,10.244.2.214:80,10.244.2.215:80   28s

kubectl create ingress demo-ingress --rule="www.ming.org/=pod-test1:80" --class=nginx -o yaml --dry-run=client > ingress-demo1.yaml

kubectl apply -f ingress-demo1.yaml

 

[root@master1 ingress]# kubectl get pod  -o wide                
NAME                         READY   STATUS    RESTARTS      AGE    IP             NODE        NOMINATED NODE   READINESS GATES
pod-test1-5d64bc66d7-cxkc6   1/1     Running   4 (43m ago)   134m   10.244.1.241   node1.org   <none>           <none>
pod-test1-5d64bc66d7-hxk9j   1/1     Running   5 (43m ago)   134m   10.244.2.245   node2.org   <none>           <none>
pod-test1-5d64bc66d7-wnkwd   1/1     Running   5 (43m ago)   134m   10.244.2.241   node2.org   <none>           <none>
pod-test2-5bdb495689-5s5bt   1/1     Running   5 (43m ago)   133m   10.244.2.243   node2.org   <none>           <none>
pod-test2-5bdb495689-cb2vc   1/1     Running   4 (43m ago)   133m   10.244.1.244   node1.org   <none>           <none>
pod-test2-5bdb495689-nbgnt   1/1     Running   5 (43m ago)   133m   10.244.2.242   node2.org   <none>           <none>

 

 

[root@master1 ~]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.97.174.247   192.168.3.10   80:32708/TCP,443:30309/TCP   12h
ingress-nginx-controller-admission   ClusterIP      10.111.133.50   <none>         443/TCP                      12h

 

[root@master1 ~]# kubectl get pod  -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS      AGE   IP             NODE        NOMINATED NODE   READINESS GATES
ingress-nginx-controller-65c7957ccd-27xqp   1/1     Running   1 (12h ago)   12h   10.244.1.242   node1.org   <none>           <none>
ingress-nginx-controller-65c7957ccd-st976   1/1     Running   0             12h   10.244.2.246   node2.org   <none>           <none>

 

node3节点访问不了，因为没有pod

C:\Users\Administrator>curl -H"host:www.ming.org" 192.168.3.63:32708
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-wnkwd, ServerIP: 10.244.2.241!

C:\Users\Administrator>curl -H"host:www.ming.org" 192.168.3.64:32708
kubernetes pod-test v0.1!! ClientIP: 10.244.2.246, ServerName: pod-test1-5d64bc66d7-cxkc6, ServerIP: 10.244.1.241!

C:\Users\Administrator>curl -H"host:www.ming.org" 192.168.3.65:32708
curl: (28) Failed to connect to 192.168.3.65 port 32708 after 21004 ms: Could not connect to server

 

 

ingress-nginx-controller service默认externalTrafficPolicy 策略是Local 访问宿主机30000+端口nodeport模式 必须访问有这个pod所在节点

externalTrafficPolicy 策略是Local  这种要知道node节点的ip地址

 

ingress-nginx-controller service默认externalTrafficPolicy 策略是Local 改成 Cluster

kubectl get svc -n ingress-nginx ingress-nginx-controller -o yaml

externalTrafficPolicy: Local

 

kubectl edit svc -n ingress-nginx ingress-nginx-controller 

 

 

 creationTimestamp: "2026-05-25T18:52:33Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.15.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "389848"
  uid: fe4d3184-9c39-48e7-b11c-a6e7ed07830a
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.97.174.247
  clusterIPs:
  - 10.97.174.247
  externalTrafficPolicy: Local
  healthCheckNodePort: 32452
  internalTrafficPolicy: Cluster
  ipFamilies:

 

 

   metallb.io/ip-allocated-from-pool: localip-pool
  creationTimestamp: "2026-05-25T18:52:33Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.15.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "389848"
  uid: fe4d3184-9c39-48e7-b11c-a6e7ed07830a
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.97.174.247
  clusterIPs:
  - 10.97.174.247
  externalTrafficPolicy: Cluster
  healthCheckNodePort: 32452
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4

 

每个节点都可以访问

C:\Users\Administrator>curl -H"host:www.ming.org" 192.168.3.65:32708
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-cxkc6, ServerIP: 10.244.1.241!

C:\Users\Administrator>curl -H"host:www.ming.org" 192.168.3.64:32708
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-hxk9j, ServerIP: 10.244.2.245!

C:\Users\Administrator>curl -H"host:www.ming.org" 192.168.3.63:32708
kubernetes pod-test v0.1!! ClientIP: 10.244.2.246, ServerName: pod-test1-5d64bc66d7-hxk9j, ServerIP: 10.244.2.245!

 

 用户不想通过30000+端口访问 ，部署一个第三方 集群外负载均衡器访问

 

两个ip 192.168.3.66， 172.20.100.39 对外vip是172.20.100.39  对集群内192.168.3.66

[root@ha1 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.66  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::20c:29ff:feaa:6281  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:aa:62:81  txqueuelen 1000  (Ethernet)
        RX packets 127  bytes 12494 (12.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 127  bytes 13635 (13.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.100.39  netmask 255.255.0.0  broadcast 172.20.255.255
        inet6 fe80::250:56ff:fe3f:982e  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:3f:98:2e  txqueuelen 1000  (Ethernet)
        RX packets 14  bytes 840 (840.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 836 (836.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

写死ingress-nginx-controller 30000端口 修改 写死service nodeport端口  

vim deploy.yaml 

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.15.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https

 

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.15.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: 30080
  - appProtocol: https
    name: https

externalTrafficPolicy 策略也改成Cluster

kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.15.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: 30080
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:

 

 

kubectl apply -f deploy.yaml

固定端口了

[root@master1 ingress-nginx]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.97.174.247   192.168.3.10   80:30080/TCP,443:30309/TCP   13h
ingress-nginx-controller-admission   ClusterIP      10.111.133.50   <none>         443/TCP                      13h
[root@master1 ingress-nginx]#

 

 

 

 配置haproxy 反向代理

#基于haproxy实现集群外访问

dnf install -y haproxy

 

#修改Haproxy的配置

vim /etc/haproxy/haproxy.cfg 

[root@ha1 ~]# cat /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    maxconn 4000
    user haproxy
    group haproxy
    daemon

defaults
    mode tcp
    log global
    option tcplog
    option dontlognull
    timeout connect 5s
    timeout client 50s
    timeout server 50s

listen stats
    mode http
    bind 0.0.0.0:8888
    stats enable
    log global
    stats uri /status
    stats auth admin:123456

listen myapp
   bind 0.0.0.0:80
   mode tcp
   #bind 172.20.100.39:80 #eth1网卡的IP
   server node1 192.168.3.63:30080 check
   server node2 192.168.3.64:30080 check

 

systemctl restart haproxy

[root@ha1 ~]# netstat -tnpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      1225/haproxy        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1225/haproxy        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      639/sshd: /usr/sbin 
tcp6       0      0 :::22                   :::*                    LISTEN      639/sshd: /usr/sbin

 

http://172.20.100.39:8888/status

c7954ec0b699c5ed76cde335ffb7e0c7

 

可以访问

C:\Users\Administrator>curl -H"host:www.ming.org" http://172.20.100.39/
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-cxkc6, ServerIP: 10.244.1.241!

C:\Users\Administrator>curl -H"host:www.ming.org" http://172.20.100.39/
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-hxk9j, ServerIP: 10.244.2.245!

C:\Users\Administrator>curl -H"host:www.ming.org" http://172.20.100.39/
kubernetes pod-test v0.1!! ClientIP: 10.244.2.246, ServerName: pod-test1-5d64bc66d7-hxk9j, ServerIP: 10.244.2.245!

C:\Users\Administrator>curl -H"host:www.ming.org" http://172.20.100.39/
kubernetes pod-test v0.1!! ClientIP: 10.244.2.246, ServerName: pod-test1-5d64bc66d7-hxk9j, ServerIP: 10.244.2.245!

C:\Users\Administrator>curl -H"host:www.ming.org" http://172.20.100.39/
kubernetes pod-test v0.1!! ClientIP: 10.244.2.246, ServerName: pod-test1-5d64bc66d7-wnkwd, ServerIP: 10.244.2.241!

C:\Users\Administrator>curl -H"host:www.ming.org" http://172.20.100.39/
kubernetes pod-test v0.1!! ClientIP: 10.244.2.246, ServerName: pod-test1-5d64bc66d7-cxkc6, ServerIP: 10.244.1.241!

 

172.20.100.39 www.ming.org 做域名解析

194de798de4276b1c79a413377d48b22

 

C:\Users\Administrator>curl www.ming.org
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-wnkwd, ServerIP: 10.244.2.241!

C:\Users\Administrator>curl www.ming.org
kubernetes pod-test v0.1!! ClientIP: 10.244.1.242, ServerName: pod-test1-5d64bc66d7-wnkwd, ServerIP: 10.244.2.241!

 

ef32dc65080cb2659022bf020e8e298d

 

用户访问 www.ming.org DNS解析到lb外部网卡ip 通过haproxy反向代理转发到k8s 任何一个node节点30000端口 的ingress-nginx-controller service，通过ingress-nginx service 找到对应ep

找到ingress-nginx-controller 的pod节点，找到pod里面ingress规则，从ingress规则找到域名 www.ming.org 转发到哪个service ，再找到这个service 找到对应ep 就找到后端的pod

 

  

 

 

posted @ 2026-05-26 17:42  minger_lcm  阅读(15)  评论(0)    收藏  举报