Kubernetes部署nginx-ingress
一、Ingress介绍
ingress可以让互联网客户访问kubernetes集群，而ClientIP和coredns只能在集群内部访问，Ingress的架构图如下：
要使用 Ingress的步骤
- 先部署 Ingress Controller 实体（相当于前端 Nginx）
- 然后再创建 Ingress （相当于 Nginx 配置的 k8s 资源体现）
- Ingress Controller 部署好后会动态检测 Ingress 的创建情况生成相应配置
Ingress Controller 的实现有很多种：
- 基于 Nginx 的，基于 Nginx 的 Ingress Controller 有两种，一种是 k8s 社区提供的 ingress-nginx，另一种是 Nginx 社区提供的nginx-ingress（常见，亦本文的方式），参见他们的区别
- 基于 HAProxy的，
- 基于 OpenResty 的 Kong Ingress Controller 等
- 更多 Controller 见：https://kubernetes.io/docs/concepts/services-networking/ingress-controllers
基于Nginx Ingress的拓扑图如下
二、安装nginx-ingress
先查找仓库
$ helm search repo nginx-ingress NAME CHART VERSION APP VERSION DESCRIPTION aliyuncs/nginx-ingress 1.30.3 0.28.0 An nginx Ingress controller that uses ConfigMap... aliyuncs/nginx-ingress-controller 5.3.4 0.29.0 Chart for the nginx Ingress controller bitnami/nginx-ingress-controller 9.0.2 1.0.4 Chart for the nginx Ingress controller aliyuncs/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
安装版本最新的第三个，要仔细看安装以后给的提示
$ helm install nginx-ingress bitnami/nginx-ingress-controller NAME: nginx-ingress LAST DEPLOYED: Fri Nov 5 15:33:30 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: nginx-ingress-controller CHART VERSION: 9.0.3 APP VERSION: 1.0.4 ** Please be patient while the chart is being deployed ** The nginx-ingress controller has been installed. Get the application URL by running these commands: NOTE: It may take a few minutes for the LoadBalancer IP to be available. You can watch its status by running 'kubectl get --namespace default svc -w nginx-ingress-nginx-ingress-controller' export SERVICE_IP=$(kubectl get svc --namespace default nginx-ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo "Visit http://${SERVICE_IP} to access your application via HTTP." echo "Visit https://${SERVICE_IP} to access your application via HTTPS." An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: name: example namespace: default spec: rules: - host: www.example.com http: paths: - backend: service: name: example-service port: number: 80 path: / pathType: Prefix # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: default data: tls.crt: <base64 encoded cert> tls.key: <base64 encoded key> type: kubernetes.io/tls
查看安装的release
$ helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx-ingress default 1 2021-11-02 08:22:43.135546577 +0000 UTC deployed nginx-ingress-1.30.3 0.28.0
查看pods
$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES gostarter-dep-df898587f-9gfmh 1/1 Running 0 3h20m 192.168.205.50 kbsm <none> <none> gostarter-dep-df898587f-p76kn 1/1 Running 0 3h20m 192.168.205.39 kbsm <none> <none> gostarter-dep-df898587f-scn8g 1/1 Running 0 3h20m 192.168.205.37 kbsm <none> <none> gostarter-dep-df898587f-wq2cz 1/1 Running 0 3h20m 192.168.184.36 kbs2 <none> <none> gostarter-dep-df898587f-ztsmc 1/1 Running 0 3h20m 192.168.205.62 kbsm <none> <none> nginx-ingress-controller-6f4cf4656d-m7wvn 0/1 Running 2 (30s ago) 2m13s 192.168.151.2 kbs1 <none> <none> nginx-ingress-default-backend-78669dcf66-md9bp 1/1 Running 0 2m13s 192.168.151.57 kbs1 <none> <none>
查看Service
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gostarter-svc ClusterIP 10.109.68.204 <none> 8000/TCP 5h44m kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21d nginx-ingress-controller LoadBalancer 10.101.177.250 <pending> 80:31534/TCP,443:32619/TCP 15m nginx-ingress-default-backend ClusterIP 10.108.181.102 <none> 80/TCP 15m
访问一下backend
$ curl http://10.108.181.102 default backend - 404
三、nginx-ingress错误
我安装过程中出现过这种问题
$ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-ingress-controller-6f4cf4656d-m7wvn 0/1 CrashLoopBackOff 9 (43s ago) 18m nginx-ingress-default-backend-78669dcf66-md9bp 1/1 Running 0 18m
查看这个pod的日志，有大量这种错误：Failed to list *v1beta1.Ingress: the server could not find the requested resource
$ kubectl logs nginx-ingress-controller-6f4cf4656d-m7wvn
I1102 08:39:16.674917 8 flags.go:205] Watching for Ingress class: nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.28.0
Build: git-1f93cb8f3
W1102 08:39:16.675175 8 flags.go:250] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.7
-------------------------------------------------------------------------------
W1102 08:39:16.675227 8 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1102 08:39:16.675433 8 main.go:193] Creating API client for https://10.96.0.1:443
I1102 08:39:16.842896 8 main.go:237] Running in Kubernetes cluster version v1.22 (v1.22.2) - git (clean) commit 8b5a19147530eaac9476b0ab82980b4088bbc1b2 - platform linux/amd64
I1102 08:39:16.858079 8 main.go:91] Validated default/nginx-ingress-default-backend as the default backend.
I1102 08:39:17.000127 8 main.go:102] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
W1102 08:39:17.047593 8 store.go:636] Unexpected error reading configuration configmap: configmaps "nginx-ingress-controller" not found
I1102 08:39:17.054741 8 nginx.go:263] Starting NGINX Ingress controller
E1102 08:39:18.379226 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1102 08:39:19.449966 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1102 08:39:20.753084 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1102 08:39:22.109971 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1102 08:39:23.136053 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1102 08:39:24.458595 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1102 08:39:25.492555 8 reflector.go:153] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:181: Failed to list *v1beta1.Ingress: the server could not find the requested resource
I1102 08:39:53.382051 8 main.go:152] Received SIGTERM, shutting down
I1102 08:39:53.382078 8 nginx.go:391] Shutting down controller queues
I1102 08:39:53.382091 8 status.go:117] updating status of Ingress rules (remove)
E1102 08:39:53.382177 8 store.go:185] timed out waiting for caches to sync
I1102 08:39:53.382249 8 nginx.go:307] Starting NGINX process
I1102 08:39:53.382435 8 leaderelection.go:242] attempting to acquire leader lease default/ingress-controller-leader-nginx...
E1102 08:39:53.382666 8 queue.go:78] queue has been shutdown, failed to enqueue: &ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}
I1102 08:39:53.398779 8 leaderelection.go:252] successfully acquired lease default/ingress-controller-leader-nginx
I1102 08:39:53.398841 8 status.go:86] new leader elected: nginx-ingress-controller-6f4cf4656d-m7wvn
E1102 08:39:53.398884 8 queue.go:78] queue has been shutdown, failed to enqueue: &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}
I1102 08:39:53.422921 8 status.go:136] removing address from ingress status ([192.168.0.106])
I1102 08:39:53.423027 8 nginx.go:407] Stopping NGINX process
2021/11/02 08:39:53 [notice] 36#36: signal process started
I1102 08:39:56.444623 8 nginx.go:420] NGINX process has stopped
I1102 08:39:56.444662 8 main.go:160] Handled quit, awaiting Pod deletion
I1102 08:40:06.445065 8 main.go:163] Exiting with 0
原因 ： 我的kubernetes:v1.22 不再支持v1beta1，所以与低版本的aliyuncs/nginx-ingress:0.28.0不匹配，要改成使用高版本的bitnami/nginx-ingress-controller:1.0.4
下面再列一下helm的nginx-ingress的chart的源的查询
$ helm search repo nginx-ingress
NAME CHART VERSION APP VERSION DESCRIPTION
aliyuncs/nginx-ingress 1.30.3 0.28.0 An nginx Ingress controller that uses ConfigMap...
aliyuncs/nginx-ingress-controller 5.3.4 0.29.0 Chart for the nginx Ingress controller
bitnami/nginx-ingress-controller 9.0.2 1.0.4 Chart for the nginx Ingress controller
aliyuncs/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
五、安装Nginx
helm中 查看chart
$ helm search repo nginx
NAME CHART VERSION APP VERSION DESCRIPTION
aliyuncs/nginx 5.1.5 1.16.1 Chart for the nginx server
aliyuncs/nginx-ingress 1.30.3 0.28.0 An nginx Ingress controller that uses ConfigMap...
aliyuncs/nginx-ingress-controller 5.3.4 0.29.0 Chart for the nginx Ingress controller
aliyuncs/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
aliyuncs/nginx-php 1.0.0 nginx-1.10.3_php-7.0 Chart for the nginx php server
bitnami/nginx 9.5.12 1.21.3 Chart for the nginx server
bitnami/nginx-ingress-controller 9.0.2 1.0.4 Chart for the nginx Ingress controller
bitnami/kong 4.1.7 2.6.0 Kong is a scalable, open source API layer (aka ...
执行安装
helm install nginx bitnami/nginx NAME: nginx LAST DEPLOYED: Fri Nov 5 16:01:35 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: nginx CHART VERSION: 9.5.13 APP VERSION: 1.21.4 ** Please be patient while the chart is being deployed ** NGINX can be accessed through the following DNS name from within your cluster: nginx.default.svc.cluster.local (port 80) To access NGINX from outside the cluster, follow the steps below: 1. Get the NGINX URL by running these commands: NOTE: It may take a few minutes for the LoadBalancer IP to be available. Watch the status with: 'kubectl get svc --namespace default -w nginx' export SERVICE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].port}" services nginx) export SERVICE_IP=$(kubectl get svc --namespace default nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo "http://${SERVICE_IP}:${SERVICE_PORT}" root@kbsm:~/k8s/nginx-ingress#
查看pod
$ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-588469f6d6-rd5gx 1/1 Running 0 82s nginx-ingress-nginx-ingress-controller-7bdbcc7787-pgk82 1/1 Running 0 29m nginx-ingress-nginx-ingress-controller-default-backend-57ftmtdx 1/1 Running 0 29m
六、Ingress配置
先配置nginx自身的ingress：vi nginx-ingress.yaml
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-nginx annotations: # use the shared ingress-nginx kubernetes.io/ingress.class: "nginx" spec: rules: - host: kbsm http: paths: - path: / pathType: Prefix backend: service: name: nginx port: number: 80
ingress的配置新版本的格式与老版本格式不一致，可以查看官方文档
对于安装的kubernetes集群支持的apiVersion版本，可以通过这个命令查看：kubectl api-versions
然后发布ingress
$ kubectl apply -f nginx-ingress.yaml
ingress.networking.k8s.io/ingress-nginx created
七、Ingress测试
八、删除本文配置
如果nginx-ingress配置失败，删除本文所有的配置
kubectl delete -f nginx-ingress.yaml
helm uninstall nginx
helm uninstall nginx-ingress
参考资料：
作者 ：秋时
本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接。
