kubernetes 系列教程 (十六) 基于 nginx ingress 实现服务暴露
视频教程连接：kubernetes 快速入门
写在前面
本章介绍 kubernetes 系列教程的 ingress 概念，在 kubernetes 中对外暴露服务的方式有两种：service（NodePort 或者外部 LoadBalancer）和 ingress，其中 service 是提供四层的负载均衡，通过 iptables DNAT 或 lvs nat 模式实现后端 Pod 的代理请求。如需实现 http，域名，URI，证书等请求方式，service 是无法实现的，需要借助于 ingress 来来实现，本文将来介绍 ingress 相关的内容。
1. Ingress 简介
An API object that manages external access to the services in a cluster, typically HTTP. Ingress can provide load balancing, SSL termination and name-based virtual hosting.
引用官方关于 ingress 的介绍我们可以得知，ingress 是一种通过 http 协议暴露 kubernetes 内部服务的 api 对象，即充当 Edge Router 边界路由器的角色对外基于七层的负载均衡调度机制，能够提供以下几个功能：
- 负载均衡，将请求自动负载均衡到后端的 Pod 上；
- SSL 加密，客户端到 Ingress Controller 为 https 加密，到后端 Pod 为明文的 http；
- 基于名称的虚拟主机，提供基于域名或 URI 更灵活的路由方式
Ingress Controller
实现 Ingress 包含的组件有：
- Ingress，客户端，负责定义 ingress 配置，将请求转发给 Ingress Controller；
- Ingress Controller，Ingress 控制器，实现七层转发的 Edge Router，通过调用 k8s 的 api 动态感知集群中 Pod 的变化而动态更新配置文件并重载， Controller 需要部署在 k8s 集群中以实现和集群中的 pod 通信，通常以 DaemonSets 或 Deployments 的形式部署，并对外暴露 80 和 443 端口，对于 DaemonSets 来说，一般是以 hostNetwork 或者 hostPort 的形式暴露，Deployments 则以 NodePort 的方式暴露，控制器的多个节点则借助外部负载均衡 ExternalLB 以实现统一接入；
- Ingress 配置规则，Controller 控制器通过 service 服务发现机制动态实现后端 Pod 路由转发规则的实现；
- Service，kuberntes 中四层的负载均衡调度机制，Ingress 借助 service 的服务发现机制实现集群中 Pod 资源的动态感知；
- Pod，后端实际负责响应请求容器，由控制器如 Deployment 创建，通过标签 Labels 和 service 关联，服务发现。
简而言之，ingress 控制器借助 service 的服务发现机制实现配置的动态更新以实现 Pod 的负载均衡机制实现，由于涉及到 Ingress Controller 的动态更新，目前社区 Ingress Controller 大体包含两种类型的控制器：
- 传统的七层负载均衡如 Nginx，HAproxy，开发了适应微服务应用的插件，具有成熟，高性能等优点；
- 新型微服务负载均衡如 Traefik，Envoy，Istio，专门适用于微服务 + 容器化应用场景，具有动态更新特点；
|
类型 |
常见类型 |
优点 |
缺点 |
|---|---|---|---|
|
传统负载均衡 |
nginx，haproxy |
成熟，稳定，高性能 |
动态更新需 reload 配置文件 |
|
微服务负载均衡 |
Traefik，Envoy，Istio |
天生为微服务而生，动态更新 |
性能还有待提升 |
2. Nginx Ingress
2.1 Nginx ingress 介绍
By default, pods of Kubernetes services are not accessible from the external network, but only by other pods within the Kubernetes cluster. Kubernetes has a built‑in configuration for HTTP load balancing, called Ingress, that defines rules for external connectivity to Kubernetes services. Users who need to provide external access to their Kubernetes services create an Ingress resource that defines rules, including the URI path, backing service name, and other information. The Ingress controller can then automatically program a frontend load balancer to enable Ingress configuration. The NGINX Ingress Controller for Kubernetes is what enables Kubernetes to configure NGINX and NGINX Plus for load balancing Kubernetes services.
Nginx ingress
Nginx Ingress Controller 是实现 ingress 的具体实现，包含有两个版本：Ngnix OSS 和 Nginx Plus 版，后者是商业化增强版，支持更多的功能，详情参考官方文档介绍 https://www.nginx.com/products/nginx/kubernetes-ingress-controller#compare-versions
Nginx ingress 版本 features 对比
2.2 Nginx ingress 安装
首先需要安装 Nginx Ingress Controller 控制器，控制器安装方式包含两种：DaemonSets 和 Deployments。
- DaemonSets 通过 hostPort 的方式暴露 80 和 443 端口，可通过 Node 的调度由专门的节点实现部署
- Deployments 则通过 NodePort 的方式实现控制器端口的暴露，借助外部负载均衡实现高可用负载均衡
除此之外，还需要部署 Namespace，ServiceAccount，RBAC，Secrets，Custom Resource Definitions 等资源，如下开始部署。
2.2.1 基础依赖环境准备
1、github 中下载源码包，安装部署文件在 kubernetes-ingress/deployments/ 目录下
[root@node-1 ~]# git clone https://github.com/nginxinc/kubernetes-ingress.git
[root@node-1 ~]# tree kubernetes-ingress/deployments/
kubernetes-ingress/deployments/
├── common
│ ├── custom-resource-definitions.yaml 自定义资源
│ ├── default-server-secret.yaml Secrets
│ ├── nginx-config.yaml
│ └── ns-and-sa.yaml Namspace+ServiceAccount
├── daemon-set
│ ├── nginx-ingress.yaml DaemonSets控制器
│ └── nginx-plus-ingress.yaml
├── deployment
│ ├── nginx-ingress.yaml Deployments控制器
│ └── nginx-plus-ingress.yaml
├── helm-chart Helm安装包
│ ├── chart-icon.png
│ ├── Chart.yaml
│ ├── README.md
│ ├── templates
│ │ ├── controller-configmap.yaml
│ │ ├── controller-custom-resources.yaml
│ │ ├── controller-daemonset.yaml
│ │ ├── controller-deployment.yaml
│ │ ├── controller-leader-election-configmap.yaml
│ │ ├── controller-secret.yaml
│ │ ├── controller-serviceaccount.yaml
│ │ ├── controller-service.yaml
│ │ ├── controller-wildcard-secret.yaml
│ │ ├── _helpers.tpl
│ │ ├── NOTES.txt
│ │ └── rbac.yaml
│ ├── values-icp.yaml
│ ├── values-plus.yaml
│ └── values.yaml
├── rbac RBAC认证授权
│ └── rbac.yaml
├── README.md
└── service Service定义
├── loadbalancer-aws-elb.yaml
├── loadbalancer.yaml DaemonSets暴露服务方式
└── nodeport.yaml Deployments暴露服务方式2、创建 Namespace 和 ServiceAccount, kubectl apply -f common/default-server-secret.yaml
apiVersion: v1
kind: Namespace
metadata:
name: nginx-ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress
namespace: nginx-ingress3、创建 Secrets 自签名证书，kubectl apply -f common/default-server-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: default-server-secret
namespace: nginx-ingress
type: Opaque
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=4、创建 ConfigMap 自定义配置文件，kubectl apply -f common/nginx-config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:5、为虚拟云主机和虚拟云主机路由定义自定义资源，支持自定义虚拟主机和虚拟路由，kubectl apply -f common/custom-resource-definitions.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualservers.k8s.nginx.org
spec:
group: k8s.nginx.org
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: virtualservers
singular: virtualserver
kind: VirtualServer
shortNames:
- vs
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualserverroutes.k8s.nginx.org
spec:
group: k8s.nginx.org
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: virtualserverroutes
singular: virtualserverroute
kind: VirtualServerRoute
shortNames:
- vsr6、配置 RBAC 认证授权，实现 ingress 控制器访问集群中的其他资源，kubectl apply -f rbac/rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nginx-ingress
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- update
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- list
- watch
- get
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers
- virtualserverroutes
verbs:
- list
- watch
- get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nginx-ingress
subjects:
- kind: ServiceAccount
name: nginx-ingress
namespace: nginx-ingress
roleRef:
kind: ClusterRole
name: nginx-ingress
apiGroup: rbac.authorization.k8s.io2.2.2 部署 Ingress 控制器
1、 部署控制器，控制器可以 DaemonSets 和 Deployment 的形式部署，如下是 DaemonSets 的配置文件
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: Always
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80 #通过hostPort的方式暴露端口
- name: https
containerPort: 443
hostPort: 443
#- name: prometheus
#containerPort: 9113
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-leader-election
#- -enable-prometheus-metricsDeployments 的配置文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
replicas: 1 #副本的个数
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: Always
name: nginx-ingress
ports: #内部暴露的服务端口，需要通过NodePort的方式暴露给外部
- name: http
containerPort: 80
- name: https
containerPort: 443
#- name: prometheus
#containerPort: 9113
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-leader-election
#- -enable-prometheus-metrics2、我们以 DaemonSets 的方式部署，DaemonSet 部署集群中各个节点都是对等，如果有外部 LoadBalancer 则通过外部负载均衡路由至 Ingress 中
[root@node-1 deployments]# kubectl apply -f daemon-set/nginx-ingress.yaml
daemonset.apps/nginx-ingress created
[root@node-1 deployments]# kubectl get daemonsets -n nginx-ingress
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
nginx-ingress 3 3 3 3 3 <none> 15s
[root@node-1 ~]# kubectl get pods -n nginx-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-7mpfc 1/1 Running 0 2m44s 10.244.0.50 node-1 <none> <none>
nginx-ingress-l2rtj 1/1 Running 0 2m44s 10.244.1.144 node-2 <none> <none>
nginx-ingress-tgf6r 1/1 Running 0 2m44s 10.244.2.160 node-3 <none> <none>3、校验 Nginx Ingress 安装情况，此时三个节点均是对等，即访问任意一个节点均能实现相同的效果，统一入口则通过外部负载均衡，如果在云环境下执行 kubectl apply -f service/loadbalancer.yaml 创建外部负载均衡实现入口调度，自建的可以通过 lvs 或 nginx 等负载均衡实现接入，本文不再赘述，读者可以自行研究。
nginx ingress 安装校验
备注说明：如果以 Deployments 的方式部署，则需要执行 service/nodeport.yaml 创建 NodePort 类型的 Service，实现的效果和 DaemonSets 类似。
3. Ingress 资源定义
上面的章节已安装了一个 Nginx Ingress Controller 控制器，有了 Ingress 控制器后，我们就可以定义 Ingress 资源来实现七层负载转发了，大体上 Ingress 支持三种使用方式：1. 基于虚拟主机转发，2. 基于虚拟机主机 URI 转发，3. 支持 TLS 加密转发。
3.1 Ingress 定义
1、环境准备，先创建一个 nginx 的 Deployment 应用，包含 2 个副本
[root@node-1 ~]# kubectl run ingress-demo --image=nginx:1.7.9 --port=80 --replicas=2
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-demo 2/2 2 2 116s2、以 service 方式暴露服务端口
[root@node-1 ~]# kubectl expose deployment ingress-demo --port=80 --protocol=TCP --target-port=80
service/ingress-demo exposed
[root@node-1 ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-demo ClusterIP 10.109.33.91 <none> 80/TCP 2m15s3、上述两个步骤已创建了一个 service，如下我们定义一个 ingress 对象将起转发至 ingress-demo 这个 service，通过 ingress.class 指定控制器的类型为 nginx
[root@node-1 nginx-ingress]# cat nginx-ingress-demo.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
rules:
- host: www.happylau.cn
http:
paths:
- path: /
backend:
serviceName: ingress-demo
servicePort: 804、创建 ingress 对象
[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-demo.yaml
ingress.extensions/nginx-ingress-demo created
查看ingress资源列表
[root@node-1 nginx-ingress]# kubectl get ingresses
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-demo www.happylau.cn 80 4m4s5、查看 ingress 详情，可以在 Rules 规则中看到后端 Pod 的列表，自动发现和关联相关 Pod
[root@node-1 ~]# kubectl describe ingresses nginx-ingress-demo
Name: nginx-ingress-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
www.happylau.cn
/ ingress-demo:80 (10.244.1.146:80,10.244.2.162:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-demo","namespace":"default"},"spec":{"rules":[{"host":"www.happylaulab.cn","http":{"paths":[{"backend":{"serviceName":"ingress-demo","servicePort":80},"path":"/"}]}}]}}
kubernets.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 9m7s nginx-ingress-controller Configuration for default/nginx-ingress-demo was added or updated
Normal AddedOrUpdated 9m7s nginx-ingress-controller Configuration for default/nginx-ingress-demo was added or updated
Normal AddedOrUpdated 9m7s nginx-ingress-controller Configuration for default/nginx-ingress-demo was added or updated6、测试验证，ingress 规则的配置信息已注入到 Ingress Controller 中，环境中 Ingress Controller 是以 DaemonSets 的方式部署在集群中，如果有外部的负载均衡，则将 www.happylau.cn 域名的地址解析为负载均衡 VIP。由于测试环境没有搭建负载均衡，将 hosts 解析执行 node-1，node-2 或者 node-3 任意一个 IP 都能实现相同的功能。
ingress 测试
上述测试解析正常，当然也可以解析为 node-1 和 node-2 的 IP，如下：
[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.101
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 24 Dec 2019 10:32:22 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
ETag: "54999765-264"
Accept-Ranges: bytes
[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 24 Dec 2019 10:32:24 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
ETag: "54999765-264"
Accept-Ranges: bytes3.2 Ingress 动态配置
上面的章节介绍了 ingress 资源对象的申明配置，在这个章节中我们探究一下 Nginx Ingress Controller 的实现机制和动态配置更新机制，以方便了解 Ingress 控制器的工作机制。
1、 查看 Nginx Controller 控制器的配置文件，在 nginx-ingress pod 中存储着 ingress 的配置文件
[root@node-1 ~]# kubectl get pods -n nginx-ingress
NAME READY STATUS RESTARTS AGE
nginx-ingress-7mpfc 1/1 Running 0 6h15m
nginx-ingress-l2rtj 1/1 Running 0 6h15m
nginx-ingress-tgf6r 1/1 Running 0 6h15m
#查看配置文件，每个ingress生成一个配置文件，文件名为：命名空间-ingres名称.conf
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- ls -l /etc/nginx/conf.d
total 4
-rw-r--r-- 1 nginx nginx 1005 Dec 24 10:06 default-nginx-ingress-demo.conf
#查看配置文件
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- cat /etc/nginx/conf.d/default-nginx-ingress-demo.conf
# configuration for default/nginx-ingress-demo
#upstream的配置，会用least_conn算法，通过service服务发现机制动态识别到后端的Pod
upstream default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 {
zone default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 256k;
random two least_conn;
server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name www.happylau.cn;
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80; #调用upstream实现代理
}
}通过上述查看配置文件可得知，Nginx Ingress Controller 实际是根据 ingress 规则生成对应的 nginx 配置文件，以实现代理转发的功能，加入 Deployments 的副本数变更后 nginx 的配置文件会发生什么改变呢？
2、更新控制器的副本数，由 2 个 Pod 副本扩容至 3 个
[root@node-1 ~]# kubectl scale --replicas=3 deployment ingress-demo
deployment.extensions/ingress-demo scaled
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-demo 3/3 3 3 123m3、再次查看 nginx 的配置文件，ingress 借助于 service 的服务发现机制，将加入的 Pod 自动加入到 nginx upstream 中
nginx 动态更新配置
4、查看 nginx pod 的日志（kubectl logs nginx-ingress-7mpfc -n nginx-ingress），有 reload 优雅重启的记录，即通过更新配置文件 + reload 实现配置动态更新。
nginx 控制器重启日志记录
通过上述的配置可知，ingress 调用 kubernetes api 去感知 kubernetes 集群中的变化情况，Pod 的增加或减少这些变化，然后动态更新 nginx ingress controller 的配置文件，并重新载入配置。当集群规模越大时，会频繁涉及到配置文件的变动和重载，因此 nginx 这方面会存在先天的劣势，专门为微服务负载均衡应运而生，如 Traefik，Envoy，Istio，这些负载均衡工具能够提供大规模，频繁动态更新的场景，但性能相比 Nginx，HAproxy 还存在一定的劣势。往后的章节中，我们再对其他的 Ingress 控制器做介绍。
3.3 Ingress 路径转发
Ingress 支持 URI 格式的转发方式，同时支持 URL 重写，如下以两个 service 为例演示，service-1 安装 nginx，service-2 安装 httpd，分别用 http://demo.happylau.cn/news 和 http://demo.happylau.cn/sports 转发到两个不同的 service
1、环境准备，创建两个应用并实现 service 暴露，创建 deployments 时指定 --explose 创建 service
[root@node-1 ~]# kubectl run service-1 --image=nginx:1.7.9 --port=80 --replicas=1 --expose=true
service/service-1 created
deployment.apps/service-1 created
[root@node-1 ~]# kubectl run service-2 --image=httpd --port=80 --replicas=1 --expose=true
service/service-2 created
deployment.apps/service-2 created
查看deployment状态
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-demo 4/4 4 4 4h36m
service-1 1/1 1 1 65s
service-2 1/1 1 1 52s
查看service状态，服务已经正常
[root@node-1 ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-demo ClusterIP 10.109.33.91 <none> 80/TCP 4h36m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 101d
service-1 ClusterIP 10.106.245.71 <none> 80/TCP 68s
service-2 ClusterIP 10.104.204.158 <none> 80/TCP 55s2、创建 ingress 对象，通过一个域名将请求转发至后端两个 service
[root@node-1 nginx-ingress]# cat nginx-ingress-uri-demo.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-uri-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: demo.happylau.cn
http:
paths:
- path: /news
backend:
serviceName: service-1
servicePort: 80
- path: /sports
backend:
serviceName: service-2
servicePort: 803、创建 ingress 规则，查看详情
[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-uri-demo.yaml
ingress.extensions/nginx-ingress-uri-demo created
#查看详情
[root@node-1 nginx-ingress]# kubectl get ingresses.
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-demo www.happylau.cn 80 4h35m
nginx-ingress-uri-demo demo.happylau.cn 80 4s
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-uri-demo
Name: nginx-ingress-uri-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules: #对应的转发url规则
Host Path Backends
---- ---- --------
demo.happylau.cn
/news service-1:80 (10.244.2.163:80)
/sports service-2:80 (10.244.1.148:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/rewrite-target":"/"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-uri-demo","namespace":"default"},"spec":{"rules":[{"host":"demo.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/news"},{"backend":{"serviceName":"service-2","servicePort":80},"path":"/sports"}]}}]}}
kubernets.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 11s nginx-ingress-controller Configuration for default/nginx-ingress-uri-demo was added or updated
Normal AddedOrUpdated 11s nginx-ingress-controller Configuration for default/nginx-ingress-uri-demo was added or updated
Normal AddedOrUpdated 11s nginx-ingress-controller Configuration for default/nginx-ingress-uri-demo was added or updated4、准备测试，站点中创建对应的路径
[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
root@service-1-7b66bf758f-xj9jh:/# echo "service-1 website page" >/usr/share/nginx/html/news
[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash
root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "service-2 website page" >/usr/local/apache2/htdocs/sports5、测试验证
[root@node-1 ~]# curl http://demo.happylau.cn/news --resolve demo.happylau.cn:80:10.254.100.101
service-1 website page
[root@node-1 ~]# curl http://demo.happylau.cn/sports --resolve demo.happylau.cn:80:10.254.100.101
service-2 website page6、通过上述的验证测试可以得知，ingress 支持 URI 的路由方式转发，其对应在 ingress 中的配置文件内容是怎样的呢，我们看下 ingress controller 生成对应的 nginx 配置文件内容，实际是通过 ingress 的 location 来实现，将不同的 localtion 转发至不同的 upstream 以实现 service 的关联，配置文件如下：
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress /bin/bash
nginx@nginx-ingress-7mpfc:/$ cat /etc/nginx/conf.d/default-nginx-ingress-uri-demo.conf |grep -v "^$"
# configuration for default/nginx-ingress-uri-demo
#定义两个upstream和后端的service关联
upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 {
zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 256k;
random two least_conn;
server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 {
zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 256k;
random two least_conn;
server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name demo.happylau.cn;
#定义location实现代理，通过proxy_pass和后端的service关联
location /news {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80;
}
location /sports {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80;
}
}3.4 Ingress 虚拟主机
ingress 支持基于名称的虚拟主机，实现单个 IP 多个域名转发的需求，通过请求头部携带主机名方式区分开，将上个章节的 ingress 删除，使用 service-1 和 service-2 两个 service 来做演示。
1、创建 ingress 规则，通过主机名实现转发规则
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-virtualname-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
rules:
- host: news.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-1
servicePort: 80
- host: sports.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-2
servicePort: 802、生成 ingress 规则并查看详情，一个 ingress 对应两个 HOSTS
[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-virtualname.yaml
ingress.extensions/nginx-ingress-virtualname-demo created
#查看列表
[root@node-1 nginx-ingress]# kubectl get ingresses nginx-ingress-virtualname-demo
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-virtualname-demo news.happylau.cn,sports.happylau.cn 80 12s
#查看详情
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-virtualname-demo
Name: nginx-ingress-virtualname-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
news.happylau.cn
/ service-1:80 (10.244.2.163:80)
sports.happylau.cn
/ service-2:80 (10.244.1.148:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-virtualname-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}]}}
kubernets.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 28s nginx-ingress-controller Configuration for default/nginx-ingress-virtualname-demo was added or updated
Normal AddedOrUpdated 28s nginx-ingress-controller Configuration for default/nginx-ingress-virtualname-demo was added or updated
Normal AddedOrUpdated 28s nginx-ingress-controller Configuration for default/nginx-ingress-virtualname-demo was added or updated3、准备测试数据并测试
[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
root@service-1-7b66bf758f-xj9jh:/# echo "news demo" >/usr/share/nginx/html/index.html
[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash
root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "sports demo" >/usr/local/apache2/htdocs/index.html测试：
[root@node-1 ~]# curl http://news.happylau.cn --resolve news.happylau.cn:80:10.254.100.102
news demo
[root@node-1 ~]# curl http://sports.happylau.cn --resolve sports.happylau.cn:80:10.254.100.102
sports demo4、查看 nginx 的配置文件内容，通过在 server 中定义不同的 server_name 以区分，代理到不同的 upstream 以实现 service 的代理。
# configuration for default/nginx-ingress-virtualname-demo
upstream default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 {
zone default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 256k;
random two least_conn;
server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 {
zone default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 256k;
random two least_conn;
server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name news.happylau.cn;
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80;
}
}
server {
listen 80;
server_tokens on;
server_name sports.happylau.cn;
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80;
}
}3.5 Ingress TLS 加密
四层的负载均衡无法支持 https 请求，当前大部分业务都要求以 https 方式接入，Ingress 能支持 https 的方式接入，通过 Secrets 存储证书 + 私钥，实现 https 接入，同时还能支持 http 跳转功能。对于用户的请求流量来说，客户端到 ingress controller 是 https 流量，ingress controller 到后端 service 则是 http，提高用户访问性能，如下介绍 ingress TLS 功能实现步骤。
1、生成自签名证书和私钥
[root@node-1 ~]# openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout tls.key -out tls.crt
Generating a 2048 bit RSA private key
....................................................+++
........................................+++
writing new private key to 'tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN #国家
State or Province Name (full name) []:GD #省份
Locality Name (eg, city) [Default City]:ShenZhen #城市
Organization Name (eg, company) [Default Company Ltd]:Tencent #公司
Organizational Unit Name (eg, section) []:HappyLau #组织
Common Name (eg, your name or your server's hostname) []:www.happylau.cn #域名
Email Address []:573302346@qq.com #邮箱地址
#tls.crt为证书，tls.key为私钥
[root@node-1 ~]# ls tls.* -l
-rw-r--r-- 1 root root 1428 12月 26 13:21 tls.crt
-rw-r--r-- 1 root root 1708 12月 26 13:21 tls.key2、配置 Secrets，将证书和私钥配置到 Secrets 中
[root@node-1 ~]# kubectl create secret tls happylau-sslkey --cert=tls.crt --key=tls.key
secret/happylau-sslkey created
查看Secrets详情,证书和私要包含在data中，文件名为两个不同的key：tls.crt和tls.key
[root@node-1 ~]# kubectl describe secrets happylau-sslkey
Name: happylau-sslkey
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1428 bytes
tls.key: 1708 bytes3、配置 ingress 调用 Secrets 实现 SSL 证书加密
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-tls-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
tls:
- hosts:
- news.happylau.cn
- sports.happylau.cn
secretName: happylau-sslkey
rules:
- host: news.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-1
servicePort: 80
- host: sports.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-2
servicePort: 804、创建 ingress 并查看 ingress 详情
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-tls-demo
Name: nginx-ingress-tls-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
happylau-sslkey terminates news.happylau.cn,sports.happylau.cn
Rules:
Host Path Backends
---- ---- --------
news.happylau.cn
/ service-1:80 (10.244.2.163:80)
sports.happylau.cn
/ service-2:80 (10.244.1.148:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-tls-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["news.happylau.cn","sports.happylau.cn"],"secretName":"happylau-sslkey"}]}}
kubernets.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 22s nginx-ingress-controller Configuration for default/nginx-ingress-tls-demo was added or updated
Normal AddedOrUpdated 22s nginx-ingress-controller Configuration for default/nginx-ingress-tls-demo was added or updated
Normal AddedOrUpdated 22s nginx-ingress-controller Configuration for default/nginx-ingress-tls-demo was added or updated5、 将 news.happylau.cn 和 sports.happylau.cn 写入到 hosts 文件中，并通过 https://news.happylau.cn 的方式访问，浏览器访问内容提示证书如下，信任证书即可访问到站点内容。
tls 测试页面
查看证书详情，正是我们制作的自签名证书，生产实际使用时，推荐使用 CA 机构颁发签名证书。
证书详情
6、接下来查看一下 tls 配置 https 的 nginx 配置文件内容，可以看到在 server 块启用了 https 并配置证书，同时配置了 http 跳转，因此直接访问 http 也能够实现自动跳转到 https 功能。
# configuration for default/nginx-ingress-tls-demo
upstream default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 {
zone default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 256k;
random two least_conn;
server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 {
zone default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 256k;
random two least_conn;
server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
listen 443 ssl; #https监听端口，证书和key，实现和Secrets关联
ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;
server_tokens on;
server_name news.happylau.cn;
#http跳转功能，即访问http会自动跳转至https
if ($scheme = http) {
return 301 https://$host:443$request_uri;
}
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80;
}
}
server {
listen 80;
listen 443 ssl;
ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;
server_tokens on;
server_name sports.happylau.cn;
if ($scheme = http) {
return 301 https://$host:443$request_uri;
}
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80;
}
}4. Nginx Ingress 高级功能
4.1 定制化参数
ingress controller 提供了基础反向代理的功能，如果需要定制化 nginx 的特性或参数，需要通过 ConfigMap 和 Annotations 来实现，两者实现的方式有所不同，ConfigMap 用于指定整个 ingress 集群资源的基本参数，修改后会被所有的 ingress 对象所继承；Annotations 则被某个具体的 ingress 对象所使用，修改只会影响某个具体的 ingress 资源，冲突时其优先级高于 ConfigMap。
4.1.1 ConfigMap 自定义参数
安装 nginx ingress controller 时默认会包含一个空的 ConfigMap，可以通过 ConfigMap 来自定义 nginx controller 的默认参数，如下以修改一些参数为例：
1、 定义 ConfigMap 参数
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:
proxy-connect-timeout: "10s"
proxy-read-timeout: "10s"
proxy-send-timeout: "10"
client-max-body-size: "3m"2、 应用配置并查看 ConfigMap 配置
[root@node-1 ~]# kubectl get configmaps -n nginx-ingress nginx-config -o yaml
apiVersion: v1
data:
client-max-body-size: 3m
proxy-connect-timeout: 10s
proxy-read-timeout: 10s
proxy-send-timeout: 10s
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"client-max-body-size":"3m","proxy-connect-timeout":"10s","proxy-read-timeout":"10s","proxy-send-timeout":"10"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"nginx-config","namespace":"nginx-ingress"}}
creationTimestamp: "2019-12-24T04:39:23Z"
name: nginx-config
namespace: nginx-ingress
resourceVersion: "13845543"
selfLink: /api/v1/namespaces/nginx-ingress/configmaps/nginx-config
uid: 9313ae47-a0f0-463e-a25a-1658f1ca0d573 、此时，ConfigMap 定义的配置参数会被集群中所有的 Ingress 资源继承（除了 annotations 定义之外）
ConfigMap 参数验证
有很多参数可以定义，详情配置可参考方文档说明：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/configmap-and-annotations.md#Summary-of-ConfigMap-and-Annotations
4.1.2 Annotations 自定义参数
ConfigMap 定义的是全局的配置参数，修改后所有的配置都会受影响，如果想针对某个具体的 ingress 资源自定义参数，则可以通过 Annotations 来实现，下面开始以实际的例子演示 Annotations 的使用。
1、修改 ingress 资源，添加 annotations 的定义，通过 nginx.org 组修改了一些参数，如 proxy-connect-timeout，调度算法为 round_robin（默认为 least _conn）
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
nginx.org/proxy-connect-timeout: "30s"
nginx.org/proxy-send-timeout: "20s"
nginx.org/proxy-read-timeout: "20s"
nginx.org/client-max-body-size: "2m"
nginx.org/fail-timeout: "5s"
nginx.org/lb-method: "round_robin"
spec:
rules:
- host: www.happylau.cn
http:
paths:
- path: /
backend:
serviceName: ingress-demo
servicePort: 802、 重新应用 ingress 对象并查看参数配置情况
Nginx Annotations 演示
由上面的演示可得知，Annotations 的优先级高于 ConfigMapMap，Annotations 修改参数只会影响到某一个具体的 ingress 资源，其定义的方法和 ConfigMap 相相近似，但又有差别，部分 ConfigMap 的参数 Annotations 无法支持，反过来 Annotations 定义的参数 ConfigMap 也不一定支持，下图列举一下常规支持参数情况：
通用参数
日志支持
请求头部
认证和安全
upstream 支持
ConfigMap 和 Annotations 详细支持说明：链接说明
4.2 虚拟主机和路由
安装 nginx ingress 时我们安装了一个 customresourcedefinitions 自定义资源，其能够提供除了默认 ingress 功能之外的一些高级特性如
- 虚拟主机 VirtualServer
- 虚拟路由 VirtualServerRoute
- 健康检查 Healthcheck
- 流量切割 Split
- 会话保持 SessionCookie
- 重定向 Redirect
这些功能大部分依赖于 Nginx Plus 高级版本的支持，社区版本仅支持部分，对于企业级开发而言，丰富更多的功能可以购买企业级 Nginx Plus 版本。如下以通过 VirtualServer 和 VirtualServerRoute 定义 upstream 配置为例演示功能使用。
1、定义 VirtualServer 资源，其配置和 ingress 资源对象类似，能支持的功能会更丰富一点
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: cafe
spec:
host: cafe.example.com
tls:
secret: cafe-secret
upstreams:
- name: tea
service: tea-svc
port: 80
name: tea
service: ingress-demo
subselector:
version: canary
lb-method: round_robin
fail-timeout: 10s
max-fails: 1
max-conns: 32
keepalive: 32
connect-timeout: 30s
read-timeout: 30s
send-timeout: 30s
next-upstream: "error timeout non_idempotent"
next-upstream-timeout: 5s
next-upstream-tries: 10
client-max-body-size: 2m
tls:
enable: true
routes:
- path: /tea
action:
pass: tea2、 应用资源并查看 VirtualServer 资源列表
[root@node-1 ~]# kubectl apply -f vs.yaml
virtualserver.k8s.nginx.org/cafe unchanged
[root@node-1 ~]# kubectl get virtualserver
NAME AGE
cafe 2m52s3、检查 ingress 控制器的配置文件情况，生成的配置和 upstream 定义一致
nginx@nginx-ingress-7mpfc:/etc/nginx/conf.d$ cat vs_default_cafe.conf
upstream vs_default_cafe_tea {
zone vs_default_cafe_tea 256k;
server 10.244.0.51:80 max_fails=1 fail_timeout=10s max_conns=32;
server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=32;
server 10.244.1.147:80 max_fails=1 fail_timeout=10s max_conns=32;
server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=32;
keepalive 32;
}
server {
listen 80;
server_name cafe.example.com;
listen 443 ssl;
ssl_certificate /etc/nginx/secrets/default;
ssl_certificate_key /etc/nginx/secrets/default;
ssl_ciphers NULL;
server_tokens "on";
location /tea {
proxy_connect_timeout 30s;
proxy_read_timeout 30s;
proxy_send_timeout 30s;
client_max_body_size 2m;
proxy_max_temp_file_size 1024m;
proxy_buffering on;
proxy_http_version 1.1;
set $default_connection_header "";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://vs_default_cafe_tea;
proxy_next_upstream error timeout non_idempotent;
proxy_next_upstream_timeout 5s;
proxy_next_upstream_tries 10;
}
}写在最后
本文详细介绍了基于 nginx 的 ingress 实现，通过实际的案例演示 ingress 的安装部署，基于虚拟主机的配置，基于 TLS 加密实现 https，高级章节中介绍了负载均衡参数定制，自定义资源虚拟主机和虚拟路由的实现，通过该章节相信能加深对 ingress 服务暴露机制的理解。实现 ingress controller 的方式有多种，下一个章节我们将介绍基于 HAproxy 和 TKE Ingress 控制器的实现。
参考文献
Ingress 配置：https://kubernetes.io/docs/concepts/services-networking/ingress/
Ingress 控制器：https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
Nginx ingress 安装文档：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md
Nginx ingress 文档说明：https://github.com/nginxinc/kubernetes-ingress/tree/master/docs
kubernetes 系列教程 (十六) 基于 nginx ingress 实现服务暴露 - 腾讯云开发者社区 - 腾讯云
kubernetes 系列教程（一）初探 kubernetes 功能与组件
kubernetes 系列教程（二）kubeadm 离线部署 1.14.1 集群
kubernetes 系列教程（三）kubernetes 快速入门
kubernetes 系列教程（四）离线 + 在线升级 kubernetes 至 1.15
kubernetes 系列教程（六）kubernetes 资源限制和服务质量
kubernetes 系列教程（十）深入学习持久化存储 PV 和 PVC
kubernetes 系列教程（十一）深入学习 Deployment 控制器
kubernetes 系列教程（十二）详解 DaemonSet 控制器
kubernetes 系列教程（十三）一次性任务 Job 和周期任务
14 statefulset（待续）
15. services 的实现（待续）
kubernetes 系列教程 (十六) 基于 nginx ingress 实现服务暴露
kubernetes 系列教程 (十七) 基于 haproxy 实现 ingress 服务暴露
kubernetes 系列教程 (十八) TKE 中实现 ingress 服务暴露
kubernetes 系列教程 (十九) 使用 metric-server 让 HPA 弹性伸缩愉快运行
摘抄自网络，便于检索查找。
浙公网安备 33010602011771号