⑨.kubernetes 之ingress
一.Ingress基本概念
二.Ingress安装与配置
三.Ingress快速入门实践
四.Ingress基于URL实现路由
五.Ingress基于名称虚拟主机
六.Ingress实现HTTPS
七.Ingress Rewrite
八.Ingress灰度发布
-
8.1 灰度发布介绍
-
8.2 部署生产应用1.0
-
8.3 部署生产应用1.1
-
8.4 配置生产环境Ingress
-
8.5 配置权重灰度发布的Ingress
-
8.6 基于Header灰度发布
-
8.7 基于Cookies灰度发布
============================================
一、Ingress基本概念
1.1 为什么需要ingress
Kubernetes暴露服务的方式目前只有三种：LoadBlancer Service、ExternalName、NodePort Service、Ingress；而我们需要将集群内服务提供外界访问就会产生以下几个问题：
1、Pod 漂移问题
Kubernetes 具有强大的副本控制能力，能保证在任意副本（Pod）挂掉时自动从其他机器启动一个新的，还可以动态扩容等，通俗地说，这个 Pod 可能在任何时刻出现在任何节点上，也可能在任何时刻死在任何节点上；那么自然随着 Pod 的创建和销毁，Pod IP 肯定会动态变化；那么如何把这个动态的 Pod IP 暴露出去？这里借助于 Kubernetes 的 Service 机制，Service 可以以标签的形式选定一组带有指定标签的 Pod，并监控和自动负载他们的 Pod IP，那么我们向外暴露只暴露 Service IP 就行了；这就是 NodePort 模式：即在每个节点上开起一个端口，然后转发到内部 Pod IP 上，如下图所示：
此时的访问方式：http://nodeip:nodeport/
2、端口管理问题
采用 NodePort 方式暴露服务面临问题是，服务一旦多起来，NodePort 在每个节点上开启的端口会及其庞大，而且难以维护；这时，我们可以能否使用一个Nginx直接对内进行转发呢？众所周知的是，Pod与Pod之间是可以互相通信的，而Pod是可以共享宿主机的网络名称空间的，也就是说当在共享网络名称空间时，Pod上所监听的就是Node上的端口。那么这又该如何实现呢？简单的实现就是使用 DaemonSet 在每个 Node 上监听 80，然后写好规则，因为 Nginx 外面绑定了宿主机 80 端口（就像 NodePort），本身又在集群内，那么向后直接转发到相应 Service IP 就行了，如下图所示：
3、域名分配及动态更新问题
从上面的方法，采用 Nginx-Pod 似乎已经解决了问题，但是其实这里面有一个很大缺陷：当每次有新服务加入又该如何修改 Nginx 配置呢？？我们知道使用Nginx可以通过虚拟主机域名进行区分不同的服务，而每个服务通过upstream进行定义不同的负载均衡池，再加上location进行负载均衡的反向代理，在日常使用中只需要修改nginx.conf即可实现，那在K8S中又该如何实现这种方式的调度呢？？？
假设后端的服务初始服务只有ecshop，后面增加了bbs和member服务，那么又该如何将这2个服务加入到Nginx-Pod进行调度呢？总不能每次手动改或者Rolling Update 前端 Nginx Pod 吧！！此时 Ingress 出现了，如果不算上面的Nginx，Ingress 包含两大组件：Ingress Controller 和 Ingress。
1.2 什么是ingress
Ingress是Kubernetes中的一种资源，它主要是用来定义流量转发规则。但Ingress资源自身并不能实现流量的转发和调度，它仅仅是一组流量路由的规则集合，这些规则要真正发挥作用还需要使用到Ingress控制器，由Ingress控制器读取对应的Ingress规则，然后完成流量的路由转发。
Ingress 简单的理解就是你原来需要改 Nginx 配置，然后配置各种域名对应哪个 Service，现在把这个动作抽象出来，变成一个 Ingress 对象，你可以用 yaml 创建，每次不要去改 Nginx 了，直接改 yaml 然后创建/更新就行了；那么问题来了：”Nginx 该怎么处理？”
1.3 Ingress Controller
Ingress Controller就是一类以代理http、https协议为主的代理程序。如：Nginx Traefix envoy haproxy。ingress controller通过Pod的形式运行在kubernetes集群上，它能够与集群上的Pod直接通讯。这样就可以让用户的流量经过ingress控制器时直接调度到对应的Pod
Ingress Controoler 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取他，按照他自己模板生成一段 Nginx 配置，再写到 Nginx Pod 里，最后 reload 一下，工作流程如下图：
实际上Ingress也是Kubernetes API的标准资源类型之一，它其实就是一组基于DNS名称（host）或URL路径把请求转发到指定的Service资源的规则。用于将集群外部的请求流量转发到集群内部完成的服务发布。我们需要明白的是，Ingress资源自身不能进行“流量穿透”，仅仅是一组规则的集合，这些集合规则还需要其他功能的辅助，比如监听某套接字，然后根据这些规则的匹配进行路由转发，这些能够为Ingress资源监听套接字并将流量转发的组件就是Ingress Controller。
PS：Ingress 控制器不同于Deployment 控制器的是，Ingress控制器不直接运行为kube-controller-manager的一部分，它仅仅是Kubernetes集群的一个附件，类似于CoreDNS，需要在集群上单独部署。
二.Ingress安装与配置
2.1 部署Ingress controller
下载ingress-nginx部署文件
[root@master-node1 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.0/deploy/static/provider/baremetal/deploy.yaml
2.2 配置ingress
[root@k8s-240 ingress]# cat deploy.yaml
apiVersion: v1
kind: Namespace
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- coordination.k8s.io
resourceNames:
- ingress-nginx-leader
resources:
- leases
verbs:
- get
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: v1
data:
allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-controller
namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: ClusterIP #改成ClusterIP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
ports:
- appProtocol: https
name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: ClusterIP
---
apiVersion: apps/v1
kind: DaemonSet #用DaemonSet确保每个节点都部署Ingress
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
minReadySeconds: 0
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
spec:
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
image: harbor.faone.cn:180/app/ingress-controller:v1.8.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 90Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 101
volumeMounts:
- mountPath: /usr/local/certificates/
name: webhook-cert
readOnly: true
dnsPolicy: ClusterFirstWithHostNet #优先使用集群内的DNS解析服务
hostNetwork: true #将80和443监听在宿主机节点上(自行添加)
nodeSelector: #节点选择器(选择哪些节点部署Ingress,默认所有)
node-role: ingress #如果节点有node-role=ingress 并且os=linux的标签，则在节点上运行ingress POD
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission-create
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission-create
spec:
containers:
- args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: harbor.faone.cn:180/app/kube-webhook-certgen:latest
imagePullPolicy: IfNotPresent
name: create
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission-patch
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission-patch
spec:
containers:
- args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: harbor.faone.cn:180/app/kube-webhook-certgen:latest
imagePullPolicy: IfNotPresent
name: patch
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: nginx
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.8.0
name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
service:
name: ingress-nginx-controller-admission
namespace: ingress-nginx
path: /networking/v1/ingresses
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.nginx.ingress.kubernetes.io
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None
2.3 节点打标签
1.为节点打上对应标签，否则Ingress无法正常调度到指定的节点运行
[root@master ingress]# kubectl label node node1 node-role=ingress
node/node1 Labeled
[root@master ingress]# kubectl label node node2 node-role=ingress
node/node2 Labeled
2.检查ingress-controller状态
[root@master ingress]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-4gx6h 1/1 Running 0 3m47s
ingress-nginx-controller-dkwqd 1/1 Running 0 3m47s
三.Ingress快速入门实践
3.1 Ingress资源清单
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: <string> # 资源名称
namespace: <string> # 名称空间
spec:
ingressClassName: "nginx" # Ingress控制器类别，必须明确指定
rules:
- host: <string> # 虚拟主机的FQDN（域名）
http:
paths:
- path: <string> # 路径匹配规则
pathType: <string> # 路径匹配类型（Prefix前缀匹配，Exact精确匹配URL）
backend:
service:
name: <string> # 后端Service的名称
port:
number: <integer> # 后端Service的端口号
3.2 Ingress发布业务
编写Ingress规则，将此前的guestBooks业务通过域名方式发布
[root@master host]# cat books-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: books-ingress
spec:
ingressClassName: "nginx"
rules:
- host: books.oldxu.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: guestbooks
port:
number: 80
3.3 客户端测试访问
配置windows，将域名解析安装了Ingress节点的地址，然后测试访问
四.Ingress基于URL实现路由
场景： 将来自同一域名，不同URL请求调度到不通Service
4.1 部署demoapp应用
cat app-deploy-service.yaml
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-prod
spec:
replicas: 2
selector:
matchLabels:
role: python
template:
metadata:
labels:
role: python
spec:
containers:
- name: app
image: oldxu3957/demoapp:v1.0
ports:
- containerPort: 80
---
# Service
apiVersion: v1
kind: Service
metadata:
name: app-service
spec:
selector:
role: python
ports:
- port: 80
targetPort: 80
4.2 部署tomcat应用
cat java-deploy-service.yaml
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-prod
spec:
replicas: 2
selector:
matchLabels:
role: java
template:
metadata:
labels:
role: java
spec:
containers:
- name: tomcat
image: tomcat:9.8.63
ports:
- containerPort: 8080
lifecycle:
postStart:
exec:
command: ["/bin/bash", "-c", "cp -rf /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps"]
---
# Service
apiVersion: v1
kind: Service
metadata:
name: java-service
spec:
selector:
role: java
ports:
- port: 8080
targetPort: 8080
4.3 配置Ingress
1.编写Yaml
cat ingress-app-java.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-ingress
spec:
ingressClassName: "nginx"
rules:
- host: foo.oldxu.net
http:
paths:
- path: /app
pathType: Prefix
backend:
service:
name: app-service
port:
number: 80
- path: /java
pathType: Prefix
backend:
service:
name: java-service
port:
number: 8080
2.访问/app会出现404错误，因为后端的Pod无法处理/app这样的接口，所以需要调整代理到后端的路径;
- 默认URL: 用户请求foo.oldxu.net/app，代理到后端请求也会带上/app，后端无法处理该url，就会报错
- 修改URL: 用户请求foo.oldxu.net/app，代理到后端后，将请求的/app删除，url为foo.oldxu.net/
cat ingress-app-java.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2 # 配置 Rewrite 规则
spec:
ingressClassName: "nginx"
rules:
- host: foo.oldxu.net
http:
paths:
- path: /app(/|$)(.*)
pathType: Prefix
backend:
service:
name: app-service
port:
number: 80
- path: /java(/|$)(.*)
pathType: Prefix
backend:
service:
name: java-service
port:
number: 8080
4.4 客户端测试
五.Ingress基于名称虚拟主机
场景： 将来自不同的域名的请求调度到不同的Service
5.1 部署demoapp应用
cat app-deploy-service.yaml
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-prod
spec:
replicas: 2
selector:
matchLabels:
role: python
template:
metadata:
labels:
role: python
spec:
containers:
- name: app
image: oldxu3957/demoapp:v1.0
ports:
- containerPort: 80
---
# Service
apiVersion: v1
kind: Service
metadata:
name: app-service
spec:
selector:
role: python
ports:
- port: 80
targetPort: 80
5.2 部署tomcat应用
cat java-deploy-service.yaml
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-prod
spec:
replicas: 2
selector:
matchLabels:
role: java
template:
metadata:
labels:
role: java
spec:
containers:
- name: tomcat
image: tomcat:9.8.63
ports:
- containerPort: 8080
lifecycle:
postStart:
exec:
command:
- "/bin/bash"
- "-c"
- "cp -rf /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps"
---
# Service
apiVersion: v1
kind: Service
metadata:
name: java-service
spec:
selector:
role: java
ports:
- port: 8888
targetPort: 8888
5.3 配置Ingress
cat ingress-domain.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wirtual-host-ingress
spec:
ingressClassName: "nginx"
rules:
- host: app.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: app-service
port:
number: 80
- host: java.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: java-service
port:
number: 8080
5.4 客户端测试
六.Ingress实现HTTPS
在Ingress中引用Secret资源，然后告诉Ingress控制器使用TLS加密从客户端到负载均衡器的通道。
6.1 创建TLS证书
openssl genrsa -out java.key 2048
openssl req -new -x509 -key java.key -out java.crt -subj "/C=CN/ST=Beijing/L=Beijing/O=oldxu/CN=java.oldxu.net"
6.2 创建Secrets
kubectl create secret tls java-oldxu-tls --key=java.key --cert=java.crt
6.3 配置Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: java-ingress-tls
spec:
ingressClassName: "nginx"
tls:
- hosts:
- java.oldxu.net
secretName: java-oldxu-tls
rules:
- host: java.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: java-service
port:
number: 8080
6.4 客户端测试
七.Ingress Rewrite
7.1 Rewrite示例1
cat ingress-domain-tls-2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: java-ingress-tls
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
ingressClassName: "nginx"
tls:
- hosts:
- java.oldxu.net
secretName: java-oldxu-tls
rules:
- host: java.oldxu.net
http:
paths:
- pathType: Prefix
path: "/java(/|$)(.*)"
backend:
service:
name: java-service
port:
number: 8080
7.2 Rewrite示例2
cat ingress-domain-tls-2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: java-ingress-tls
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/configuration-snippet: |
# 自定义跳转
rewrite ^/manager/(.*)$ /java/manager/$1 redirect;
rewrite ^/host-manager/(.*)$ /java/host-manager/$1 redirect;
rewrite ^/docs/(.*)$ /java/docs/$1 redirect;
spec:
ingressClassName: "nginx"
tls:
- hosts:
- java.oldxu.net
secretName: java-oldxu-tls
rules:
- host: java.oldxu.net
http:
paths:
- pathType: Prefix
path: "/java(/|$)(.*)"
backend:
service:
name: java-service
port:
number: 8080
八.Ingress灰度发布
8.1 灰度发布介绍
Ingress实现灰度发布可以基于权重，也可以基于用户请求来实现
- 基于权重发布
# 灰度版本的annotations
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "30"
|70%------> 生产版本
users --- 100% ---> Nginx Ingress---- |
|30%------> 灰度版本
- 基于用户header发布
# 灰度版本的annotations
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "deploy"
nginx.ingress.kubernetes.io/canary-by-header-value: "new"
others
|----------------> 生产
users -----> Nginx Ingress -----|"deploy:newusers
|----------------> 灰度版本
8.2 部署生产应用1.0
apiVersion: apps/v1
kind: Deployment
metadata:
name: demoapp-prod
spec:
replicas: 2
selector:
matchLabels:
app: demoapp
version: v1.0
template:
metadata:
labels:
app: demoapp
version: v1.0
spec:
containers:
- name: web
image: oldxu3957/demoapp:v1.0
imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: demoapp-prod
spec:
selector:
app: demoapp
version: v1.0
ports:
- port: 80
targetPort: 80
8.3 部署生产应用1.1
apiVersion: apps/v1
kind: Deployment
metadata:
name: demoapp-canary
spec:
replicas: 2
selector:
matchLabels:
app: demoapp
version: v1.1
template:
metadata:
labels:
app: demoapp
version: v1.1
spec:
containers:
- name: web
image: oldxu3957/demoapp:v1.1
imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: demoapp-canary
spec:
selector:
app: demoapp
version: v1.1
ports:
- port: 80
targetPort: 80
8.4 配置生产环境Ingress
cat demoapp-ingress-prod.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demoapp-ingress-prod
spec:
ingressClassName: "nginx"
rules:
- host: demoapp.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: demoapp-prod
port:
number: 80
8.5 配置权重灰度发布的Ingress
cat demoapp-ingress-canary.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demoapp-ingress-canary
annotations:
nginx.ingress.kubernetes.io/canary: "true" # 启用灰度发布
nginx.ingress.kubernetes.io/canary-weight: "30" # 分配30%流量到当前canary版本
spec:
ingressClassName: "nginx"
rules:
- host: demoapp.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: demoapp-canary
port:
number: 80
2.检查对应的Ingress详情
Name: demoapp-ingress-canary
Namespace:
Address:
Default backend: default-http-backend:80 (<error:endpoints "default-http-backend" not found>)
Rules:
Path Backends
---- ---------
1 demoapp.oldxu.net demoapp-canary:80(192.168.2.77:80,192.168.2.78:80)
Annotations:
nginx.ingress.kubernetes.io/canary: true
nginx.ingress.kubernetes.io/canary-weight: 30
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 1s (x4 over 3m36s) nginx-ingress-controller Scheduled for sync
Normal Sync 1s (x4 over 3m36s) nginx-ingress-controller Scheduled for sync
3.通过For循环访问10次，会发现有30%的流量调度到新版本
[root@node1 ~]# for i in [1..10]; do curl -H"Host:demoapp .oldxu . net" http://10.0.0.211/version;done
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.1!! PodIP: 192.168.2.77!
demoapp v1.0!! PodIP: 192.168.2.76!
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.0!! PodIP: 192.168.2.76!
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.0!! PodIP: 192.168.2.76!
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.1!! PodIP: 192.168.2.78!
8.6 基于Header灰度发布
基于Header的方式优先级比较高，所以会忽略weight权重的配置。但当Header无法匹配时，则会按照权重进行匹配。
1.配置yaml
cat demoapp-ingress-canary.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demoapp-ingress-canary
annotations:
nginx.ingress.kubernetes.io/canary: "true" # 启用灰度发布
nginx.ingress.kubernetes.io/canary-by-header: "deploy" # Header的key
nginx.ingress.kubernetes.io/canary-by-header-value: "new" # Header的value
nginx.ingress.kubernetes.io/canary-weight: "30" # 分配30%流量到当前canary版本
spec:
ingressClassName: "nginx"
rules:
- host: demoapp.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: demoapp-canary
port:
number: 80
2、如果将depLoy的Header设定为old，它无法匹配到canary版本，然后继续匹配规则，能匹配到权重。
[root@node1 ~]# for i in {1..10]; do curl -H "deploy:old" -H "Host:demoapp.oldxu.net" http://10.0.0.211/version; done
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp 1.1!! PodIP: 192.168.2.77!
demoapp v1.0!! PodIP: 192.168.2.76!
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp 1.0!! PodIP: 192.168.2.76!
demoapp v1.0!! PodIP: 192.168.2.75!
demoapp v1.0!! PodIP: 192.168.2.76!
demoapp v1.0!! PodIP: 192.168.2.75!
3、如果将请求的Header修改为“depLoy:new"，则会百分百请求到canary版本
[root@node1 ~]# for i in {1..10]; do curl -H "deploy:new" -H "Host:demoapp.oldxu.net" http://10.0.0.211/version; done
demoapp v1.1!! PodIP: 192.168.2.77!
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp v1.1!! PodIP: 192.168.2.77!
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp v1.1!! PodIP: 192.168.2.77!
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp v1.1!! PodIP: 192.168.2.77!
demoapp v1.1!! PodIP: 192.168.2.78!
demoapp v1.1!! PodIP: 192.168.2.77!
demoapp v1.1!! PodIP: 192.168.2.78!
8.7 基于Cookies灰度发布
假设我们需要让武汉的用户访问Canary版本，通过后台检查登录的用户IP归属地，如果来源IP归属为武汉，则为其设定对应的Cookies，比如request_from_wh，也就是说，只要来源的IP归属为武汉，那么程序会自动添加这个Cookies，并将对应的值设定为 always，这样，这个区域的用户会被直接调度到Canary版本;
1.修改Ingress资源对象
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demoapp-ingress-canary
annotations:
nginx.ingress.kubernetes.io/canary: "true" # 启用灰度发布
nginx.ingress.kubernetes.io/canary-by-cookie: "request_from_wh" # 基于cookie
spec:
ingressClassName: "nginx"
rules:
- host: demoapp.oldxu.net
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: demoapp-canary
port:
number: 80
2、请求测试，值为always则调度Canary版本，值为never则调度正常版本;
[root@node1 ~] for i in {1..10}; do curl -s -b "request_from_wh=always" -H "Host: demoapp.oldxu.net" http://<your-ingress-ip>/; done
demoapp v1.1!! PodIP: 192.168.3.73!
demoapp v1.1!! PodIP: 192.168.2.250!
demoapp v1.1!! PodIP: 192.168.2.251!
demoapp v1.1!! PodIP: 192.168.3.73!
demoapp v1.1!! PodIP: 192.168.2.250!
demoapp v1.1!! PodIP: 192.168.2.251!
demoapp v1.1!! PodIP: 192.168.3.73!
demoapp v1.1!! PodIP: 192.168.2.250!
demoapp v1.1!! PodIP: 192.168.2.251!
demoapp v1.1!! PodIP: 192.168.3.73!
