Kubernetes-1.16部署之三 Traefik 2.0

一、创建Traefik CRD资源

traefik v2.0 版本后，开始使用CRD（Custom Resource Definition）来完成路由配置

# mkdir -p /opt/kubernetes/traefik/yaml
# cd /opt/kubernetes/traefik/yaml

# vi crd.yaml
## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

创建Rraefik CRD资源

# kubectl create -f crd.yaml
customresourcedefinition.apiextensions.k8s.io/ingressroutes.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/ingressroutetcps.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/middlewares.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/tlsoptions.traefik.containo.us created

二、创建Traefik RBAC权限

# vi rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

创建Traefik RBAC资源

# kubectl create -f rbac.yaml
serviceaccount/traefik-ingress-controller created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created

三、创建Traefik配置文件

用DaemonSet方式部署，便于在多服务器间扩展

# vi traefik.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      restartPolicy: Always
      tolerations:
      - operator: "Exists"
      containers:
      - image: traefik:v2.0.7
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 1024Mi
        ports:
        - name: web
          containerPort: 80
          hostPort: 80
        - name: websecure
          containerPort: 443
          hostPort: 443
        - name: mysql
          containerPort: 3306
          hostPort: 3306
        - name: redis
          containerPort: 6379
          hostPort: 6379
        - name: admin
          containerPort: 8080
          hostPort: 9999
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --entrypoints.web.Address=:80
        - --entrypoints.websecure.Address=:443
        - --entrypoints.mysql.Address=:3306
        - --entrypoints.redis.Address=:6379
        - --providers.kubernetescrd
        - --api
        - --api.dashboard=true
        - --api.insecure=true
        - --metrics.prometheus=true
        - --tracing.zipkin=true
        - --accesslog
        - --accesslog.filepath=/var/log/access.log
      nodeSelector:
        edgenode: "true"

---

kind: Service
apiVersion: v1
metadata:
  name: traefik
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - name: admin
      port: 9999
      protocol: TCP

创建Traefik资源

# kubectl create -f traefik.yaml
daemonset.apps/traefik created
service/traefik created

四、设置节点Label标签

由于是使用Kubernetes DeamonSet这种方式部署traefik，所以需要提前给节点设置label，这样当程序部署时pod会自动调度到设置label的点上

# kubectl get nodes --show-labels
NAME           STATUS   ROLES    AGE     VERSION   LABELS
192.168.168.3  Ready    <none>   4d11h   v1.16.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.168.3,kubernetes.io/os=linux
192.168.168.4  Ready    <none>   4d11h   v1.16.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.168.4,kubernetes.io/os=linux

# kubectl label nodes 192.168.168.3 edgenode=true
node/192.168.168.3 labeled

注：如想删除label标签使用如下命令

# kubectl label nodes 192.168.168.3 edgenode-

五、配置Traefik路由规则

想让外部访问Kubernetes内部服务，需要配置路由规则，这里配置Traefik Dashboard的路由规则，使外部能够访问Traefik Dashboard

# vi IngressRoute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-webui
  namespace: kube-system
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`traefik.k8s.local`)
    kind: Rule
    services:
    - name: traefik
      port: 9999

注:Host(` `)中的内容也可为自定义域名，如配置为traefik.

创建Traefik Dashboard https协议路由规则对象

# kubectl create -f IngressRoute.yaml
ingressroute.traefik.containo.us/traefik-webui created

注：卸载Traefik时先卸载IngressRoute.yaml再卸载其它资源

在管理机hosts里配置映射192.168.168.3 traefik.k8s.local(或搭建内网DNS服务器)，然后在浏览器中输入http://traefik.k8s.local:9999

六、配置多边缘节点高可用

1）在work-node01/02上安装keepalived｛安装过程此文略过｝

设置一个VIP IP指向自定义域名traefik.k8s.local，本文示例为192.168.168.100，这样集群外部就可以通过service的DNS映射名称来访问服务。

2）设置节点Label标签

# kubectl label nodes 192.168.168.3 edgenode=true
# kubectl label nodes 192.168.168.4 edgenode=true

3）查看DaemonSet启动情况

# kubectl -n kube-system get ds
NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
traefik                  2         2         2       2            2           edgenode=true                 16m

4）配置keepalived

# vi $KEEPALIVED_HOME/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from k8s_admin@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 6666
    }
    virtual_ipaddress {
        192.168.168.100
    }
}

virtual_server 192.168.168.100 9999{
    delay_loop 6
    lb_algo loadbalance
    lb_kind DR
    nat_mask 255.255.255.0
    persistence_timeout 0
    protocol TCP

    real_server 192.168.168.3 9999{
        weight 1
        TCP_CHECK {
        connect_timeout 3
        }
    }
    real_server 192.168.168.4 9999{
        weight 1
        TCP_CHECK {
        connect_timeout 3
        }
    }
}

注：real_server的IP和端口即traefik供外网访问的IP和端口

在管理机hosts里配置映射192.168.168.100 traefik.k8s.local，然后在在浏览器中输入http://traefik.k8s.local