k8s部署nginx集群
1.nginx集群介绍
在Kubernetes（k8s）中部署Nginx集群，是通过声明式配置实现高可用、可扩展的Web服务。其核心是通过Deployment管理Nginx容器副本的自动扩缩容和故障恢复，并借助Service提供负载均衡和统一的访问入口。Kubernetes的调度机制保障了集群的弹性与稳定性，支持滚动更新、资源限制、健康检查等关键特性，适用于生产环境的大规模流量分发与业务托管。
2.部署环境
| IP | 节点 | 操作系统 | k8s版本 |
nginx版本 |
docker版本 |
| 172.16.4.85 | master1 | centos7.8 | 1.23.17 | 20.10.9 | |
| 172.16.4.86 | node1 | centos7.8 | 1.23.17 | 20.10.9 | |
| 172.16.4.87 | node2 | centos7.8 | 1.23.17 | 1.20.1 | 20.10.9 |
| 172.16.4.89 | node3 | centos7.8 | 1.23.17 | 1.20.1 | 20.10.9 |
| 172.16.4.90 | node4 | centos7.8 | 1.23.17 | 1.20.1 | 20.10.9 |
3.nginx集群部署
3.1 nfs csi部署
https://www.cnblogs.com/Leonardo-li/p/188131403.2 创建namespace
kubectl create namespace nginx3.3 创建nginx configmap
# nginx-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
namespace: nginx
data:
nginx.conf: |
user nginx;
worker_processes auto;
# 日志路径指向 /data/service/nginx/logs
error_log /data/service/nginx/logs/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
server_tokens off;
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /data/service/nginx/logs/access.log main;
sendfile on;
keepalive_timeout 65;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream ltas-server {
server lt-algstore.ltzx.svc.cluster.local:8080;
}
upstream minio-server {
server minio-svc.minio.svc.cluster.local:9001;
}
server {
listen 80;
server_name localhost;
client_max_body_size 1024M;
location / {
root /data/service/nginx/html;
try_files $uri $uri/ /index.html;
}
location /prod-api/ {
proxy_pass http://ltas-server/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
location /prod-ws/ {
proxy_pass http://ltas-server/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_read_timeout 600s;
}
location /prod-file/ {
proxy_pass http://minio-server/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}3.4 创建nginx html pvc
# html-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-html-pvc
namespace: nginx
spec:
accessModes:
- ReadWriteMany # 必须为多节点读写
storageClassName: nfs-csi
resources:
requests:
storage: 10Gi3.5 创建nginx service
# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: nginx
spec:
type: NodePort
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30080 # 根据需求调整端口范围（30000-32767）3.6 创建nginx statefulset
- 此段配置，是因为我在做nginx镜像的时候，没有将日志输出到前台，所以在执行kubectl logs的时候是没有输出的，所以在增加此段配置，来输出nginx日志
- command: ["/bin/sh", "-c"]
args:
- |
mkdir -p /data/service/nginx/logs
ln -sf /dev/stdout /data/service/nginx/logs/access.log
ln -sf /dev/stderr /data/service/nginx/logs/error.log
exec nginx -g 'daemon off;'
# ng-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nginx
namespace: nginx
spec:
serviceName: nginx
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
securityContext:
fsGroup: 1000
containers:
- name: nginx
image: 10.142.99.123:8060/public/nginx:v1.20.1
env:
- name: TZ
value: "Asia/Shanghai"
ports:
- containerPort: 39988
command: ["/bin/sh", "-c"]
args:
- |
mkdir -p /data/service/nginx/logs
ln -sf /dev/stdout /data/service/nginx/logs/access.log
ln -sf /dev/stderr /data/service/nginx/logs/error.log
exec nginx -g 'daemon off;'
volumeMounts:
- name: host-timezone
mountPath: /etc/localtime
readOnly: true
- name: html-shared
mountPath: /data/service/nginx/html
- name: logs-volume
mountPath: /data/service/nginx/logs
- name: nginx-config
mountPath: /data/service/nginx/conf/nginx.conf
subPath: nginx.conf
# 关键修复：volumes 必须放在 Pod 模板内部
volumes:
- name: host-timezone
hostPath:
path: /etc/localtime
- name: html-shared
persistentVolumeClaim:
claimName: nginx-html-pvc
- name: nginx-config
configMap:
name: nginx-config
# volumeClaimTemplates 保持在 StatefulSet 顶层
volumeClaimTemplates:
- metadata:
name: logs-volume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: nfs-csi
resources:
requests:
storage: 50Gi3.7 执行并创建各种资源类
kubectl apply -f ng-cm.yaml
kubectl apply -f ng-html-pvc.yaml
kubectl apply -f ng-svc.yaml
kubectl apply -f ng-sts.yaml 3.8 验证状态
[root@master1 nginx-n6]# kubectl get pv | grep nginx
pvc-1200d1d4-6186-4629-9980-5372f3a7584c 50Gi RWO Retain Bound nginx/logs-volume-nginx-1 nfs-csi 48m
pvc-48f293ad-a6ae-4b57-883d-59e6797ce165 50Gi RWO Retain Bound nginx/logs-volume-nginx-2 nfs-csi 48m
pvc-6baae14c-0f7f-4251-8a1f-4606194677e7 10Gi RWX Retain Bound nginx/nginx-html-pvc nfs-csi 52m
pvc-dc0037af-7a9e-4547-9ea9-f3ecf692c335 50Gi RWO Retain Bound nginx/logs-volume-nginx-0 nfs-csi 48m
[root@master1 nginx-n6]# kubectl get pvc -n nginx
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
logs-volume-nginx-0 Bound pvc-dc0037af-7a9e-4547-9ea9-f3ecf692c335 50Gi RWO nfs-csi 48m
logs-volume-nginx-1 Bound pvc-1200d1d4-6186-4629-9980-5372f3a7584c 50Gi RWO nfs-csi 48m
logs-volume-nginx-2 Bound pvc-48f293ad-a6ae-4b57-883d-59e6797ce165 50Gi RWO nfs-csi 48m
nginx-html-pvc Bound pvc-6baae14c-0f7f-4251-8a1f-4606194677e7 10Gi RWX nfs-csi 52m
[root@master1 nginx-n6]# kubectl get sts -n nginx
NAME READY AGE
nginx 3/3 47m
[root@master1 nginx-n6]# kubectl get pods -n nginx
NAME READY STATUS RESTARTS AGE
nginx-0 1/1 Running 0 47m
nginx-1 1/1 Running 0 47m
nginx-2 1/1 Running 0 47m
[root@master1 nginx-n6]# kubectl get svc -n nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx NodePort 10.102.171.84 <none> 80:30080/TCP 88m3.9 前端页面目录
- 因为我的nginx是需要做web服务器的，所以将业务的前端放到nfs csi的自动创建的pvc（nginx-html-pvc）中，它对应的pv是挂载到容器中的html目录的，这样就可以正常访问了nginx发布的web前端了。
浙公网安备 33010602011771号