StatefulSet部署postgresql报错initdb: error: directory "/var/lib/postgresql/data" exists but is not empty & Back-off restarting failed container
容器状态一直重启
[root@k8s-master01 sonarqube]# kubectl get pod -n ops
NAME READY STATUS RESTARTS AGE
gitlab-0 1/1 Running 0 170m
pgsql-0 0/1 CrashLoopBackOff 20 (5m9s ago) 82m
describe查看
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 26s default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 24s default-scheduler Successfully assigned ops/pgsql-0 to k8s-node02
Normal SuccessfulAttachVolume 23s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-94dc7710-a36d-47d5-8b08-30210b63f49a"
Normal Pulled 2s (x3 over 16s) kubelet Container image "harbor.oldxu.net/ops/postgres:13.8" already present on machine
Normal Created 2s (x3 over 16s) kubelet Created container postgresql
Normal Started 2s (x3 over 16s) kubelet Started container postgresql
Warning BackOff 1s (x3 over 14s) kubelet Back-off restarting failed container
[root@k8s-master01 sonarqube]# kubectl describe pod pgsql-0 -n ops
查看容器日志
[root@k8s-master01 sonarqube]# kubectl logs pgsql-0 -n ops
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
It contains a lost+found directory, perhaps due to it being a mount point.
Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.
检查了pvc与pv
[root@k8s-master01 sonarqube]# kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ops data-gitlab-0 Bound pvc-9cda0998-983e-406d-9d6a-e05ded53ff11 50Gi RWO csi-rbd-sc 3h39m
ops db-pgsql-0 Bound pvc-9473f390-9e68-4f06-b0fe-c8bb5f45b180 50Gi RWO csi-rbd-sc 16m
[root@k8s-master01 sonarqube]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-9473f390-9e68-4f06-b0fe-c8bb5f45b180 50Gi RWO Delete Bound ops/db-pgsql-0 csi-rbd-sc 16m
pvc-9cda0998-983e-406d-9d6a-e05ded53ff11 50Gi RWO Delete Bound ops/data-gitlab-0 csi-rbd-sc 3h39m
解决方法
根据pod的logs说明
使用 PVC 创建的卷默认会有一个 lost+found 目录，所以导致了这个错误
方法一
volumeMounts 中新增 subPath 参数
volumeMounts:
- name: postgresql
mountPath: /var/lib/postgresql/data
subPath: data
yaml文件
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: pgsql
namespace: ops
spec:
serviceName: "pgsql-svc"
selector:
matchLabels:
app: pgsql
template:
metadata:
labels:
app: pgsql
spec:
imagePullSecrets:
- name: harbor-admin
containers:
- name: postgresql
image: harbor.wjl.net/ops/postgres:13.8
imagePullPolicy: IfNotPresent
env:
- name: POSTGRES_DB # 数据库
value: sonardb
- name: POSTGRES_USER # 用户
value: sonar
- name: POSTGRES_PASSWORD # 密码
value: "123456"
ports:
- containerPort: 5432
volumeMounts:
- name: db
mountPath: /var/lib/postgresql/data
subPath: data
volumeClaimTemplates:
- metadata:
name: db
labels:
app: pgsql
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: csi-rbd-sc
resources:
requests:
storage: 50Gi
方法二
在部署的清单文件上添加了 initContainers 配置
initContainers:
- command:
- rm
- -fr
- /var/lib/postgresql/data/lost+found
image: harbor.wjl.net/ops/busybox:1.32
imagePullPolicy: IfNotPresent
name: remove-lost-found
resources:
requests:
cpu: 10m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: db
imagePullSecrets:
- name: harbor-admin
yaml文件
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresql
namespace: ops
spec:
serviceName: "pgsql-svc"
selector:
matchLabels:
app: pgsql
template:
metadata:
labels:
app: pgsql
spec:
imagePullSecrets:
- name: harbor-admin
containers:
- name: postgresql
image: harbor.wjl.net/ops/postgres:13.8
imagePullPolicy: IfNotPresent
env:
- name: POSTGRES_DB # 数据库
value: sonardb
- name: POSTGRES_USER # 用户
value: sonar
- name: POSTGRES_PASSWORD # 密码
value: "123456"
ports:
- containerPort: 5432
volumeMounts:
- name: db
mountPath: /var/lib/postgresql/data
#subPath: data
#mountPath: /var/lib/postgresql/data
initContainers:
- command:
- rm
- -fr
- /var/lib/postgresql/data/lost+found
image: harbor.wjl.net/ops/busybox:1.32
imagePullPolicy: IfNotPresent
name: remove-lost-found
resources:
requests:
cpu: 10m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: db
imagePullSecrets:
- name: harbor-admin
volumeClaimTemplates:
- metadata:
name: db
labels:
app: pgsql
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: csi-rbd-sc
resources:
requests:
storage: 50Gi
