Kubernetes（k8s）之分布式文件系统glusterfs
1、为什么使用分布式文件系统glusterfs？
答：使用持久化存储nfs，但是使用nfs这个共享存储特别简单，但是后期在扩容和管理方面都特别的不方便，在生产中一般都是使用分布式文件系统，这里使用的是分布式文件系统glusterfs。
2、什么是分布式文件系统glusterfs？
答：分布式文件系统glusterfs是一个开源分布式文件系统，具有强大的横向扩展能力，可以支持数PB存储容器和数千客户端，通过网络互联成一个并行的网络文件系统。具有可扩展性、高性能、高可用性等特点。
3、安装分布式文件系统glusterfs，需要所有节点都需要进行安装的，如下所示：
首先在三台节点上都安装glusterfs的yum源，如下所示：
1 [root@k8s-master ~]# yum install centos-release-gluster -y 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.bfsu.edu.cn 8 * extras: mirrors.bfsu.edu.cn 9 * updates: mirrors.bfsu.edu.cn 10 base | 3.6 kB 00:00:00 11 extras | 2.9 kB 00:00:00 12 updates | 2.9 kB 00:00:00 13 Resolving Dependencies 14 --> Running transaction check 15 ---> Package centos-release-gluster7.noarch 0:1.0-2.el7.centos will be installed 16 --> Processing Dependency: centos-release-storage-common for package: centos-release-gluster7-1.0-2.el7.centos.noarch 17 --> Running transaction check 18 ---> Package centos-release-storage-common.noarch 0:2-2.el7.centos will be installed 19 --> Finished Dependency Resolution 20 21 Dependencies Resolved 22 23 ================================================================================================================================================================================================================= 24 Package Arch Version Repository Size 25 ================================================================================================================================================================================================================= 26 Installing: 27 centos-release-gluster7 noarch 1.0-2.el7.centos extras 5.2 k 28 Installing for dependencies: 29 centos-release-storage-common noarch 2-2.el7.centos extras 5.1 k 30 31 Transaction Summary 32 ================================================================================================================================================================================================================= 33 Install 1 Package (+1 Dependent package) 34 35 Total download size: 10 k 36 Installed size: 2.4 k 37 Downloading packages: 38 (1/2): centos-release-gluster7-1.0-2.el7.centos.noarch.rpm | 5.2 kB 00:00:00 39 (2/2): centos-release-storage-common-2-2.el7.centos.noarch.rpm | 5.1 kB 00:00:00 40 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 41 Total 13 kB/s | 10 kB 00:00:00 42 Running transaction check 43 Running transaction test 44 Transaction test succeeded 45 Running transaction 46 Installing : centos-release-storage-common-2-2.el7.centos.noarch 1/2 47 Installing : centos-release-gluster7-1.0-2.el7.centos.noarch 2/2 48 Verifying : centos-release-gluster7-1.0-2.el7.centos.noarch 1/2 49 Verifying : centos-release-storage-common-2-2.el7.centos.noarch 2/2 50 51 Installed: 52 centos-release-gluster7.noarch 0:1.0-2.el7.centos 53 54 Dependency Installed: 55 centos-release-storage-common.noarch 0:2-2.el7.centos 56 57 Complete! 58 [root@k8s-master ~]#
可以看到yum源里面多了一些文件，默认安装的是CentOS-Gluster-7.repo这个7.x版本的，如下所示：
1 [root@k8s-master ~]# ls /etc/yum.repos.d/ 2 CentOS-Base.repo CentOS-CR.repo CentOS-Debuginfo.repo CentOS-fasttrack.repo CentOS-Gluster-7.repo CentOS-Media.repo CentOS-Sources.repo CentOS-Storage-common.repo CentOS-Vault.repo 3 [root@k8s-master ~]#
然后开始在三台节点上安装glusterfs-server包安装上，最主要的就是安装这个包的，如下所示：
1 [root@k8s-master ~]# yum install install glusterfs-server -y 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.bfsu.edu.cn 8 * centos-gluster7: mirrors.tuna.tsinghua.edu.cn 9 * extras: mirrors.bfsu.edu.cn 10 * updates: mirrors.bfsu.edu.cn 11 centos-gluster7 | 3.0 kB 00:00:00 12 centos-gluster7/7/x86_64/primary_db | 69 kB 00:00:00 13 No package install available. 14 Resolving Dependencies 15 --> Running transaction check 16 ---> Package glusterfs-server.x86_64 0:7.6-1.el7 will be installed 17 --> Processing Dependency: glusterfs = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 18 --> Processing Dependency: glusterfs-api = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 19 --> Processing Dependency: glusterfs-cli = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 20 --> Processing Dependency: glusterfs-client-xlators = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 21 --> Processing Dependency: glusterfs-fuse = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 22 --> Processing Dependency: glusterfs-libs = 7.6-1.el7 for package: glusterfs-server-7.6-1.el7.x86_64 23 --> Processing Dependency: libgfapi.so.0(GFAPI_6.0)(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 24 --> Processing Dependency: libgfapi.so.0(GFAPI_PRIVATE_6.0)(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 25 --> Processing Dependency: libgfapi.so.0(GFAPI_PRIVATE_6.1)(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 26 --> Processing Dependency: liburcu-bp.so.6()(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 27 --> Processing Dependency: liburcu-cds.so.6()(64bit) for package: glusterfs-server-7.6-1.el7.x86_64 28 --> Running transaction check 29 ---> Package glusterfs.x86_64 0:3.12.2-18.el7 will be updated 30 ---> Package glusterfs.x86_64 0:7.6-1.el7 will be an update 31 ---> Package glusterfs-api.x86_64 0:3.12.2-18.el7 will be updated 32 ---> Package glusterfs-api.x86_64 0:7.6-1.el7 will be an update 33 ---> Package glusterfs-cli.x86_64 0:3.12.2-18.el7 will be updated 34 ---> Package glusterfs-cli.x86_64 0:7.6-1.el7 will be an update 35 ---> Package glusterfs-client-xlators.x86_64 0:3.12.2-18.el7 will be updated 36 ---> Package glusterfs-client-xlators.x86_64 0:7.6-1.el7 will be an update 37 ---> Package glusterfs-fuse.x86_64 0:7.6-1.el7 will be installed 38 ---> Package glusterfs-libs.x86_64 0:3.12.2-18.el7 will be updated 39 ---> Package glusterfs-libs.x86_64 0:7.6-1.el7 will be an update 40 ---> Package userspace-rcu.x86_64 0:0.10.0-3.el7 will be installed 41 --> Finished Dependency Resolution 42 43 Dependencies Resolved 44 45 ================================================================================================================================================================================================================= 46 Package Arch Version Repository Size 47 ================================================================================================================================================================================================================= 48 Installing: 49 glusterfs-server x86_64 7.6-1.el7 centos-gluster7 1.3 M 50 Installing for dependencies: 51 glusterfs-fuse x86_64 7.6-1.el7 centos-gluster7 156 k 52 userspace-rcu x86_64 0.10.0-3.el7 centos-gluster7 93 k 53 Updating for dependencies: 54 glusterfs x86_64 7.6-1.el7 centos-gluster7 640 k 55 glusterfs-api x86_64 7.6-1.el7 centos-gluster7 114 k 56 glusterfs-cli x86_64 7.6-1.el7 centos-gluster7 198 k 57 glusterfs-client-xlators x86_64 7.6-1.el7 centos-gluster7 850 k 58 glusterfs-libs x86_64 7.6-1.el7 centos-gluster7 425 k 59 60 Transaction Summary 61 ================================================================================================================================================================================================================= 62 Install 1 Package (+2 Dependent packages) 63 Upgrade ( 5 Dependent packages) 64 65 Total download size: 3.7 M 66 Downloading packages: 67 No Presto metadata available for centos-gluster7 68 warning: /var/cache/yum/x86_64/7/centos-gluster7/packages/glusterfs-cli-7.6-1.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e451e5b5: NOKEY ] 0.0 B/s | 0 B --:--:-- ETA 69 Public key for glusterfs-cli-7.6-1.el7.x86_64.rpm is not installed 70 (1/8): glusterfs-cli-7.6-1.el7.x86_64.rpm | 198 kB 00:00:00 71 (2/8): glusterfs-api-7.6-1.el7.x86_64.rpm | 114 kB 00:00:00 72 (3/8): glusterfs-libs-7.6-1.el7.x86_64.rpm | 425 kB 00:00:00 73 (4/8): glusterfs-fuse-7.6-1.el7.x86_64.rpm | 156 kB 00:00:01 74 (5/8): glusterfs-client-xlators-7.6-1.el7.x86_64.rpm | 850 kB 00:00:03 75 (6/8): glusterfs-7.6-1.el7.x86_64.rpm | 640 kB 00:00:03 76 (7/8): glusterfs-server-7.6-1.el7.x86_64.rpm | 1.3 MB 00:00:04 77 (8/8): userspace-rcu-0.10.0-3.el7.x86_64.rpm | 93 kB 00:00:13 78 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 79 Total 242 kB/s | 3.7 MB 00:00:15 80 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage 81 Importing GPG key 0xE451E5B5: 82 Userid : "CentOS Storage SIG (http://wiki.centos.org/SpecialInterestGroup/Storage) <security@centos.org>" 83 Fingerprint: 7412 9c0b 173b 071a 3775 951a d4a2 e50b e451 e5b5 84 Package : centos-release-storage-common-2-2.el7.centos.noarch (@extras) 85 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage 86 Running transaction check 87 Running transaction test 88 Transaction test succeeded 89 Running transaction 90 Updating : glusterfs-libs-7.6-1.el7.x86_64 1/13 91 Updating : glusterfs-client-xlators-7.6-1.el7.x86_64 2/13 92 Updating : glusterfs-7.6-1.el7.x86_64 3/13 93 Updating : glusterfs-api-7.6-1.el7.x86_64 4/13 94 Installing : glusterfs-fuse-7.6-1.el7.x86_64 5/13 95 Updating : glusterfs-cli-7.6-1.el7.x86_64 6/13 96 Installing : userspace-rcu-0.10.0-3.el7.x86_64 7/13 97 Installing : glusterfs-server-7.6-1.el7.x86_64 8/13 98 Cleanup : glusterfs-api-3.12.2-18.el7.x86_64 9/13 99 Cleanup : glusterfs-3.12.2-18.el7.x86_64 10/13 100 Cleanup : glusterfs-client-xlators-3.12.2-18.el7.x86_64 11/13 101 Cleanup : glusterfs-cli-3.12.2-18.el7.x86_64 12/13 102 Cleanup : glusterfs-libs-3.12.2-18.el7.x86_64 13/13 103 Verifying : glusterfs-libs-7.6-1.el7.x86_64 1/13 104 Verifying : glusterfs-api-7.6-1.el7.x86_64 2/13 105 Verifying : glusterfs-cli-7.6-1.el7.x86_64 3/13 106 Verifying : glusterfs-fuse-7.6-1.el7.x86_64 4/13 107 Verifying : glusterfs-client-xlators-7.6-1.el7.x86_64 5/13 108 Verifying : glusterfs-server-7.6-1.el7.x86_64 6/13 109 Verifying : glusterfs-7.6-1.el7.x86_64 7/13 110 Verifying : userspace-rcu-0.10.0-3.el7.x86_64 8/13 111 Verifying : glusterfs-3.12.2-18.el7.x86_64 9/13 112 Verifying : glusterfs-cli-3.12.2-18.el7.x86_64 10/13 113 Verifying : glusterfs-client-xlators-3.12.2-18.el7.x86_64 11/13 114 Verifying : glusterfs-libs-3.12.2-18.el7.x86_64 12/13 115 Verifying : glusterfs-api-3.12.2-18.el7.x86_64 13/13 116 117 Installed: 118 glusterfs-server.x86_64 0:7.6-1.el7 119 120 Dependency Installed: 121 glusterfs-fuse.x86_64 0:7.6-1.el7 userspace-rcu.x86_64 0:0.10.0-3.el7 122 123 Dependency Updated: 124 glusterfs.x86_64 0:7.6-1.el7 glusterfs-api.x86_64 0:7.6-1.el7 glusterfs-cli.x86_64 0:7.6-1.el7 glusterfs-client-xlators.x86_64 0:7.6-1.el7 glusterfs-libs.x86_64 0:7.6-1.el7 125 126 Complete! 127 [root@k8s-master ~]#
然后开始启动glusterfs，并设置开机自启动，如下所示：
1 [root@k8s-master ~]# systemctl start glusterd.service 2 [root@k8s-master ~]# systemctl enable glusterd.service 3 [root@k8s-master ~]#
生产环境可以增加一个/dev/sdb或者/dev/sdc用于挂载，将它挂载到某一个目录下面去使用的，默认只有/dev/sda。这里直接增加目录来使用了，在三台机器上都执行下面的创建目录的命令，如下所示：
1 [root@k8s-master ~]# mkdir -p /gfs/test1 2 [root@k8s-master ~]# mkdir -p /gfs/test2 3 [root@k8s-master ~]#
添加存储资源池，即将节点都组装起来，在master主节点进行操作，可以查看gluster的资源池（如果gluster没有提示，可以先退出xshell，再重新登录即可），如下所示：
1 [root@k8s-master ~]# gluster pool list 2 UUID Hostname State 3 d2ea56bf-6402-49c5-93a2-0d784110b231 localhost Connected 4 [root@k8s-master ~]# gluster pool 5 unrecognized command 6 7 Usage: gluster [options] <help> <peer> <pool> <volume> 8 Options: 9 --help Shows the help information 10 --version Shows the version 11 --print-logdir Shows the log directory 12 --print-statedumpdir Shows the state dump directory 13 14 [root@k8s-master ~]#
资源池默认只能识别出自己本身，这里需要将另外两个节点加入到资源池中，根据hostname加入到资源池，需要做host解析的，如下所示：
1 [root@k8s-master ~]# gluster peer 2 detach probe status 3 [root@k8s-master ~]# gluster peer probe k8s-node2 4 peer probe: success. 5 [root@k8s-master ~]# gluster peer probe k8s-node3 6 peer probe: success. 7 [root@k8s-master ~]#
此时，再次查看资源池，发现已经有了三个节点了，如下所示：
1 [root@k8s-master ~]# gluster pool list 2 UUID Hostname State 3 977fb392-500c-4353-b03c-0ee9e53cee29 k8s-node2 Connected 4 3b7c328f-2eff-4098-b3ac-ebc730975d22 k8s-node3 Connected 5 d2ea56bf-6402-49c5-93a2-0d784110b231 localhost Connected 6 [root@k8s-master ~]#
资源池添加完毕之后，开始进行glusterfs卷管理，创建分布式复制卷，如下所示：
可以在资源池里面使用资源池创建一个卷，在生产环境中glusterfs支持七种模式的卷，使用最多的是分布式复制卷，其他有的卷不稳定肯能会丢失书数据，最稳定的一种就是分布式复制卷，分布式复制卷要求至少四个存储单元，就是刚才创建的目录，正常情况下，一个目录就要挂载一个空的硬盘，一个硬盘就是一个存储单元，或者说一个目录就是一个存储单元，glusterfs这个分布式复制卷，最少复制的副本是2，再加上是分布式，至少要有四个目录作为存储单元。
1 [root@k8s-master ~]# gluster volume create biehl replica 2 k8s-master:/gfs/test1 k8s-master:/gfs/test2 k8s-node2:/gfs/test1 k8s-node2:/gfs/test2 force 2 volume create: biehl: success: please start the volume to access data 3 [root@k8s-master ~]#
卷已经创建成功了，现在需要启动一下才可以访问里面的数据，启动卷并进行查看卷，如下所示：
1 [root@k8s-master ~]# gluster volume start biehl 2 volume start: biehl: success 3 [root@k8s-master ~]# gluster volume info biehl 4 5 Volume Name: biehl 6 Type: Distributed-Replicate 7 Volume ID: 5729192c-627f-4444-92eb-32a245086f43 8 Status: Started 9 Snapshot Count: 0 10 Number of Bricks: 2 x 2 = 4 11 Transport-type: tcp 12 Bricks: 13 Brick1: k8s-master:/gfs/test1 14 Brick2: k8s-master:/gfs/test2 15 Brick3: k8s-node2:/gfs/test1 16 Brick4: k8s-node2:/gfs/test2 17 Options Reconfigured: 18 transport.address-family: inet 19 storage.fips-mode-rchecksum: on 20 nfs.disable: on 21 performance.client-io-threads: off 22 [root@k8s-master ~]#
下面开始挂载卷，可以在三台机器的任意一个节点上进行挂载，如下所示：
1 [root@k8s-master ~]# mount -t glusterfs 192.168.110.133:/biehl /mnt 2 [root@k8s-master ~]#
含义就是将192.168.110.133的biehl卷，挂载到mnt目录下面，现在可以进行查看，如下所示：
1 [root@k8s-master ~]# df -h 2 Filesystem Size Used Avail Use% Mounted on 3 /dev/mapper/centos-root 18G 14G 3.9G 79% / 4 devtmpfs 1.2G 0 1.2G 0% /dev 5 tmpfs 1.2G 0 1.2G 0% /dev/shm 6 tmpfs 1.2G 40M 1.1G 4% /run 7 tmpfs 1.2G 0 1.2G 0% /sys/fs/cgroup 8 /dev/sda1 197M 157M 41M 80% /boot 9 tmpfs 229M 12K 229M 1% /run/user/42 10 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/fbf4133aed53d4e17a706be46fadcbea9fe6eb1e615379bfaa157570f0de7bc6/merged 11 shm 64M 0 64M 0% /var/lib/docker/containers/5e72b0961647003c757fd41ffe824027c857d1f3938b9a53535645f4f372a40a/shm 12 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/80967f34f76b4afe42411f1b1dc6514b2b19d997644dfa4687a3a0924df06a08/merged 13 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/e085a4fba182d19139db32c664fcc1f08895ca603db3b1f9855183fc071c1adc/merged 14 shm 64M 0 64M 0% /var/lib/docker/containers/276a98c9d5ccd61f42a0a1ef55c30f76beb2977483ed2d79281dfcec79922029/shm 15 shm 64M 0 64M 0% /var/lib/docker/containers/0f89914f64c09c0fbd777e637d8f987414420861628d7b78f8a1bafc9054df59/shm 16 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/23afbc15307f6e5b397db4b1fd0e124c943bdb95312234e8d41e4f80a104b0cd/merged 17 shm 64M 0 64M 0% /var/lib/docker/containers/f686976b5743236b368b8b70e222d230b737612acadc716ca4cbabcd9a187011/shm 18 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/15696b3573a066b5d3fd7aa202086b2bcc4145d9d0bb46e54c7d8d7787d1cc73/merged 19 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/6d9b6978705654a815dd1e72d5501408cead42493c25bdd33e4faa0c9decdcd0/merged 20 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/534837088c8ba6021c80d2ebb4ddb87a075106a784c499cc9ffca541d2e86dce/merged 21 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/8cdf1249e1823f24caf1d14ca9c05caa4e8e569d81292d06c642e618bc90b82b/merged 22 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/3ee0d48c04a32a2f0938af17898697042423d7d147df3ee8c1178bfda6e5fd9a/merged 23 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/31f5db66511cc7fd15cf1e2a23b7178d0eafbf1d7ddbdef1e0f82a254b6f6fc5/merged 24 overlay 18G 14G 3.9G 79% /var/lib/docker/overlay2/16288e087e06b2c1fc369d19528eeb933cde44b8aa785f35b6756ec4da3a0f60/merged 25 tmpfs 229M 0 229M 0% /run/user/0 26 192.168.110.133:/biehl 18G 11G 7.1G 61% /mnt 27 [root@k8s-master ~]# df -h /mnt/ 28 Filesystem Size Used Avail Use% Mounted on 29 192.168.110.133:/biehl 18G 11G 7.1G 61% /mnt 30 [root@k8s-master ~]#
分布式复制卷扩容，可以先使用df -h命令进行查看扩容前的容量，然后扩容后查看容量，在运行状态也可以进行扩容的哦，如下所示：
1 [root@k8s-master ~]# df -h /mnt/ 2 Filesystem Size Used Avail Use% Mounted on 3 192.168.110.133:/biehl 18G 11G 7.1G 61% /mnt 4 [root@k8s-master ~]# gluster volume add-brick biehl k8s-node3:/gfs/test1 k8s-node3:/gfs/test2 force 5 volume add-brick: success 6 [root@k8s-master ~]# df -h /mnt/ 7 Filesystem Size Used Avail Use% Mounted on 8 192.168.110.133:/biehl 23G 15G 8.8G 62% /mnt 9 [root@k8s-master ~]#
接下来，可以向/mnt目录里面上传一些文件进行观察，如下所示：
1 [root@k8s-master ~]# cd /mnt/ 2 [root@k8s-master mnt]# rz -E 3 rz waiting to receive. 4 [root@k8s-master mnt]# ls 5 book-master6.0.war 6 [root@k8s-master mnt]# unzip book-master6.0.war
解压之后显示一堆文件，这些文件会分别在放到不同的存储单元中，首先查看master主节点的目录单元，如下所示：
1 [root@k8s-master mnt]# yum install tree 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.bfsu.edu.cn 8 * centos-gluster7: mirrors.tuna.tsinghua.edu.cn 9 * extras: mirrors.bfsu.edu.cn 10 * updates: mirrors.bfsu.edu.cn 11 Resolving Dependencies 12 --> Running transaction check 13 ---> Package tree.x86_64 0:1.6.0-10.el7 will be installed 14 --> Finished Dependency Resolution 15 16 Dependencies Resolved 17 18 ================================================================================================================================================================================================================= 19 Package Arch Version Repository Size 20 ================================================================================================================================================================================================================= 21 Installing: 22 tree x86_64 1.6.0-10.el7 base 46 k 23 24 Transaction Summary 25 ================================================================================================================================================================================================================= 26 Install 1 Package 27 28 Total download size: 46 k 29 Installed size: 87 k 30 Is this ok [y/d/N]: y 31 Downloading packages: 32 tree-1.6.0-10.el7.x86_64.rpm | 46 kB 00:00:05 33 Running transaction check 34 Running transaction test 35 Transaction test succeeded 36 Running transaction 37 Installing : tree-1.6.0-10.el7.x86_64 1/1 38 Verifying : tree-1.6.0-10.el7.x86_64 1/1 39 40 Installed: 41 tree.x86_64 0:1.6.0-10.el7 42 43 Complete! 44 [root@k8s-master mnt]# tree /gfs/
由于我的文件太多，就不进行展示了，使用命令tree /gfs就可以进行查看了，可以在三台机器都进行查看观察的。
4、glusterfs作为k8s的后端存储，可以使用命令查看k8s支持的后端存储类型，如下所示：
1 [root@k8s-master test1]# kubectl explain pv.spec 2 RESOURCE: spec <Object> 3 4 DESCRIPTION: 5 Spec defines a specification of a persistent volume owned by the cluster. 6 Provisioned by an administrator. More info: 7 http://kubernetes.io/docs/user-guide/persistent-volumes#persistent-volumes 8 9 PersistentVolumeSpec is the specification of a persistent volume. 10 11 FIELDS: 12 awsElasticBlockStore <Object> 13 AWSElasticBlockStore represents an AWS Disk resource that is attached to a 14 kubelet's host machine and then exposed to the pod. More info: 15 http://kubernetes.io/docs/user-guide/volumes#awselasticblockstore 16 17 cephfs <Object> 18 CephFS represents a Ceph FS mount on the host that shares a pod's lifetime 19 20 fc <Object> 21 FC represents a Fibre Channel resource that is attached to a kubelet's host 22 machine and then exposed to the pod. 23 24 rbd <Object> 25 RBD represents a Rados Block Device mount on the host that shares a pod's 26 lifetime. More info: 27 http://releases.k8s.io/HEAD/examples/volumes/rbd/README.md 28 29 accessModes <[]Object> 30 AccessModes contains all ways the volume can be mounted. More info: 31 http://kubernetes.io/docs/user-guide/persistent-volumes#access-modes 32 33 azureFile <Object> 34 AzureFile represents an Azure File Service mount on the host and bind mount 35 to the pod. 36 37 capacity <object> 38 A description of the persistent volume's resources and capacity. More info: 39 http://kubernetes.io/docs/user-guide/persistent-volumes#capacity 40 41 cinder <Object> 42 Cinder represents a cinder volume attached and mounted on kubelets host 43 machine More info: 44 http://releases.k8s.io/HEAD/examples/mysql-cinder-pd/README.md 45 46 flocker <Object> 47 Flocker represents a Flocker volume attached to a kubelet's host machine 48 and exposed to the pod for its usage. This depends on the Flocker control 49 service being running 50 51 nfs <Object> 52 NFS represents an NFS mount on the host. Provisioned by an admin. More 53 info: http://kubernetes.io/docs/user-guide/volumes#nfs 54 55 azureDisk <Object> 56 AzureDisk represents an Azure Data Disk mount on the host and bind mount to 57 the pod. 58 59 claimRef <Object> 60 ClaimRef is part of a bi-directional binding between PersistentVolume and 61 PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName 62 is the authoritative bind between PV and PVC. More info: 63 http://kubernetes.io/docs/user-guide/persistent-volumes#binding 64 65 gcePersistentDisk <Object> 66 GCEPersistentDisk represents a GCE Disk resource that is attached to a 67 kubelet's host machine and then exposed to the pod. Provisioned by an admin. 68 More info: http://kubernetes.io/docs/user-guide/volumes#gcepersistentdisk 69 70 persistentVolumeReclaimPolicy <string> 71 What happens to a persistent volume when released from its claim. Valid 72 options are Retain (default) and Recycle. Recycling must be supported by the 73 volume plugin underlying this persistent volume. More info: 74 http://kubernetes.io/docs/user-guide/persistent-volumes#recycling-policy 75 76 photonPersistentDisk <Object> 77 PhotonPersistentDisk represents a PhotonController persistent disk attached 78 and mounted on kubelets host machine 79 80 vsphereVolume <Object> 81 VsphereVolume represents a vSphere volume attached and mounted on kubelets 82 host machine 83 84 flexVolume <Object> 85 FlexVolume represents a generic volume resource that is 86 provisioned/attached using an exec based plugin. This is an alpha feature 87 and may change in future. 88 89 glusterfs <Object> 90 Glusterfs represents a Glusterfs volume that is attached to a host and 91 exposed to the pod. Provisioned by an admin. More info: 92 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md 93 94 hostPath <Object> 95 HostPath represents a directory on the host. Provisioned by a developer or 96 tester. This is useful for single-node development and testing only! On-host 97 storage is not supported in any way and WILL NOT WORK in a multi-node 98 cluster. More info: http://kubernetes.io/docs/user-guide/volumes#hostpath 99 100 iscsi <Object> 101 ISCSI represents an ISCSI Disk resource that is attached to a kubelet's 102 host machine and then exposed to the pod. Provisioned by an admin. 103 104 quobyte <Object> 105 Quobyte represents a Quobyte mount on the host that shares a pod's lifetime 106 107 108 [root@k8s-master test1]#
可以查看详细的glusterfs配置需要什么，endpoints就是svc里面的一个资源，endpoints就是（clusterIP + 端口号），path就是glusterfs卷的一个路径，readOnly只读模式，如下所示：
1 [root@k8s-master test1]# kubectl explain pv.spec.glusterfs 2 RESOURCE: glusterfs <Object> 3 4 DESCRIPTION: 5 Glusterfs represents a Glusterfs volume that is attached to a host and 6 exposed to the pod. Provisioned by an admin. More info: 7 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md 8 9 Represents a Glusterfs mount that lasts the lifetime of a pod. Glusterfs volumes do not support ownership management or SELinux relabeling. 10 11 FIELDS: 12 endpoints <string> -required- 13 EndpointsName is the endpoint name that details Glusterfs topology. More 14 info: 15 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md#create-a-pod 16 17 path <string> -required- 18 Path is the Glusterfs volume path. More info: 19 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md#create-a-pod 20 21 readOnly <boolean> 22 ReadOnly here will force the Glusterfs volume to be mounted with read-only 23 permissions. Defaults to false. More info: 24 http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md#create-a-pod 25 26 27 [root@k8s-master test1]#
开始创建endpoints的glusterfs-ep.yaml配置文件，如下所示：
1 [root@k8s-master k8s]# ls 2 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip volume 3 [root@k8s-master k8s]# mkdir glusterfs 4 [root@k8s-master k8s]# cd glusterfs/ 5 [root@k8s-master glusterfs]# ls 6 [root@k8s-master glusterfs]# vim glusterfs-ep.yaml 7 [root@k8s-master glusterfs]#
具体内容，如下所示：
1 apiVersion: v1 2 kind: Endpoints 3 metadata: 4 name: glusterfs 5 namespace: default 6 subsets: 7 # k8s连接glusterfs的ip地址和端口号 8 - addresses: 9 - ip: 192.168.110.133 10 - ip: 192.168.110.134 11 - ip: 192.168.110.135 12 ports: 13 - port: 49152 14 protocol: TCP 15
每个节点启动了glusterfs就存在了49152端口号，如下所示：
1 [root@k8s-node3 test1]# netstat -lntup 49152 2 Active Internet connections (only servers) 3 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 4 tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 47623/glusterd 5 tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 14515/kubelet 6 tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 6779/kube-proxy 7 tcp 0 0 0.0.0.0:40842 0.0.0.0:* LISTEN - 8 tcp 0 0 192.168.110.135:10250 0.0.0.0:* LISTEN 14515/kubelet 9 tcp 0 0 192.168.110.135:10255 0.0.0.0:* LISTEN 14515/kubelet 10 tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd 11 tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 90097/rpc.mountd 12 tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 7122/X 13 tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 7304/dnsmasq 14 tcp 0 0 0.0.0.0:55318 0.0.0.0:* LISTEN 52362/rpc.statd 15 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6776/sshd 16 tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 6778/cupsd 17 tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7094/master 18 tcp 0 0 0.0.0.0:49152 0.0.0.0:* LISTEN 71495/glusterfsd
开始创建这个endpoint，如下所示：
1 [root@k8s-master glusterfs]# kubectl create -f glusterfs-ep.yaml 2 endpoints "glusterfs" created 3 [root@k8s-master glusterfs]#
创建成功可以查看一下，如下所示：
1 [root@k8s-master glusterfs]# kubectl get ep 2 NAME ENDPOINTS AGE 3 glusterfs 192.168.110.133:49152,192.168.110.134:49152,192.168.110.135:49152 34s 4 kubernetes 192.168.110.133:6443 25d 5 mysql 172.16.66.6:3306 5h 6 myweb 172.16.66.5:8080 6h 7 [root@k8s-master glusterfs]# kubectl get endpoints 8 NAME ENDPOINTS AGE 9 glusterfs 192.168.110.133:49152,192.168.110.134:49152,192.168.110.135:49152 1m 10 kubernetes 192.168.110.133:6443 25d 11 mysql 172.16.66.6:3306 5h 12 myweb 172.16.66.5:8080 6h 13 [root@k8s-master glusterfs]#
接下来，给这个endpoint创建一个Servic，需要注意的是Service和endpoint是靠名称来进行关联的，他们两个的名称必须一致的，如下所示：
1 [root@k8s-master glusterfs]# vim glusterfs-svc.yaml 2 [root@k8s-master glusterfs]#
配置内容，如下所示：
1 apiVersion: v1 2 kind: Service 3 metadata: 4 name: glusterfs 5 namespace: default 6 spec: 7 ports: 8 - port: 49152 9 protocol: TCP 10 targetPort: 49152 11 sessionAffinity: None 12 type: ClusterIP 13
开始进行创建Service，并进行查看，如下所示：
1 [root@k8s-master glusterfs]# vim glusterfs-svc.yaml 2 [root@k8s-master glusterfs]# kubectl create -f glusterfs-svc.yaml 3 service "glusterfs" created 4 [root@k8s-master glusterfs]# kubectl get svc 5 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE 6 glusterfs 10.254.182.41 <none> 49152/TCP 10s 7 kubernetes 10.254.0.1 <none> 443/TCP 25d 8 mysql 10.254.126.11 <none> 3306/TCP 5h 9 myweb 10.254.188.155 <nodes> 8080:30008/TCP 6h 10 [root@k8s-master glusterfs]#
可以使用查看详情，看是否已经关联上了，如下所示：
1 [root@k8s-master glusterfs]# kubectl describe svc glusterfs 2 Name: glusterfs 3 Namespace: default 4 Labels: <none> 5 Selector: <none> 6 Type: ClusterIP 7 IP: 10.254.182.41 8 Port: <unset> 49152/TCP 9 Endpoints: 192.168.110.133:49152,192.168.110.134:49152,192.168.110.135:49152 10 Session Affinity: None 11 No events. 12 [root@k8s-master glusterfs]#
接下来，开始创建glusterfs类型的PV，k8s对接各种后端存储的时候最关键的地方就是创建PV，如下所示：
1 [root@k8s-master glusterfs]# vim glusterfs-pv.yaml 2 [root@k8s-master glusterfs]#
配置内容，如下所示：
1 apiVersion: v1 2 kind: PersistentVolume 3 metadata: 4 name: gluster 5 labels: 6 type: glusterfs 7 spec: 8 capacity: 9 storage: 50Gi 10 accessModes: 11 - ReadWriteMany 12 glusterfs: 13 endpoints: "glusterfs" 14 path: "biehl" 15 readOnly: false
创建并查看PV，如下所示：
1 [root@k8s-master glusterfs]# gluster volume list 2 biehl 3 [root@k8s-master glusterfs]# kubectl create -f glusterfs-pv.yaml 4 persistentvolume "gluster" created 5 [root@k8s-master glusterfs]# kubectl get pv 6 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 7 gluster 50Gi RWX Retain Available 9s 8 mysql 5Gi RWX Recycle Bound default/mysql 5h 9 [root@k8s-master glusterfs]#
开始创建PVC，才可以使用PV，开始创建PVC，如下所示：
1 [root@k8s-master glusterfs]# cp ../tomcat_demo/mysql-pvc.yaml . 2 [root@k8s-master glusterfs]# ls 3 glusterfs-ep.yaml glusterfs-pv.yaml glusterfs-svc.yaml mysql-pvc.yaml 4 [root@k8s-master glusterfs]# mv mysql-pvc.yaml glusterfs-pvc.yaml 5 [root@k8s-master glusterfs]# ls 6 glusterfs-ep.yaml glusterfs-pvc.yaml glusterfs-pv.yaml glusterfs-svc.yaml 7 [root@k8s-master glusterfs]#
配置内容，如下所示：
1 kind: PersistentVolumeClaim 2 apiVersion: v1 3 metadata: 4 name: gluster 5 spec: 6 accessModes: 7 - ReadWriteMany 8 resources: 9 requests: 10 storage: 5Gi
创建并查看PVC、PV的内容，如下所示：
1 [root@k8s-master glusterfs]# kubectl create -f glusterfs-pvc.yaml 2 persistentvolumeclaim "gluster" created 3 [root@k8s-master glusterfs]# kubectl get pvc 4 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE 5 gluster Bound gluster 50Gi RWX 4s 6 mysql Bound mysql 5Gi RWX 6h 7 [root@k8s-master glusterfs]# kubectl get pv 8 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 9 gluster 50Gi RWX Retain Bound default/gluster 5m 10 mysql 5Gi RWX Recycle Bound default/mysql 6h 11 [root@k8s-master glusterfs]#
PVC绑定PV之后就可以使用了，可以对pod进行使用，如下所示：
1 [root@k8s-master glusterfs]# ls 2 glusterfs-ep.yaml glusterfs-pvc.yaml glusterfs-pv.yaml glusterfs-svc.yaml 3 [root@k8s-master glusterfs]# cp ../pod/nginx_pod.yaml . 4 [root@k8s-master glusterfs]# ls 5 glusterfs-ep.yaml glusterfs-pvc.yaml glusterfs-pv.yaml glusterfs-svc.yaml nginx_pod.yaml 6 [root@k8s-master glusterfs]# vim nginx_pod.yaml 7 [root@k8s-master glusterfs]#
将volumeMounts、volumes都配置好，如下所示：
1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: nginx 5 labels: 6 app: web 7 env: nginx 8 spec: 9 containers: 10 - name: nginx 11 image: 192.168.110.133:5000/nginx:1.13 12 ports: 13 - containerPort: 80 14 volumeMounts: 15 - name: nfs-vol2 16 mountPath: /usr/share/nginx/html 17 volumes: 18 - name: nfs-vol2 19 persistentVolumeClaim: 20 claimName: gluster
创建nginx这个pod，并进行查看，发现已经正常运行起来了，如下所示：
1 [root@k8s-master glusterfs]# kubectl create -f nginx_pod.yaml 2 pod "nginx" created 3 [root@k8s-master glusterfs]# kubectl get pods -o wide 4 NAME READY STATUS RESTARTS AGE IP NODE 5 mysql-wldks 1/1 Running 0 5h 172.16.66.6 k8s-node3 6 myweb-c8sf6 1/1 Running 1 7h 172.16.66.5 k8s-node3 7 nginx 1/1 Running 0 3s 172.16.66.3 k8s-node3 8 [root@k8s-master glusterfs]#
然后可以访问一下这个Pod，如下所示：
1 [root@k8s-master glusterfs]# kubectl get pods -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE 3 mysql-wldks 1/1 Running 0 5h 172.16.66.6 k8s-node3 4 myweb-c8sf6 1/1 Running 1 7h 172.16.66.5 k8s-node3 5 nginx 1/1 Running 0 3s 172.16.66.3 k8s-node3 6 [root@k8s-master glusterfs]# curl 172.16.66.3
我们的容器除了SVC里面通过反向代理和端口映射的方式访问之外，还可以让Pod不经过我们的SVC进行访问，可以在Pod里面，新增一个hostPort进行访问。
将之前的nginx的pod删除掉，再重新创建一个，如下所示：
1 [root@k8s-master glusterfs]# kubectl get pods -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE 3 mysql-wldks 1/1 Running 0 7h 172.16.66.6 k8s-node3 4 myweb-c8sf6 1/1 Running 1 9h 172.16.66.5 k8s-node3 5 nginx 1/1 Running 0 2h 172.16.66.3 k8s-node3 6 [root@k8s-master glusterfs]# kubectl delete pod nginx 7 pod "nginx" deleted 8 [root@k8s-master glusterfs]# kubectl get pods -o wide 9 NAME READY STATUS RESTARTS AGE IP NODE 10 mysql-wldks 1/1 Running 0 7h 172.16.66.6 k8s-node3 11 myweb-c8sf6 1/1 Running 1 9h 172.16.66.5 k8s-node3 12 [root@k8s-master glusterfs]# kubectl create -f nginx_pod.yaml 13 pod "nginx" created 14 [root@k8s-master glusterfs]# kubectl get pods -o wide 15 NAME READY STATUS RESTARTS AGE IP NODE 16 mysql-wldks 1/1 Running 0 7h 172.16.66.6 k8s-node3 17 myweb-c8sf6 1/1 Running 1 9h 172.16.66.5 k8s-node3 18 nginx 1/1 Running 0 2s 172.16.66.3 k8s-node3 19 [root@k8s-master glusterfs]#
可以看到新创建的nginx在k8s-node3这个节点上了，但是此时我发现无论curl访问还是浏览器访问都是403错误，这不都是很郁闷了。
1 [root@k8s-master glusterfs]# curl -I 172.16.66.3 2 HTTP/1.1 403 Forbidden 3 Server: nginx/1.15.12 4 Date: Tue, 30 Jun 2020 14:03:56 GMT 5 Content-Type: text/html 6 Content-Length: 154 7 Connection: keep-alive
后来经过仔细地分析和查找，发现是确实访问不到页面才出现地错误，究其原因，主要就是/mnt目录下面要有可以直接访问地界面才可以地。
1 [root@k8s-master glusterfs]# cd /mnt/ 2 [root@k8s-master mnt]# ls 3 css img index.html js
删除之前地nginx地pod，创建新的pod，如下所示：
1 [root@k8s-master glusterfs]# kubectl get pod -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE 3 mysql-wldks 1/1 Running 0 8h 172.16.66.6 k8s-node3 4 myweb-c8sf6 1/1 Running 1 10h 172.16.66.5 k8s-node3 5 nginx 1/1 Running 0 6m 172.16.66.3 k8s-node3 6 nginx2 1/1 Running 0 17m 172.16.74.5 k8s-node2 7 [root@k8s-master glusterfs]# kubectl delete pod nginx 8 pod "nginx" deleted 9 [root@k8s-master glusterfs]# kubectl get pod -o wide 10 NAME READY STATUS RESTARTS AGE IP NODE 11 mysql-wldks 1/1 Running 0 8h 172.16.66.6 k8s-node3 12 myweb-c8sf6 1/1 Running 1 10h 172.16.66.5 k8s-node3 13 nginx2 1/1 Running 0 17m 172.16.74.5 k8s-node2 14 [root@k8s-master glusterfs]# kubectl create -f nginx_pod.yaml 15 pod "nginx" created 16 [root@k8s-master glusterfs]# kubectl get pod -o wide 17 NAME READY STATUS RESTARTS AGE IP NODE 18 mysql-wldks 1/1 Running 0 8h 172.16.66.6 k8s-node3 19 myweb-c8sf6 1/1 Running 1 10h 172.16.66.5 k8s-node3 20 nginx 1/1 Running 0 2s 172.16.66.3 k8s-node3 21 nginx2 1/1 Running 0 18m 172.16.74.5 k8s-node2 22 [root@k8s-master glusterfs]# curl 172.16.66.3 23 <!DOCTYPE html> 24 <html lang="en"> 25 <head> 26 <meta charset="UTF-8"> 27 <meta name="viewport" content="width=device-width, initial-scale=1.0"> 28 <meta http-equiv="X-UA-Compatible" content="ie=edge"> 29 <title>HTML5ȫı·ɻСԎϷ - Դë֮¼м/title> 30 31 <style> 32 html { 33 width: 100%; 34 height: 100%; 35 margin: 0; 36 padding: 0; 37 position: relative; 38 background-image: linear-gradient(#2C3E50,#4CA1AF); 39 40 } 41 42 .canvasbig { 43 position: absolute; 44 left: calc(50% - 260px); 45 top: calc(50% - 400px); 46 width: 520px; 47 height: 800px; 48 49 } 50 51 .canvasdiv { 52 position: absolute; 53 cursor: pointer; 54 left: 160px; 55 top: 500px; 56 width: 200px; 57 height: 53px; 58 background-image: url(img/starting.png); 59 } 60 61 .none { 62 display: none; 63 } 64 </style> 65 66 </head> 67 <body> 68 69 <!-- <script src="js/circle.js"></script> --> 70 71 <div class="canvasbig"> 72 <div class="canvasdiv"></div> 73 <canvas width="520" height="800" id="canvas1"></canvas> 74 </div> 75 76 <script src="js/index.js"></script> 77 <div style="text-align:center;"> 78 <p>¸Ϗ·£º<a href="http://www.mycodes.net/" target="_blank">Դë֮¼м/a></p> 79 </div> 80 </body> 81 </html>[root@k8s-master glusterfs]#
访问http://192.168.110.135:8889/，在分析地过程中，我将nginx_pod.yaml地hostPort: 8889改成了8889端口号哦。
游戏是可以玩的，功能也算是搞完了。
