存储的分类：
块存储：
硬盘
LVM
DAS(Direct Attach Storage)
是直接连接于主机服务器的一种存储方式，每一台主机服务器有独立的存储设备，每台主机服务器的存储设备无法互通。
通常用单一网络环境下且数据交换量不大，性能要求不高的环境下，可以说是一种应用较早的技术实现
SAN
1、FC-SAN
2、IP-SAN
分布式存储-Ceph
Ceph是开源实现的PB级别的分布式文件系统，其分布式对象存储机制为上层提供了文件接口、块存储接口和对象存储接口
DAS(Direct Attach Storage)
文件存储：
FS
对象存储：
SAN(storage Area Network)
NAS
Cinder组件：
chnder-api。接受API请求并将请求路由到cinder-volume来执行。
cinder-volume。相应请求，读取或写向块存储数据库为维护状态，通过信息队列机制与其他进程交互（如cinder-scheduler），或直接与上层块存储提供的硬件或软件进行交互。通过驱动结构，他可以与众多的存储提供者进行交互。
cinder-scheduler。守护进程。类型于nova-scheduler，为存储卷的实例选取最优的块存储供应节点。
没有Cinder时候虚拟机磁盘存放位置；
[root@hostname linux-node2 ~]# ls /var/lib/nova/instances/ _base compute_nodes e7585ece-c8e9-4ad8-8796-8a58bca459bf locks snapshots [root@hostname linux-node2 ~]# ls /var/lib/nova/instances/e7585ece-c8e9-4ad8-8796-8a58bca459bf/ console.log disk disk.info libvirt.xml
安装Cinder控制节点：
[root@hostnamelinux-node1 ~]# yum -y install openstack-cinder python-cinderclient
查看是否已创建cinder数据库，我这里在keystone安装时候已经创建；
[root@hostnamelinux-node1 ~]# mysql -uroot -psmoke520 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 264 Server version: 10.1.20-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | cinder | | glance | | information_schema | | keystone | | mysql | | neutron | | nova | | performance_schema | +--------------------+ 8 rows in set (1.25 sec) MariaDB [(none)]> exit; Bye
配置Cinder：
[root@hostnamelinux-node1 ~]# vim /etc/cinder/cinder.conf [database] connection = mysql://cinder:cinder@192.168.56.11/cinder
同步数据库：
[root@hostnamelinux-node1 ~]# su -s /bin/sh -c "cinder-manage db sync" cinder [root@hostnamelinux-node1 ~]# mysql -ucinder -pcinder -h192.168.56.11 -e "use cinder;show tables;" +----------------------------+ | Tables_in_cinder | +----------------------------+ | backups | | cgsnapshots | | consistencygroups | | driver_initiator_data | | encryption | | image_volume_cache_entries | | iscsi_targets | | migrate_version | | quality_of_service_specs | | quota_classes | | quota_usages | | quotas | | reservations | | services | | snapshot_metadata | | snapshots | | transfers | | volume_admin_metadata | | volume_attachment | | volume_glance_metadata | | volume_metadata | | volume_type_extra_specs | | volume_type_projects | | volume_types | | volumes | +----------------------------+
创建用户：
[root@hostnamelinux-node1 ~]# source admin-openrc.sh
用户名cinder，密码cinder；
[root@hostnamelinux-node1 ~]# openstack user create --domain default --password-prompt cinder User Password: Repeat User Password: +-----------+----------------------------------+ | Field | Value | +-----------+----------------------------------+ | domain_id | default | | enabled | True | | id | 5ffa5acdd3d8438e994a6fcd9319708a | | name | cinder | +-----------+----------------------------------+
将cinder用户加入service项目并赋予admin角色；
[root@hostnamelinux-node1 ~]# openstack role add --project service --user cinder admin
更改chinder配置文件：
[root@hostnamelinux-node1 ~]# vim /etc/cinder/cinder.conf [DEFAULT] auth_strategy = keystone rpc_backend = rabbit glance_host = 192.168.56.11 [keystone_authtoken] auth_uri = http://192.168.56.11:5000 auth_url = http://192.168.56.11:35357 auth_plugin = password project_domain_id = default user_domain_id = default project_name = service username = cinder password = cinder [oslo_messaging_rabbit] rabbit_host = 192.168.56.11 rabbit_port = 5672 rabbit_userid = openstack rabbit_password = openstack [oslo_concurrency] lock_path = /var/lib/cinder/tmp [root@hostnamelinux-node1 ~]# grep '^[a-z]' /etc/cinder/cinder.conf glance_host = 192.168.56.11 auth_strategy = keystone rpc_backend = rabbit connection = mysql://cinder:cinder@192.168.56.11/cinder auth_uri = http://192.168.56.11:5000 auth_url = http://192.168.56.11:35357 auth_plugin = password project_domain_id = default user_domain_id = default project_name = service username = cinder password = cinder lock_path = /var/lib/cinder/tmp rabbit_host = 192.168.56.11 rabbit_port = 5672 rabbit_userid = openstack rabbit_password = openstack [root@hostnamelinux-node1 ~]# vim /etc/nova/nova.conf [cinder] os_region_name = RegionOne
重启服务：
[root@hostnamelinux-node1 ~]# systemctl restart openstack-nova-api.service [root@hostnamelinux-node1 ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-api.service to /usr/lib/systemd/system/openstack-cinde r-api.service. Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-scheduler.service to /usr/lib/systemd/system/openstack -cinder-scheduler.service. [root@hostnamelinux-node1 ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service
注册Cinder：
创建Cinder v1服务：
[root@hostnamelinux-node1 ~]# openstack service create --name cinder \ --description "OpenStack Block Storage" volume +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Block Storage | | enabled | True | | id | 44c68286553947049187cd9886a85807 | | name | cinder | | type | volume | +-------------+----------------------------------+
创建Cinder v2服务：
[root@hostnamelinux-node1 ~]# openstack service create --name cinderv2 \ --description "OpenStack Block Storage" volumev2 +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Block Storage | | enabled | True | | id | 9bc7d84ec806418d9046635275518741 | | name | cinderv2 | | type | volumev2 | +-------------+----------------------------------+
创建v1 endpoint：
公网注册：
[root@hostnamelinux-node1 ~]# openstack endpoint create --region RegionOne \ > volume public http://192.168.56.11:8776/v1/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 22d5172503014abc99e1448e8990fe01 | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 44c68286553947049187cd9886a85807 | | service_name | cinder | | service_type | volume | | url | http://192.168.56.11:8776/v1/%(tenant_id)s | +--------------+--------------------------------------------+
私网注册：
[root@hostnamelinux-node1 ~]# openstack endpoint create --region RegionOne volume internal http://192.168.56.11:8776/v1/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 184ede7d6cb64e52a14e884cd90b8455 | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 44c68286553947049187cd9886a85807 | | service_name | cinder | | service_type | volume | | url | http://192.168.56.11:8776/v1/%(tenant_id)s | +--------------+--------------------------------------------+
admin注册：
[root@hostnamelinux-node1 ~]# openstack endpoint create --region RegionOne volume admin http://192.168.56.11:8776/v1/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 38796e145c55401e8140815af281d0fd | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 44c68286553947049187cd9886a85807 | | service_name | cinder | | service_type | volume | | url | http://192.168.56.11:8776/v1/%(tenant_id)s | +--------------+--------------------------------------------+
创建v2 endpoint：
公网注册：
[root@hostnamelinux-node1 ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.56.11:8776/v2/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | c7246b6f7c8d4227b2cb0b03c7ffe694 | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 9bc7d84ec806418d9046635275518741 | | service_name | cinderv2 | | service_type | volumev2 | | url | http://192.168.56.11:8776/v2/%(tenant_id)s | +--------------+--------------------------------------------+
私网注册：
[root@hostnamelinux-node1 ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.56.11:8776/v2/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | e738d99d2a7a46e5a122ed6b006de63e | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 9bc7d84ec806418d9046635275518741 | | service_name | cinderv2 | | service_type | volumev2 | | url | http://192.168.56.11:8776/v2/%(tenant_id)s | +--------------+--------------------------------------------+
admin注册：
[root@hostnamelinux-node1 ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.56.11:8776/v2/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | c7c6845176214a64ab9abefb7cc7acf8 | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 9bc7d84ec806418d9046635275518741 | | service_name | cinderv2 | | service_type | volumev2 | | url | http://192.168.56.11:8776/v2/%(tenant_id)s | +--------------+--------------------------------------------+
安装Cinder存储节点到计算节点：
需要先关机给计算节点添加硬盘给Cinder使用；
[root@hostname linux-node2 ~]# fdisk -l 磁盘 /dev/sda：53.7 GB, 53687091200 字节，104857600 个扇区 Units = 扇区 of 1 * 512 = 512 bytes 扇区大小(逻辑/物理)：512 字节 / 512 字节 I/O 大小(最小/最佳)：512 字节 / 512 字节 磁盘标签类型：dos 磁盘标识符：0x000c55d1 设备 Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 104857599 51379200 8e Linux LVM 磁盘 /dev/sdb：53.7 GB, 53687091200 字节，104857600 个扇区 Units = 扇区 of 1 * 512 = 512 bytes 扇区大小(逻辑/物理)：512 字节 / 512 字节 I/O 大小(最小/最佳)：512 字节 / 512 字节 磁盘 /dev/mapper/cl-root：50.5 GB, 50457477120 字节，98549760 个扇区 Units = 扇区 of 1 * 512 = 512 bytes 扇区大小(逻辑/物理)：512 字节 / 512 字节 I/O 大小(最小/最佳)：512 字节 / 512 字节 磁盘 /dev/mapper/cl-swap：2147 MB, 2147483648 字节，4194304 个扇区 Units = 扇区 of 1 * 512 = 512 bytes 扇区大小(逻辑/物理)：512 字节 / 512 字节 I/O 大小(最小/最佳)：512 字节 / 512 字节
创建LVM：
[root@hostname linux-node2 ~]# pvcreate /dev/sdb Physical volume "/dev/sdb" successfully created. [root@hostname linux-node2 ~]# vgcreate cinder-volumes /dev/sdb Volume group "cinder-volumes" successfully created
设置LVM配置文件：
[root@hostname linux-node2 ~]# vim /etc/lvm/lvm.conf filter = [ "a/sdb/", "r/.*/" ] [root@hostname linux-node2 ~]# yum -y install openstack-cinder targetcli python-oslo-policy
配置Cinder：
[root@hostname linux-node2 ~]# scp root@192.168.56.11:/etc/cinder/cinder.conf /etc/cinder/ root@192.168.56.11's password: cinder.conf 100% 117KB 116.7KB/s 00:00 [root@hostname linux-node2 ~]# vim /etc/cinder/cinder.conf [lvm] volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_group = cinder-volumes iscsi_protocol = iscsi iscsi_helper = lioadm [DEFAULT] enabled_backends = lvm [root@hostname linux-node2 ~]# grep '^[a-z]' /etc/cinder/cinder.conf glance_host = 192.168.56.11 auth_strategy = keystone enabled_backends = lvm rpc_backend = rabbit connection = mysql://cinder:cinder@192.168.56.11/cinder auth_uri = http://192.168.56.11:5000 auth_url = http://192.168.56.11:35357 auth_plugin = password project_domain_id = default user_domain_id = default project_name = service username = cinder password = cinder lock_path = /var/lib/cinder/tmp rabbit_host = 192.168.56.11 rabbit_port = 5672 rabbit_userid = openstack rabbit_password = openstack volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_group = cinder-volumes iscsi_protocol = iscsi iscsi_helper = lioadm
启动Cinder：
[root@hostname linux-node2 ~]# systemctl enable openstack-cinder-volume.service target.service Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack -cinder-volume.service. Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. [root@hostname linux-node2 ~]# systemctl start openstack-cinder-volume.service target.service
如果启动openstack-cinder-volume.service报错为ImportError: No module named Crypto.Random，主要是因为缺少pycrypto模块，通过pip install pycryton安装；
[root@hostnamelinux-node1 ~]# source admin-openrc.sh [root@hostnamelinux-node1 ~]# cinder service-list +------------------+------------------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+------------------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | hostnamelinux-node1.smoke.com | nova | enabled | up | 2018-10-13T10:24:53.000000 | - | | cinder-volume | hostname linux-node2.smoke.com@lvm | nova | enabled | up | 2018-10-13T10:24:29.000000 | - | +------------------+------------------------------------+------+---------+-------+----------------------------+-----------------+
创建云硬盘：
使用demo密码demo用户登录Dashboard；
创建成功；
创建一个云硬盘会创建一个LVS；
[root@hostname linux-node2 ~]# lvdisplay --- Logical volume --- LV Path /dev/cinder-volumes/volume-4d5fbaae-8eaa-43ea-aff7-933b5a1f7564 LV Name volume-4d5fbaae-8eaa-43ea-aff7-933b5a1f7564 VG Name cinder-volumes LV UUID 1l2TMz-ZjRM-R3Kw-as7G-Ga7V-ufFG-ePfq17 LV Write Access read/write LV Creation host, time hostname linux-node2.smoke.com, 2018-10-13 18:35:48 +0800 LV Status available # open 0 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:2
挂载云硬盘：
选挂载的虚拟机实例；
挂载成功；
连接虚拟机实例进行验证；
$ sudo fdisk -l
Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/vda1 * 16065 2088449 1036192+ 83 Linux
Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/vdb doesn't contain a valid partition table
$ sudo fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x12712ffb.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
$ sudo mkfs.ext4 /dev/vdb1
mke2fs 1.42.2 (27-Mar-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 261888 blocks
13094 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mkdir /data
$ sudo mount /dev/vdb1 /data/
$ sudo mount
rootfs on / type rootfs (rw)
/dev on /dev type devtmpfs (rw,relatime,size=248148k,nr_inodes=62037,mode=755)
/dev/vda1 on / type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered)
/proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=200k,mode=755)
/dev/vdb1 on /data type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
$ cd /data/
$ ls
lost+found
$ sudo mkdir hehe
$ sudo touch test
$ ls
hehe lost+found test
卸载云硬盘，不能直接删除硬盘，虚拟机会坏掉；
$ cd $ sudo umount /data $ sudo mount rootfs on / type rootfs (rw) /dev on /dev type devtmpfs (rw,relatime,size=248148k,nr_inodes=62037,mode=755) /dev/vda1 on / type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered) /proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777) tmpfs on /run type tmpfs (rw,nosuid,relatime,size=200k,mode=755)
创建实例允许修改密码（控制节点）：
[root@hostnamelinux-node1 ~]# vim /etc/openstack-dashboard/local_settings
OPENSTACK_HYPERVISOR_FEATURES = {
'can_set_mount_point': True,
'can_set_password': True,
'requires_keypair': True,
}
[root@hostnamelinux-node1 ~]# systemctl restart httpd
允许注入密码（计算节点）：
[root@hostname linux-node2 ~]# vim /etc/nova/nova.conf [libvirt] inject_password=true inject_key=true [root@hostname linux-node2 ~]# systemctl restart openstack-nova-compute
创建虚拟机：
访问和安全
创建后脚本：
创建成功；
使用设置的密码smoke520进行登录；
浙公网安备 33010602011771号