RHEL 6.5----rsync+inotify数据同步服务
Rsync特性：
可以镜像保存整个目录树和文件系统；
可以保持原文件的权限、时间、软硬链接等；
安装简单。
传输特点：
速度快：rsync首次同步会复制同步全部内容，以后只传输修改过的文件；
压缩传输：rsync传输数据时，采用压缩解压缩的的方式，因此可以大大节省带宽。
安全：可以使用scp、ssh等方式传输文件，当然也可以通过直接的socket连接。
支持匿名传输，便于进行网站镜像；
选择性保持：符号连接、硬链接、文件属性、权限、时间等。
| 主机名 | IP | 安装的软件 |
| master | 192.168.30.130 | xinetd、rsync |
| slave | 192.168.30.131 |
[root@master ~]# yum install -y xinetd rsync [root@master ~]# vim /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync { disable = no flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID } [root@master ~]# /etc/init.d/xinetd restart Stopping xinetd: [FAILED] Starting xinetd: [ OK ] [root@master ~]# netstat -antup | grep 873 //rsync的工作端口是873 tcp 0 0 :::873 :::* LISTEN 17551/xinetd
以root用户同步测试
[root@slave ~]# mkdir /data-back [root@slave ~]# rsync -azP root@192.168.30.130:/var/www/html/ /data-back/ The authenticity of host '192.168.30.130 (192.168.30.130)' can't be established. RSA key fingerprint is 11:d7:10:45:df:f3:49:15:b4:b5:8e:6d:bc:4f:75:63. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.30.130' (RSA) to the list of known hosts. root@192.168.30.130's password: receiving incremental file list ./ group 965 100% 942.38kB/s 0:00:00 (xfer#1, to-check=3/5) hosts 372 100% 363.28kB/s 0:00:00 (xfer#2, to-check=2/5) passwd 2049 100% 1.95MB/s 0:00:00 (xfer#3, to-check=1/5) shadow 1156 100% 1.10MB/s 0:00:00 (xfer#4, to-check=0/5) sent 90 bytes received 2072 bytes 393.09 bytes/sec total size is 4542 speedup is 2.10
对比文件权限
[root@master html]# getfacl group # file: group # owner: root # group: root user::rw- group::r-- other::r-- 可以看出master和slave上的两个文件权限相同 [root@slave ~]# getfacl /data-back/group getfacl: Removing leading '/' from absolute path names # file: data-back/group # owner: root # group: root user::rw- group::r-- other::r--
基于系统用户的备份
创建上传和下载用户 [root@master ~]# useradd rget01 [root@master ~]# echo "rget01:123456" | chpasswd [root@master ~]# useradd rput01 [root@master ~]# echo "rput01:123456" | chpasswd [root@master ~]# setfacl -R -m user:rput01:rwx /var/www/html/ [root@master ~]# setfacl -R -m default:user:rput01:rwx /var/www/html/ [root@master ~]# getfacl /var/www/html/ getfacl: Removing leading '/' from absolute path names # file: var/www/html/ # owner: root # group: root user::rwx user:rput01:rwx group::r-x mask::rwx other::r-x default:user::rwx default:user:rput01:rwx default:group::r-x default:mask::rwx default:other::r-x [root@master ~]# setfacl -R -m user:rget01:rwx /var/www/html/ [root@master ~]# setfacl -R -m default:user:rget01:rwx /var/www/html/ [root@master ~]# getfacl /var/www/html/ getfacl: Removing leading '/' from absolute path names # file: var/www/html/ # owner: root # group: root user::rwx user:rget01:rwx user:rput01:rwx group::r-x mask::rwx other::r-x default:user::rwx default:user:rget01:rwx default:user:rput01:rwx default:group::r-x default:mask::rwx default:other::r-x [root@master ~]# getfacl /var/www/html/passwd //之前存在的文件权限 getfacl: Removing leading '/' from absolute path names # file: var/www/html/passwd # owner: root # group: root user::rw- user:rget01:rwx user:rput01:rwx group::r-- mask::rwx other::r-- [root@master ~]# cp -r /root/install.log /var/www/html/ [root@master ~]# getfacl /var/www/html/install.log //新建文件的权限 getfacl: Removing leading '/' from absolute path names # file: var/www/html/install.log # owner: root # group: root user::rw- user:rget01:rwx #effective:r-- //如果在后面的试验中同步不到客户端，注意看这里是否有生效的读取的权限 user:rput01:rwx #effective:r-- group::r-x #effective:r-- mask::r-- other::r-- [root@slave ~]# rsync -azP --delete rget01@192.168.30.130:/var/www/html /data-back //注意这里加 / 和不加 /的区别；不加 / 会把目录同步过去 rget01@192.168.30.130's password: receiving incremental file list html/ html/group 965 100% 942.38kB/s 0:00:00 (xfer#1, to-check=4/6) html/hosts 372 100% 90.82kB/s 0:00:00 (xfer#2, to-check=3/6) html/install.log 61802 100% 9.82MB/s 0:00:00 (xfer#3, to-check=2/6) html/passwd 2049 100% 333.50kB/s 0:00:00 (xfer#4, to-check=1/6) html/shadow 1156 100% 188.15kB/s 0:00:00 (xfer#5, to-check=0/6) sent 110 bytes received 16047 bytes 4616.29 bytes/sec total size is 66344 speedup is 4.11 [root@slave ~]# cd /data-back/ [root@slave data-back]# ll total 20 -rw-r--r-- 1 root root 965 May 24 20:36 group -rw-r--r-- 1 root root 372 May 24 20:36 hosts drwxrwxr-x 2 root root 4096 May 24 20:48 html -rw-r--r-- 1 root root 2049 May 24 20:36 passwd ---------- 1 root root 1156 May 24 20:36 shadow [root@slave data-back]# cd /data-back/html/ [root@slave html]# getfacl install.log # file: install.log # owner: root # group: root user::rw- group::r-- other::r--
[root@master ~]# rm -rf /var/www/html/install.log [root@slave ~]# ssh-copy-id rget01@192.168.30.130 rget01@192.168.30.130's password: Now try logging into the machine, with "ssh 'rget01@192.168.30.130'", and check in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. [root@slave ~]# rsync -azP --delete rget01@192.168.30.130:/var/www/html /tmp receiving incremental file list sent 15 bytes received 1319 bytes 2668.00 bytes/sec total size is 95038 speedup is 71.24 [root@slave ~]# rsync -azP --delete rget01@192.168.30.130:/var/www/html /tmp receiving incremental file list deleting html/install.log html/ sent 18 bytes received 1291 bytes 2618.00 bytes/sec total size is 33236 speedup is 25.39
定时任务的时候，设置免密码登陆，写个简单的脚本即可。
[root@slave ~]# rsync -azP root@192.168.30.130:/var/www/html /tmp root@192.168.30.130's password: receiving incremental file list html/ html/install.log 61802 100% 58.94MB/s 0:00:00 (xfer#1, to-check=0/2) sent 34 bytes received 13989 bytes 5609.20 bytes/sec total size is 61802 speedup is 4.41 [root@slave ~]# getfacl /tmp/html/install.log getfacl: Removing leading '/' from absolute path names # file: tmp/html/install.log # owner: root # group: root user::rw- group::r-- other::r-- [root@slave ~]# rsync -azp --delete rget01@192.168.30.130:/var/www/html /tmp //小写的p不打印传输过程，--delete会删除之前备份的文件，可以看出现当前只有html目录，没有之前的install.log文件了 rget01@192.168.30.130's password: [root@slave ~]# ll /tmp/ total 4 drwxrwxr-x 4 root root 4096 May 25 10:38 html -rw-------. 1 root root 0 May 7 23:58 yum.log 注意有些文件，rget01和rput01并没有有效权限 [root@master ~]# cp -r /boot/grub /var/www/html/ [root@master ~]# ll /var/www/html/ total 96 drwxr-xr-x+ 2 root root 4096 May 25 10:46 grub -rw-rwxr--+ 1 root root 61802 May 25 10:30 install.log drwxr-xr-x+ 2 root root 4096 May 25 10:38 pam.d drwxr-xr-x+ 3 root root 4096 May 25 10:38 pango -rw-r--r--+ 1 root root 1861 May 25 10:38 passwd -rw-r--r--+ 1 root root 1820 May 25 10:38 passwd- [root@master ~]# getfacl /var/www/html/grub/ getfacl: Removing leading '/' from absolute path names # file: var/www/html/grub/ # owner: root # group: root user::rwx user:rget01:rwx #effective:r-x user:rput01:rwx #effective:r-x group::r-x mask::r-x other::r-x default:user::rwx default:user:rget01:rwx default:user:rput01:rwx default:group::r-x default:mask::rwx default:other::r-x [root@master ~]# getfacl /var/www/html/grub/ device.map grub.conf minix_stage1_5 stage2 e2fs_stage1_5 iso9660_stage1_5 reiserfs_stage1_5 ufs2_stage1_5 fat_stage1_5 jfs_stage1_5 splash.xpm.gz vstafs_stage1_5 ffs_stage1_5 menu.lst stage1 xfs_stage1_5 [root@master ~]# getfacl /var/www/html/grub/grub.conf getfacl: Removing leading '/' from absolute path names # file: var/www/html/grub/grub.conf # owner: root # group: root user::rw- user:rget01:rwx #effective:--- user:rput01:rwx #effective:--- group::r-x #effective:--- mask::--- other::--- [root@slave ~]# rsync -azP --delete rget01@192.168.30.130:/var/www/html /tmp rget01@192.168.30.130's password: receiving incremental file list html/ html/grub/ html/grub/device.map 63 100% 61.52kB/s 0:00:00 (xfer#1, to-check=78/86) html/grub/e2fs_stage1_5 13380 100% 3.19MB/s 0:00:00 (xfer#2, to-check=77/86) html/grub/fat_stage1_5 12620 100% 2.01MB/s 0:00:00 (xfer#3, to-check=76/86) html/grub/ffs_stage1_5 11748 100% 1.87MB/s 0:00:00 (xfer#4, to-check=75/86) rsync: send_files failed to open "/var/www/html/grub/grub.conf": Permission denied (13) //这种错误是rget01没有有效的读取权限 html/grub/iso9660_stage1_5 11756 100% 1.02MB/s 0:00:00 (xfer#5, to-check=73/86) html/grub/jfs_stage1_5 13268 100% 996.69kB/s 0:00:00 (xfer#6, to-check=72/86) html/grub/menu.lst -> ./grub.conf html/grub/minix_stage1_5 11956 100% 898.14kB/s 0:00:00 (xfer#7, to-check=70/86) html/grub/reiserfs_stage1_5 14412 100% 1005.30kB/s 0:00:00 (xfer#8, to-check=69/86) html/grub/splash.xpm.gz 1341 100% 93.54kB/s 0:00:00 (xfer#9, to-check=68/86) html/grub/stage1 512 100% 35.71kB/s 0:00:00 (xfer#10, to-check=67/86) html/grub/stage2 126100 100% 5.01MB/s 0:00:00 (xfer#11, to-check=66/86) html/grub/ufs2_stage1_5 12024 100% 469.69kB/s 0:00:00 (xfer#12, to-check=65/86) html/grub/vstafs_stage1_5 11364 100% 426.83kB/s 0:00:00 (xfer#13, to-check=64/86) html/grub/xfs_stage1_5 13964 100% 524.49kB/s 0:00:00 (xfer#14, to-check=63/86) sent 310 bytes received 149583 bytes 59957.20 bytes/sec total size is 350356 speedup is 2.34 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1505) [generator=3.0.6] 在master上执行 [root@master ~]# chmod 744 /var/www/html/grub/grub.conf 然后在slave上再次执行 [root@slave ~]# rsync -azP --delete rget01@192.168.30.130:/var/www/html /tmp rget01@192.168.30.130's password: receiving incremental file list html/grub/grub.conf 799 100% 780.27kB/s 0:00:00 (xfer#1, to-check=74/86) sent 35 bytes received 2169 bytes 629.71 bytes/sec total size is 350356 speedup is 158.96 在master上删除一些文件测试同步 [root@master ~]# rm -rf /var/www/html/grub/ 在slave上同步测试 [root@slave ~]# rsync -azP --delete rget01@192.168.30.130:/var/www/html /tmp rget01@192.168.30.130's password: receiving incremental file list deleting html/grub/xfs_stage1_5 deleting html/grub/vstafs_stage1_5 deleting html/grub/ufs2_stage1_5 deleting html/grub/stage2 deleting html/grub/stage1 deleting html/grub/splash.xpm.gz deleting html/grub/reiserfs_stage1_5 deleting html/grub/minix_stage1_5 deleting html/grub/menu.lst deleting html/grub/jfs_stage1_5 deleting html/grub/iso9660_stage1_5 deleting html/grub/grub.conf deleting html/grub/ffs_stage1_5 deleting html/grub/fat_stage1_5 deleting html/grub/e2fs_stage1_5 deleting html/grub/device.map deleting html/grub/ html/ sent 18 bytes received 1322 bytes 536.00 bytes/sec total size is 95038 speedup is 70.92
自动定时备份简单脚本
[root@slave ~]# vim /root/rsync_auto.sh #!/bin/bash rsync -az --delete rget01@192.168.30.130:/var/www/html /web-back tar -czvf data-back-`date +%Y-%m-%d`.tar.gz /data-back/* [root@slave ~]# chmod +x /root/rsync_auto.sh [root@slave ~]# echo "0 3 * * * sh /root/rsync-auto.sh &" > /var/spool/cron/root
基于非系统用户的数据备份实例
在master上建立rsyncd.conf文件 [root@master ~]# vim /etc/rsyncd.conf uid = nobody gid = nobody address = 192.168.30.130 port = 873 hosts allow = 192.168.30.131 use chroot = yes max connections = 5 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log motd file = /etc/rsyncd.motd [rsync-data] path = /var/www/html comment = used for web-data root read only = yes list = yes auth users = rsyncuser secrets file = /etc/rsync.passwd 建立欢迎信息文件 [root@master ~]# echo ""Welcome to Rsync data back" > /etc/rsyncd.motd [root@master ~]# cat !$ cat /etc/rsyncd.motd <h1>Welcome to Rsync data back</h1> 建立rsync用户密码文件 [root@master ~]# vim /etc/rsync.passwd rsyncuser:admin123 修改密码文件权限 [root@master ~]# chmod 600 /etc/rsync.passwd 重启rsync服务（两种方式） [root@master ~]# service xinetd restart Stopping xinetd: [ OK ] Starting xinetd: [ OK ] [root@master ~]# netstat -antup | grep 873 tcp 0 0 :::873 :::* LISTEN 49021/xinetd [root@master ~]# service xinetd stop Stopping xinetd: [ OK ] [root@master ~]# mv /etc/rsync.conf /etc/rsyncd.conf [root@master ~]# rsync --daemon --config=/etc/rsyncd.conf [root@master ~]# netstat -antup | grep 873 tcp 0 0 192.168.30.130:873 0.0.0.0:* LISTEN 49042/rsync
在slave端同步测试
[root@slave ~]# rsync -azP rsyncuser@192.168.30.130::backdata /data-back/ Welcome to Rsync data back //这是我们设置的欢迎信息 Password: #输入admin123，回车 receiving incremental file list ./ passwd 1861 100% 1.77MB/s 0:00:00 (xfer#1, to-check=1/3) rsync: send_files failed to open "/shadow" (in backdata): Permission denied (13) sent 104 bytes received 1037 bytes 175.54 bytes/sec total size is 3088 speedup is 2.71 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1505) [generator=3.0.6] 在master上更改权限重试即可 [root@master ~]# chmod 744 /var/www/html/shadow [root@slave ~]# rsync -azP rsyncuser@192.168.30.130::backdata /data-back/ Welcome to Rsync data back Password: receiving incremental file list shadow 1227 100% 1.17MB/s 0:00:00 (xfer#1, to-check=0/3) sent 82 bytes received 706 bytes 175.11 bytes/sec total size is 3088 speedup is 3.92
实现自动备份的脚本
[root@slave ~]# vim /root/auto-rsync-passwd.sh #!/bin/bash export RSYNC_PASSWORD=admin123 rsync -avz rsyncuser@191.168.30.130::backdata /data-back
生产环境下，开启iptables，然后放行873端口
[root@master ~]# iptables -A INPUT -p tcp --dport 873 -j ACCEPT
Rsync命令 rsync命令和scp命令很相似 -a, --archive archive mode 权限保存模式,相当于 -rlptgoD 参数，存档，递归，保持属性等 -r, --recursive 复制所有下面的资料，递归处理 -p, --perms 保留档案权限 ，文件原有属性 -t, --times 保留时间点，文件原有时间 -g, --group 保留原有属组 -o, --owner 保留档案所有者(root only) -D, --devices 保留device资讯(root only) -l, --links 复制所有的连接 ，拷贝连接文件 -z, --compress 压缩模式, 当资料在传送到目的端进行档案压缩. –azP -H, --hard-links 保留硬链接文件 -A, --acls 保留ACL属性文件，需要配合--perms -P,-P参数和 --partial --progress 相同.只是为了把参数简单化,表示传进度 --version， 输出rsync版本 -v , --verbose 复杂的输出信息 -u, --update 仅仅进行更新，也就是跳过已经存在的目标位置，并且文件时间要晚于要备份的文件，不覆盖新的文件 --port=PORT， 定义rsyncd(daemon)要运行的port(预设为tcp 873) --delete， 删除那些目标位置有的文件而备份源没有的文件 --password-file=FILE ，从 FILE 中得到密码 --bwlimit=KBPS， 限制 I/O 带宽 --filter “-filename”，需要过滤的文件 --exclude=filname，需要过滤的文件 --progress，显示备份过程
inotify实时同步测试
inotify下载地址：
https://sourceforge.net/projects/inotify-tools/files/latest/download
Linux 内核从 2.6.13 版本开始提供了 inotify 通知接口，用来监控文件系统的各种变化情况，如文件存取、删除、移动等。利用这一机制，可以非常方便地实现文件异动告警、增量备份，并针对目录或文件的变化及时作出响应。使用 rsync 工具与 inotify 机制相结合，可以实现触发式备份（实时同步），只要原始位置的文档发生变化，则立即启动增量备份操作，否则处于静态等侍状态，这样一来，就避免了按固定周期备份进存在的延迟性、周期过密等问题。
[root@slave ~]# uname -r 2.6.32-431.el6.x86_64 [root@slave ~]# ls /proc/sys/fs/inotify/ max_queued_events max_user_instances max_user_watches max_queued_events #监控时间队列 max_user_instances #最多监控实例数 max_user_watches #每个实例最多监控的文件数 [root@slave ~]# cat /proc/sys/fs/inotify//max_queued_events 16384 [root@slave ~]# cat /proc/sys/fs/inotify//max_user_ max_user_instances max_user_watches [root@slave ~]# cat /proc/sys/fs/inotify//max_user_instances 128 [root@slave ~]# cat /proc/sys/fs/inotify//max_user_watches 8192 [root@slave ~]# vim /etc/sysctl.conf #末尾添加三行内容 fs.inotify.max_queued_events = 32768 fs.inotify.user_instances = 1024 fs.inotify.max_user_watches = 900000000 使立即生效并查看 [root@slave ~]# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key error: "net.bridge.bridge-nf-call-iptables" is an unknown key error: "net.bridge.bridge-nf-call-arptables" is an unknown key kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 fs.inotify.max_queued_events = 32768 error: "fs.inotify.user_instances" is an unknown key fs.inotify.max_user_watches = 900000000 [root@slave ~]# cat /proc/sys/fs/inotify/max_user_watches 900000000
安装inotify-tools
[root@slave ~]# tar -xf inotify-tools-3.13.tar.gz -C /usr/local/src/ [root@slave ~]# cd /usr/local/src/inotify-tools-3.13/ [root@slave inotify-tools-3.13]# ./configure --prefix=/usr/local/inotify-tools;make;make install [root@slave ~]# ln -s /usr/local/inotify-tools/bin/* /usr/bin/ 使用同样方法在master上也安装inotify-tools inotifywait常用参数： -e 用来指定要监控哪些事件。这些事件包括： create 创建,move 移 动,delete 删除,modify 修改文件内容,attrib 属性更改。 -m 表示持续监控 -r 表示递归整个目录 -q 表示简化输出信息。
测试
首先打开两个终端 在第一个终端内输入 [root@master ~]# ln -s /usr/local/inotify-tools/bin/* /usr/bin/ [root@master ~]# inotifywait -mrq -e create.move,delete,modify 在另外一个终端内输入 [root@master ~]# cp install.log.syslog /var/www/html/ [root@master ~]# mkdir /var/www/html/inotify 然后在第一个中可以看到 [root@master ~]# ln -s /usr/local/inotify-tools/bin/* /usr/bin/ [root@master ~]# inotifywait -mrq -e create.move,delete,modify /var/www/html/ 'create.move,delete,modify' is not a valid event! Run with the '--help' option to see a list of events. [root@master ~]# inotifywait -mrq -e create,move,delete,modify /var/www/html/ /var/www/html/ CREATE install.log.syslog /var/www/html/ MODIFY install.log.syslog /var/www/html/ CREATE,ISDIR inotify
如果配置自动备份脚本
[root@master ~]# vim /root/inotify.sh #!/bin/bash SRC=/var/www/html DST=root@192.168.30.130:/data-back inotifywait -mrq -e modify,delete,create,attrib ${SRC} | while read D E F do /usr/bin/rsync -ahqzt --delete $SRC $DST done #D E F 分别对应前面的三个变量，因为前面输出三段内容 ~
