多puppetmaster，多ca，keepalived+haproxy（nginx）puppet集群搭建
多puppetmaster，多ca，keepalived+haproxy（nginx）puppet集群搭建
一、服务器详情
192.168.122.111 pm01.jq.com pm01 #（puppetmaster服务器）
192.168.122.112 pm02.jq.com pm02 #（puppetmaster服务器）
192.168.122.121 ag01.jq.com ag01 #（puppet agent服务器）
192.168.122.122 ag02.jq.com ag02 #（puppet agent服务器）
192.168.122.131 ca01.jq.com ca01 #（puppet ca服务器）
192.168.122.132 ca02.jq.com ca02 #（puppet ca服务器）
192.168.122.141 lvs01.jq.com lvs01 #（puppet 负载服务器）
192.168.122.142 lvs02.jq.com lvs02 #（puppet 负载服务器）
#vip暂时用于测试，使用ip addr的方式绑定，后续用高可用软件实现bind
192.168.122.130 pc.jq.com pc #（ca服务器的vip，前期绑定在ca01服务器上）
192.168.122.115 lvs.jq.com lvs #（负载服务器的vip，前期绑定在puppetmaster上，后期需要绑定在lvs服务器上）
二、CA服务器部署
CA服务器单独用于签署和撤销证书，当puppetCA服务不可用时，新的客户端将不能获得证书，从而会影响使用，而已签发证书的客户端缺不受影响。因此将CA进行独立队架构，这对容错性而言是非常有必要的。
2.1 安装软件包
[root@ca01 ~]# groupadd -g 3000 puppet
[root@ca01 ~]# useradd -u 3000 -g 3000 puppet
[root@ca01 ~]# yum install puppet puppet-server –y
2.2 bind vip
绑定ca的vip 192.168.122.130到ca01服务器
[root@ca01 ~]#ip addr add 192.168.122.130/24 dev eth0
2.3 生成证书
使用puppet cert命令生成CA服务器与服务器域名证书。生成pc.jq.com和lvs.jq.com两个域名的授权证书文件,前面是证书别名，后面是证书名称。
[root@ca01 ssl]# puppet cert --generate --dns_alt_names pc pc.jq.com
[root@ca01 ssl]# puppet cert --generate --dns_alt_names lvs lvs.jq.com [root@ca01 ssl]# puppet cert --list --all
+ "lvs.jq.com" (SHA256) D6:5B:51:D6:6E:35:61:A4:45:D8:37:17:5B:85:A1:1B:34:BB:2F:D7:48:E8:44:57:B7:1D:42:8E:11:18:81:34 (alt names: "DNS:lvs", "DNS:lvs.jq.com")
+ "pc.jq.com" (SHA256) A7:71:E1:46:1E:F0:F1:70:72:E3:B5:16:03:91:17:6D:68:5B:55:39:B6:79:6B:30:DD:41:ED:10:21:27:2A:33 (alt names: "DNS:pc", "DNS:pc.jq.com")
2.4 配置puppet.conf，添加标签[master]
[root@ca01 ~]# cat /etc/puppet/puppet.conf | grep -v "#"
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
pluginsync = false
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = lvs.jq.com
ca_server = pc.jq.com
environment = jqprd
[master]
confdir = /etc/puppet
certname = pc.jq.com
ca = true #开启CA认证
2.5 启动puppetmaster，CA部署完成
[root@ca01 ~]# /etc/init.d/puppetmaster start
[root@ca01 ~]# chkconfig puppetmaster on
2.6 ca02服务部署
ca02跟ca01的部署方式完全一致，证书是从ca01 拷贝过来的。直接copy /var/lib/ssl/puppet目录
三、PuppetMaster服务器部署
PuppetMaster服务器部署可采用默认的WebRick方式，也可以采用apache+passenger或nginx+passenger方式。
3.1 WebRick方式：
3.1.1 安装软件包
[root@pm01 ~]# groupadd -g 3000 puppet
[root@pm01 ~]# useradd -u 3000 -g 3000 puppet
[root@pm01 ~]# yum install puppet puppet-server -y
3.1.2 设置hosts文件
[root@pm01 ~]# vim /etc/hosts
192.168.122.111 pm01.jq.com pm01
192.168.122.112 pm02.jq.com pm02
192.168.122.121 ag01.jq.com ag01
192.168.122.122 ag02.jq.com ag02
192.168.122.131 ca01.jq.com ca01
192.168.122.132 ca02.jq.com ca02
192.168.122.141 lvs01.jq.com lvs01
192.168.122.142 lvs02.jq.com lvs02
192.168.122.130 pc.jq.com pc
192.168.122.115 lvs.jq.com lvs
3.1.3 bind master vip
绑定LVS的vip 192.168.122.115到pm01服务器，测试用，在没有负载之前，绑定在master上。
ip addr add 192.168.122.115/24 dev eth0
3.1.4 创建证书目录
[root@pm01 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} –p
3.1.5 将puppetca上生成的puppetmaster公钥、私钥和根证书复制到pm01
scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/signed/lvs.jq.com.pem /var/lib/puppet/ssl/certs/lvs.jq.com.pem
scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem
scp -r root@192.168.122.130:/var/lib/puppet/ssl/private_keys/lvs.jq.com.pem /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem
scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem
3.1.6 配置puppet.conf，添加标签[master]，关闭ca
[root@pm01 ~]# grep -v "#" /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
privatekeydir = $ssldir/private_keys { group = service }
hostprivkey = $privatekeydir/$certname.pem { mode = 640 }
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = lvs.jq.com #puppetmaster域名，应该与之前手动生成的证书匹配
ca_server = pc.jq.com #ca证书服务器
[master]
certname = lvs.jq.com #puppetmaster的域名，应该与之前手动生成的证书匹配
ca = false #关闭ca验证
3.1.7 启动puppetmaster服务，puppetmaster部署完成
[root@pm01 ssl]# /etc/init.d/puppetmaster restart
3.1.8 运行puppet命令进行本地证书申请
[root@pm01 ssl]# puppet agent -t
Info: Creating a new SSL key for pm01.jq.com
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for pm01.jq.com
Info: Certificate Request fingerprint (SHA256): 2C:09:32:E1:13:CA:0F:44:3B:93:4B:0F:0E:2D:46:19:3A:37:E1:47:C7:D3:E8:2C:A6:83:44:B3:D3:94:63:D6
Exiting; no certificate found and waitforcert is disabled
3.1.9 登录puppetca进行证书签发
[root@ca01 ~]# puppet cert --sign pm01.jq.com
Notice: Signed certificate request for pm01.jq.com
Notice: Removing file Puppet::SSL::CertificateRequest pm01.jq.com at '/var/lib/puppet/ssl/ca/requests/pm01.jq.com.pem'
3.1.10 再次运行puppet命令进行测试连通性
[root@pm01 ssl]# puppet agent -t
Info: Caching certificate for pm01.jq.com
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for pm01.jq.com
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for pm01.jq.com
Info: Applying configuration version '1425526708'
Notice: Finished catalog run in 0.17 seconds
3.1.11 在kspupt-ca上申请本地证书
[root@ca01 ~]# vim /etc/puppet/puppet.conf
[agent]
server = lvs.jq.com
ca_server = pc.jq.com
[root@ca01 ~]# puppet agent -t
[root@ca01 ~]# puppet cert --sign ca01.jq.com
[root@ca01 ~]# puppet agent –t
3.2 Nginx+Passenger方式：
注：可参考 http://kisspuppet.com/2014/10/20/puppet_learning_ext4/
3.2.1、配置nginx
[root@pm01 ssl]# cat /usr/local/nginx/conf/vhosts/passenger.conf
server {
listen 8140 ssl;
server_name puppetmaster;
passenger_enabled on;
passenger_set_cgi_param HTTP_X_CLIENT_DN $ssl_client_s_dn;
passenger_set_cgi_param HTTP_X_CLIENT_VERIFY $ssl_client_verify;
proxy_buffer_size 4000k;
proxy_buffering on;
proxy_buffers 32 1280k;
proxy_busy_buffers_size 17680k;
client_max_body_size 10m;
client_body_buffer_size 4096k;
access_log /var/log/nginx/puppet_access.log;
error_log /var/log/nginx/puppet_error.log;
root /etc/puppet/rack/public;
ssl off;
ssl_session_timeout 5m;
ssl_certificate /var/lib/puppet/ssl/certs/lvs.jq.com.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem;
ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem;
ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;
ssl_verify_client optional;
ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_prefer_server_ciphers on;
ssl_verify_depth 1;
ssl_session_cache shared:SSL:128m;
# File sections
location /production/file_content/files/ {
types { }
default_type application/x-raw;
alias /etc/puppet/files/;
}
}
3.2.2、配置puppet.conf
[root@pm01 ssl]# grep -v "#" /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
privatekeydir = $ssldir/private_keys { group = service }
hostprivkey = $privatekeydir/$certname.pem { mode = 640 }
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = lvs.jq.com
ca_server = pc.jq.com
[master]
certname = lvs.jq.com
ca = false
ssl_client_verify_header = HTTP_X_CLIENT_VERIFY
ssl_client_header = HTTP_X_CLIENT_DN
3.3 master02服务器部署
master02的部署与master01的完全一致，包括拷贝证书部分
4 Puppet LB负载均衡器部署
4.1 puppet认证建立
4.1.1、安装软件包
[root@lvs01 ~]# groupadd -g 3000 puppet
[root@lvs01 ~]# useradd -u 3000 -g 3000 puppet
[root@lvs01 ~]# yum install puppet
4.1.2、编辑hosts文件
[root@lvs01 ~]# vim /etc/hosts
192.168.122.111 pm01.jq.com pm01
192.168.122.112 pm02.jq.com pm02
192.168.122.121 ag01.jq.com ag01
192.168.122.122 ag02.jq.com ag02
192.168.122.131 ca01.jq.com ca01
192.168.122.132 ca02.jq.com ca02
192.168.122.141 lvs01.jq.com lvs01
192.168.122.142 lvs02.jq.com lvs02
192.168.122.130 pc.jq.com pc
192.168.122.115 lvs.jq.com lvs
4.1.3、创建证书目录
[root@lvs01 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} –p
4.1.4、将ca上生成的puppetmaster公钥、私钥和根证书复制到lvs01
scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/signed/lvs.jq.com.pem /var/lib/puppet/ssl/certs/lvs.jq.com.pem
scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem
scp -r root@192.168.122.130:/var/lib/puppet/ssl/private_keys/lvs.jq.com.pem /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem
scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem
4.1.5、配置puppet.conf，修改标签[agent]，增加server和ca_server字段
[root@lvs01 ~]# vim /etc/puppet/puppet.conf
[agent]
server = lvs.jq.com
ca_server = pc.jq.com
4.1.6、运行puppet命令进行本地证书申请
[root@lvs01 ~]# puppet agent -t
4.1.7、登录ca进行证书签发
[root@ca01 ~]# puppet cert --sign lvs01.jq.com
4.1.8、再次运行puppet命令进行测试连通性
[root@lvs01 ~]# puppet agent -t
Info: Caching certificate for lvs01.jq.com
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for lvs01.jq.com
Info: Loading facts
Info: Caching catalog for lvs01.jq.com
Info: Applying configuration version '1425527450'
Notice: Finished catalog run in 0.24 seconds
4.2 安装并配置nginx负载均衡器
4.2.1、安装nginx软件
[root@lvs01 ~]# groupadd -g 3001 nginx
[root@lvs01 ~]# useradd -u 3001 -g 3001 nginx
[root@lvs01 ~]# yum install nginx
4.2.2、临时设置VIP地址（后面通过高可用软件代替）
[root@lvs01 ~]# ip addr add 192.168.122.115/24 dev eth0
此处请将之前bind到pm01的vip取消
4.2.3、配置nginx虚拟主机，添加upstrem
[root@lvs01 ~]# cat /etc/nginx/conf.d/puppetmaster.conf
upstream puppet-master {
server 192.168.122.111:8140;
server 192.168.122.112:8140;
}
server {
listen 8140 ssl;
server_name puppetmaster;
access_log /var/log/nginx/puppet_access.log;
error_log /var/log/nginx/puppet_error.log;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-Client-Verify $ssl_client_verify;
client_max_body_size 100m;
client_body_buffer_size 1024k;
proxy_buffer_size 100m;
proxy_buffers 8 100m;
proxy_busy_buffers_size 100m;
proxy_temp_file_write_size 100m;
proxy_read_timeout 500;
ssl on;
ssl_session_timeout 5m;
ssl_certificate /var/lib/puppet/ssl/certs/lvs.jq.com.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem;
ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem;
ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;
ssl_verify_client optional;
ssl_prefer_server_ciphers on;
ssl_verify_depth 1;
ssl_session_cache shared:SSL:128m;
location / {
proxy_redirect off;
proxy_pass https://puppet-master;
}
}
4.2.4、编辑hosts文件，puppetmaster解析指向VIP
[root@lvs01 ~]# vim /etc/hosts
192.168.122.111 pm01.jq.com pm01
192.168.122.112 pm02.jq.com pm02
192.168.122.121 ag01.jq.com ag01
192.168.122.122 ag02.jq.com ag02
192.168.122.131 ca01.jq.com ca01
192.168.122.132 ca02.jq.com ca02
192.168.122.141 lvs01.jq.com lvs01
192.168.122.142 lvs02.jq.com lvs02
192.168.122.130 pc.jq.com pc
192.168.122.115 lvs.jq.com lvs
4.2.5、修改ca01和pm01的hosts文件puppetmaster解析
[root@kspupt-ca1 ~]# vim /etc/hosts
192.168.122.111 pm01.jq.com pm01
192.168.122.112 pm02.jq.com pm02
192.168.122.121 ag01.jq.com ag01
192.168.122.122 ag02.jq.com ag02
192.168.122.131 ca01.jq.com ca01
192.168.122.132 ca02.jq.com ca02
192.168.122.141 lvs01.jq.com lvs01
192.168.122.142 lvs02.jq.com lvs02
192.168.122.130 pc.jq.com pc
192.168.122.115 lvs.jq.com lvs
[root@pm01 ~]# vim /etc/hosts
192.168.122.111 pm01.jq.com pm01
192.168.122.112 pm02.jq.com pm02
192.168.122.121 ag01.jq.com ag01
192.168.122.122 ag02.jq.com ag02
192.168.122.131 ca01.jq.com ca01
192.168.122.132 ca02.jq.com ca02
192.168.122.141 lvs01.jq.com lvs01
192.168.122.142 lvs02.jq.com lvs02
192.168.122.130 pc.jq.com pc
192.168.122.115 lvs.jq.com lvs
4.2.6、启动nginx服务器
[root@lvs01 ~]# /etc/init.d/nginx start
4.2.7、再次运行puppet命令进行测试连通性
[root@kspupt-ca1 ~]# puppet agent -t
[root@pm01 ~]# puppet agent -t
[root@lvs01 ~]# puppet agent -t
[root@pm01 ~]# tailf /var/log/nginx/puppet_access.log
[root@lvs01 ~]# tailf /var/log/nginx/puppet_access.log
4.3 安装配置Haproxy负载均衡
安装haproxy和keepalived过程略去，网上很多教程
[root@lvs01 keepalived]# cat /etc/haproxy/haproxy.cfg
global
maxconn 40000
ulimit-n 500000
log 127.0.0.1 local0
uid 99
gid 99
chroot /tmp
# nbproc 4
daemon
defaults
log global
retries 2
option redispatch
option dontlognull
balance roundrobin
timeout connect 30000ms
timeout client 30000ms
timeout server 30000ms
timeout check 2000
listen admin_stats
bind 0.0.0.0:8080
mode http
stats refresh 5s
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /haproxy
stats auth admin:password
listen puppetmaster *:8140
mode tcp
option ssl-hello-chk
# option tcplog
#balance source
# balance roundrobin
balance source
server pm01 pm01.jq.com:8140 check inter 2000 fall 3
server pm02 pm02.jq.com:8140 check inter 2000 fall 3
4.4 配置keepalived，取消vip 通过ip addr 绑定
Keepalived的备机配置略去，网上也可以搜索到，需要修改的地方很少。
[root@lvs01 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
test@gmail.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_http_port {
script "/etc/keepalived/check_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_http_port
}
virtual_ipaddress {
192.168.122.115 #此处为负载的VIP，配置keepalived之后，切记去掉之前ip addr的绑定
}
}
4.4.1 Keepalived监控haproxy脚本
[root@lvs01 ~]# cat /etc/keepalived/check_haproxy.sh
#!/bin/bash
. /etc/profile
A=`ps -C haproxy --no-header |wc -l`
if [ $A -eq 0 ];then
/etc/init.d/haproxy start
sleep 3
if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then
/etc/init.d/keepalived stop
fi
fi
4.5 Lvs02服务器部署
Lvs02的配置与lvs01的配置完全一致，将此服务器作为lvs01的备机，包括keepalived+haproxy。
几乎照搬了http://kisspuppet.com/2014/10/21/puppet_learning_ext6/ 的文章，非常感谢kisspuppet！
