[SaltStack] Salt高可用和负载均衡部署

Saltstack HA部署

Salt官网是有HA/Rebalance/failover解决方案的, 但版本必须是2014.7, 目前该版本还没有release, 从官网下载的源码包安装使用测试起来一直有问题, 因此采用DNS方案实现HA/Rebalance/failover架构.

• Master01是leader节点, Master02和Master03是follower节点;

• leader上的配置, state文件, return等文件变动都会完全同步到其他follower节点上;

• Master01绑定DNS, Minion上的配置是:

  master:
  - saltstack.online.net
  - 2.2.2.2
  - 3.3.3.3

  一台Minion启动是会使用multi-master模块建立每个Master的TCP链接, Zero-MQ通信机制

• 配置Minion自动拉取时, 随机选取Master节点, 达到Rebalance目的;

• 当leader节点Master宕机, 只需要切换DNS, 更新Minion-cnf配置即可, 达到半自动HA目的;

  一般情况下minion节点配置每天同步一次master配置, 因此完全有主动时间处理master节点宕机问题 -😃

---

• Saltstack版本

  • Master版本:

    Salt:2014.1.10
    Python:2.6.6(r266:84292, Sep 12 2011, 14:03:14)
    Jinja2:unknown
    M2Crypto:0.20.2
    Msgpack-python:0.1.13
    Msgpack-pure:Not Installed
    Pycrypto:2.0.1
    PyYAML:3.10
    PyZMQ:2.2.0.1
    ZMQ:3.2.2

  • Minion版本:

    Salt: 2014.1.10
    Python: 2.6.6 (r266:84292, Sep 12 2011, 14:03:14)
    Jinja2: unknown
    M2Crypto: 0.20.2
    msgpack-python: 0.1.13
    msgpack-pure: Not Installed
    pycrypto: 2.0.1
    PyYAML: 3.10
    PyZMQ: 2.2.0.1
    ZMQ: 3.2.2

• Salt config

  • Master site config

    cat master | grep -vE "^#|$"
    file_roots:
    base:
    - /etc/salt/base

  • Minion site config

    cat minion | grep -vE "^#|$"
    master:
    - saltstack.online.net
    - 2.2.2.2
    - 3.3.3.3

---

• Master01推送:

  salt '' state.highstate
  :
  ----------
  ID: /root/myfile-renotest.txt
  Function: file.managed
  Result: True
  Comment: File /root/myfile-renotest.txt is in the correct state
  Changes:

  Summary
  ------------
  Succeeded: 1
  Failed: 0
  ------------
  Total: 1

• Master02推送:

  salt '' state.highstate
  :
  ----------
  ID: /root/myfile-renotest.txt
  Function: file.managed
  Result: True
  Comment: File /root/myfile-renotest.txt is in the correct state
  Changes:

  Summary
  ------------
  Succeeded: 1
  Failed: 0
  ------------
  Total: 1

• Minion拉取:

  salt-call state.highstate --master 2.2.2.2
  [INFO ] Loading fresh modules for state activity
  [INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/modules'
  [INFO ] Syncing modules for environment 'base'
  [INFO ] Loading cache from salt://_modules, for base)
  [INFO ] Caching directory '_modules/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/states'
  [INFO ] Syncing states for environment 'base'
  [INFO ] Loading cache from salt://_states, for base)
  [INFO ] Caching directory '_states/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
  [INFO ] Syncing grains for environment 'base'
  [INFO ] Loading cache from salt://_grains, for base)
  [INFO ] Caching directory '_grains/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
  [INFO ] Syncing renderers for environment 'base'
  [INFO ] Loading cache from salt://_renderers, for base)
  [INFO ] Caching directory '_renderers/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/returners'
  [INFO ] Syncing returners for environment 'base'
  [INFO ] Loading cache from salt://_returners, for base)
  [INFO ] Caching directory '_returners/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
  [INFO ] Syncing outputters for environment 'base'
  [INFO ] Loading cache from salt://_outputters, for base)
  [INFO ] Caching directory '_outputters/' for environment 'base'
  [INFO ] Loading fresh modules for state activity
  [INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://myfile/init.sls'
  [INFO ] Running state [/root/myfile-renotest.txt] at time 14:23:24.228722
  [INFO ] Executing state file.managed for /root/myfile-renotest.txt
  [INFO ] File /root/myfile-renotest.txt is in the correct state
  [INFO ] Completed state [/root/myfile-renotest.txt] at time 14:23:24.679833
  local:
  ----------
  ID: /root/myfile-renotest.txt
  Function: file.managed
  Result: True
  Comment: File /root/myfile-renotest.txt is in the correct state
  Changes:

  Summary
  ------------
  Succeeded: 1
  Failed: 0
  ------------
  Total: 1

• 模拟Master01宕机

  stop Master01

  /etc/init.d/salt-master stop
  Stopping salt-master daemon: [ OK ]

• 切换DNS

  将dns绑定到2.2.2.2上

• 修改minion-conf文件

  cat minion
  master:
  - saltstack.online.net
  - 3.3.3.3
  id: {{ grains['localhost'] }}
  return: mysql_minion_return

• 推送minion-conf文件到各个Minion节点, 并且restart minion

  先看看state文件

  cat init.sls
  get_minion:
  file.managed:
  - source: salt://minion_conf/minion
  - name: /etc/salt/minion
  - user: root
  - group: root
  - mode: 640
  - template: jinja
  cmd.wait:
  - name: /etc/init.d/salt-minion restart
  - watch:
  - file: get_minion

  这里minion会watch minion-conf文件, 如果本地minion配置文件更新, 则cmd.wait来restart salt-minion

• Master02推送:

  salt '' state.highstate
  :
  ----------
  ID: /root/myfile-renotest.txt
  Function: file.managed
  Result: True
  Comment: File /root/myfile-renotest.txt is in the correct state
  Changes:

  Summary
  ------------
  Succeeded: 1
  Failed: 0
  ------------
  Total: 1

• Master03推送:

  salt '' state.highstate
  :
  ----------
  ID: /root/myfile-renotest.txt
  Function: file.managed
  Result: True
  Comment: File /root/myfile-renotest.txt is in the correct state
  Changes:

  Summary
  ------------
  Succeeded: 1
  Failed: 0
  ------------
  Total: 1

• Minion拉取:

  salt-call state.highstate --master 2.2.2.2
  [INFO ] Loading fresh modules for state activity
  [INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/modules'
  [INFO ] Syncing modules for environment 'base'
  [INFO ] Loading cache from salt://_modules, for base)
  [INFO ] Caching directory '_modules/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/states'
  [INFO ] Syncing states for environment 'base'
  [INFO ] Loading cache from salt://_states, for base)
  [INFO ] Caching directory '_states/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
  [INFO ] Syncing grains for environment 'base'
  [INFO ] Loading cache from salt://_grains, for base)
  [INFO ] Caching directory '_grains/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
  [INFO ] Syncing renderers for environment 'base'
  [INFO ] Loading cache from salt://_renderers, for base)
  [INFO ] Caching directory '_renderers/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/returners'
  [INFO ] Syncing returners for environment 'base'
  [INFO ] Loading cache from salt://_returners, for base)
  [INFO ] Caching directory '_returners/' for environment 'base'
  [INFO ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
  [INFO ] Syncing outputters for environment 'base'
  [INFO ] Loading cache from salt://_outputters, for base)
  [INFO ] Caching directory '_outputters/' for environment 'base'
  [INFO ] Loading fresh modules for state activity
  [INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://myfile/init.sls'
  [INFO ] Running state [/root/myfile-renotest.txt] at time 14:23:24.228722
  [INFO ] Executing state file.managed for /root/myfile-renotest.txt
  [INFO ] File /root/myfile-renotest.txt is in the correct state
  [INFO ] Completed state [/root/myfile-renotest.txt] at time 14:23:24.679833
  local:
  ----------
  ID: /root/myfile-renotest.txt
  Function: file.managed
  Result: True
  Comment: File /root/myfile-renotest.txt is in the correct state
  Changes:

  Summary
  ------------
  Succeeded: 1
  Failed: 0
  ------------
  Total: 1

可以看到无论是Master推送还是Minion来拉去都是ok的

如果没有及时切换dns会导致minion可能拉取到fail状态的master, 从而auth失败:

salt-call state.highstate
[WARNING ] Attempted to authenticate with master salt.dbfree.tbsite.net and failed
[WARNING ] Attempted to authenticate with master 10.194.232.8 and failed
local:
Data failed to compile:
----------
The function "state.highstate" is running as PID 32237 and was started at 2014, Sep 03 15:38:11.202456 with jid 20140903153811202456

切换dns后minion的拉去是没问题的; 状态ok的master推送也是正常的, 但是可能return log会有多个, 影响日志入库后校验结果.

---

先整理到这里吧! 先撤, 回家休息..

最近白天事情多, 晚上还要机房演练, 感觉有点累, i need short rest -😃