[SaltStack] Salt高可用和负载均衡部署

Saltstack HA部署

Salt官网是有HA/Rebalance/failover解决方案的, 但版本必须是2014.7, 目前该版本还没有release, 从官网下载的源码包安装使用测试起来一直有问题, 因此采用DNS方案实现HA/Rebalance/failover架构.

• Master01是leader节点, Master02和Master03是follower节点;

• leader上的配置, state文件, return等文件变动都会完全同步到其他follower节点上;

• Master01绑定DNS, Minion上的配置是:

    master:
      - saltstack.online.net
      - 2.2.2.2
      - 3.3.3.3
  

  一台Minion启动是会使用multi-master模块建立每个Master的TCP链接, Zero-MQ通信机制

• 配置Minion自动拉取时, 随机选取Master节点, 达到Rebalance目的;

• 当leader节点Master宕机, 只需要切换DNS, 更新Minion-cnf配置即可, 达到半自动HA目的;

  一般情况下minion节点配置每天同步一次master配置, 因此完全有主动时间处理master节点宕机问题 -😃

---

• Saltstack版本

  • Master版本:

      Salt:2014.1.10
      Python:2.6.6(r266:84292, Sep 12 2011, 14:03:14)
      Jinja2:unknown
      M2Crypto:0.20.2
      Msgpack-python:0.1.13
      Msgpack-pure:Not Installed
      Pycrypto:2.0.1
      PyYAML:3.10
      PyZMQ:2.2.0.1
      ZMQ:3.2.2
    

  • Minion版本:

      Salt: 2014.1.10
      Python: 2.6.6 (r266:84292, Sep 12 2011, 14:03:14)
      Jinja2: unknown
      M2Crypto: 0.20.2
      msgpack-python: 0.1.13
      msgpack-pure: Not Installed
      pycrypto: 2.0.1
      PyYAML: 3.10
      PyZMQ: 2.2.0.1
      ZMQ: 3.2.2
    

• Salt config

  • Master site config

      cat master  | grep -vE "^#|^$"
      file_roots:
        base:
          - /etc/salt/base
    

  • Minion site config

      cat minion | grep -vE "^#|^$"
      master:
        - saltstack.online.net
        - 2.2.2.2
        - 3.3.3.3
    

---

• Master01推送:

    salt '<minion-id>' state.highstate 
    <minion-id>:
    ----------
          ID: /root/myfile-renotest.txt
    Function: file.managed
      Result: True
     Comment: File /root/myfile-renotest.txt is in the correct state
     Changes:   
  
    Summary
    ------------
    Succeeded: 1
    Failed:    0
    ------------
    Total:     1
  

• Master02推送:

    salt '<minion-id>' state.highstate 
    <minion-id>:
    ----------
          ID: /root/myfile-renotest.txt
    Function: file.managed
      Result: True
     Comment: File /root/myfile-renotest.txt is in the correct state
     Changes:   
  
    Summary
    ------------
    Succeeded: 1
    Failed:    0
    ------------
    Total:     1
  

• Minion拉取:

    salt-call state.highstate --master 2.2.2.2
    [INFO    ] Loading fresh modules for state activity
    [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
    [INFO    ] Syncing modules for environment 'base'
    [INFO    ] Loading cache from salt://_modules, for base)
    [INFO    ] Caching directory '_modules/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
    [INFO    ] Syncing states for environment 'base'
    [INFO    ] Loading cache from salt://_states, for base)
    [INFO    ] Caching directory '_states/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
    [INFO    ] Syncing grains for environment 'base'
    [INFO    ] Loading cache from salt://_grains, for base)
    [INFO    ] Caching directory '_grains/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
    [INFO    ] Syncing renderers for environment 'base'
    [INFO    ] Loading cache from salt://_renderers, for base)
    [INFO    ] Caching directory '_renderers/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
    [INFO    ] Syncing returners for environment 'base'
    [INFO    ] Loading cache from salt://_returners, for base)
    [INFO    ] Caching directory '_returners/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
    [INFO    ] Syncing outputters for environment 'base'
    [INFO    ] Loading cache from salt://_outputters, for base)
    [INFO    ] Caching directory '_outputters/' for environment 'base'
    [INFO    ] Loading fresh modules for state activity
    [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://myfile/init.sls'
    [INFO    ] Running state [/root/myfile-renotest.txt] at time 14:23:24.228722
    [INFO    ] Executing state file.managed for /root/myfile-renotest.txt
    [INFO    ] File /root/myfile-renotest.txt is in the correct state
    [INFO    ] Completed state [/root/myfile-renotest.txt] at time 14:23:24.679833
    local:
    ----------
    	  ID: /root/myfile-renotest.txt
    Function: file.managed
      Result: True
     Comment: File /root/myfile-renotest.txt is in the correct state
     Changes:   
  
    Summary
    ------------
    Succeeded: 1
    Failed:    0
    ------------
    Total:     1
  

• 模拟Master01宕机

  stop Master01

    /etc/init.d/salt-master stop
    Stopping salt-master daemon:                               [  OK  ]
  

• 切换DNS

  将dns绑定到2.2.2.2上

• 修改minion-conf文件

    cat minion
    master:
      - saltstack.online.net
      - 3.3.3.3
    id: {{ grains['localhost'] }}
    return: mysql_minion_return
  

• 推送minion-conf文件到各个Minion节点, 并且restart minion

  先看看state文件

    cat init.sls
    get_minion:
      file.managed:
        - source: salt://minion_conf/minion
        - name: /etc/salt/minion
        - user: root
        - group: root
        - mode: 640
        - template: jinja
      cmd.wait:
        - name: /etc/init.d/salt-minion restart
        - watch:
        - file: get_minion
  

  这里minion会watch minion-conf文件, 如果本地minion配置文件更新, 则cmd.wait来restart salt-minion

• Master02推送:

    salt '<minion-id>' state.highstate 
    <minion-id>:
    ----------
          ID: /root/myfile-renotest.txt
    Function: file.managed
      Result: True
     Comment: File /root/myfile-renotest.txt is in the correct state
     Changes:   
  
    Summary
    ------------
    Succeeded: 1
    Failed:    0
    ------------
    Total:     1
  

• Master03推送:

    salt '<minion-id>' state.highstate 
    <minion-id>:
    ----------
          ID: /root/myfile-renotest.txt
    Function: file.managed
      Result: True
     Comment: File /root/myfile-renotest.txt is in the correct state
     Changes:   
  
    Summary
    ------------
    Succeeded: 1
    Failed:    0
    ------------
    Total:     1
  

• Minion拉取:

    salt-call state.highstate --master 2.2.2.2
    [INFO    ] Loading fresh modules for state activity
    [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
    [INFO    ] Syncing modules for environment 'base'
    [INFO    ] Loading cache from salt://_modules, for base)
    [INFO    ] Caching directory '_modules/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
    [INFO    ] Syncing states for environment 'base'
    [INFO    ] Loading cache from salt://_states, for base)
    [INFO    ] Caching directory '_states/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
    [INFO    ] Syncing grains for environment 'base'
    [INFO    ] Loading cache from salt://_grains, for base)
    [INFO    ] Caching directory '_grains/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
    [INFO    ] Syncing renderers for environment 'base'
    [INFO    ] Loading cache from salt://_renderers, for base)
    [INFO    ] Caching directory '_renderers/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
    [INFO    ] Syncing returners for environment 'base'
    [INFO    ] Loading cache from salt://_returners, for base)
    [INFO    ] Caching directory '_returners/' for environment 'base'
    [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
    [INFO    ] Syncing outputters for environment 'base'
    [INFO    ] Loading cache from salt://_outputters, for base)
    [INFO    ] Caching directory '_outputters/' for environment 'base'
    [INFO    ] Loading fresh modules for state activity
    [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://myfile/init.sls'
    [INFO    ] Running state [/root/myfile-renotest.txt] at time 14:23:24.228722
    [INFO    ] Executing state file.managed for /root/myfile-renotest.txt
    [INFO    ] File /root/myfile-renotest.txt is in the correct state
    [INFO    ] Completed state [/root/myfile-renotest.txt] at time 14:23:24.679833
    local:
    ----------
    	  ID: /root/myfile-renotest.txt
    Function: file.managed
      Result: True
     Comment: File /root/myfile-renotest.txt is in the correct state
     Changes:   
  
    Summary
    ------------
    Succeeded: 1
    Failed:    0
    ------------
    Total:     1
  

可以看到无论是Master推送还是Minion来拉去都是ok的

如果没有及时切换dns会导致minion可能拉取到fail状态的master, 从而auth失败:

	salt-call state.highstate    
	[WARNING ] Attempted to authenticate with master salt.dbfree.tbsite.net and failed
	[WARNING ] Attempted to authenticate with master 10.194.232.8 and failed
	local:
	    Data failed to compile:
	----------
	    The function "state.highstate" is running as PID 32237 and was started at 2014, Sep 03 15:38:11.202456 with jid 20140903153811202456

切换dns后minion的拉去是没问题的; 状态ok的master推送也是正常的, 但是可能return log会有多个, 影响日志入库后校验结果.

---

先整理到这里吧! 先撤, 回家休息..

最近白天事情多, 晚上还要机房演练, 感觉有点累, i need short rest -😃