[SaltStack] Salt高可用和负载均衡部署

Saltstack HA部署

Salt官网是有HA/Rebalance/failover解决方案的, 但版本必须是2014.7, 目前该版本还没有release, 从官网下载的源码包安装使用测试起来一直有问题, 因此采用DNS方案实现HA/Rebalance/failover架构.

  • Master01是leader节点, Master02和Master03是follower节点;

  • leader上的配置, state文件, return等文件变动都会完全同步到其他follower节点上;

  • Master01绑定DNS, Minion上的配置是:

      master:
        - saltstack.online.net
        - 2.2.2.2
        - 3.3.3.3
    

    一台Minion启动是会使用multi-master模块建立每个Master的TCP链接, Zero-MQ通信机制

  • 配置Minion自动拉取时, 随机选取Master节点, 达到Rebalance目的;

  • 当leader节点Master宕机, 只需要切换DNS, 更新Minion-cnf配置即可, 达到半自动HA目的;

    一般情况下minion节点配置每天同步一次master配置, 因此完全有主动时间处理master节点宕机问题 -😃


  • Saltstack版本

    • Master版本:

        Salt:2014.1.10
        Python:2.6.6(r266:84292, Sep 12 2011, 14:03:14)
        Jinja2:unknown
        M2Crypto:0.20.2
        Msgpack-python:0.1.13
        Msgpack-pure:Not Installed
        Pycrypto:2.0.1
        PyYAML:3.10
        PyZMQ:2.2.0.1
        ZMQ:3.2.2
      
    • Minion版本:

        Salt: 2014.1.10
        Python: 2.6.6 (r266:84292, Sep 12 2011, 14:03:14)
        Jinja2: unknown
        M2Crypto: 0.20.2
        msgpack-python: 0.1.13
        msgpack-pure: Not Installed
        pycrypto: 2.0.1
        PyYAML: 3.10
        PyZMQ: 2.2.0.1
        ZMQ: 3.2.2
      
  • Salt config

    • Master site config

        cat master  | grep -vE "^#|^$"
        file_roots:
          base:
            - /etc/salt/base
      
    • Minion site config

        cat minion | grep -vE "^#|^$"
        master:
          - saltstack.online.net
          - 2.2.2.2
          - 3.3.3.3
      

  • Master01推送:

      salt '<minion-id>' state.highstate 
      <minion-id>:
      ----------
            ID: /root/myfile-renotest.txt
      Function: file.managed
        Result: True
       Comment: File /root/myfile-renotest.txt is in the correct state
       Changes:   
    
      Summary
      ------------
      Succeeded: 1
      Failed:    0
      ------------
      Total:     1
    
  • Master02推送:

      salt '<minion-id>' state.highstate 
      <minion-id>:
      ----------
            ID: /root/myfile-renotest.txt
      Function: file.managed
        Result: True
       Comment: File /root/myfile-renotest.txt is in the correct state
       Changes:   
    
      Summary
      ------------
      Succeeded: 1
      Failed:    0
      ------------
      Total:     1
    
  • Minion拉取:

      salt-call state.highstate --master 2.2.2.2
      [INFO    ] Loading fresh modules for state activity
      [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
      [INFO    ] Syncing modules for environment 'base'
      [INFO    ] Loading cache from salt://_modules, for base)
      [INFO    ] Caching directory '_modules/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
      [INFO    ] Syncing states for environment 'base'
      [INFO    ] Loading cache from salt://_states, for base)
      [INFO    ] Caching directory '_states/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
      [INFO    ] Syncing grains for environment 'base'
      [INFO    ] Loading cache from salt://_grains, for base)
      [INFO    ] Caching directory '_grains/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
      [INFO    ] Syncing renderers for environment 'base'
      [INFO    ] Loading cache from salt://_renderers, for base)
      [INFO    ] Caching directory '_renderers/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
      [INFO    ] Syncing returners for environment 'base'
      [INFO    ] Loading cache from salt://_returners, for base)
      [INFO    ] Caching directory '_returners/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
      [INFO    ] Syncing outputters for environment 'base'
      [INFO    ] Loading cache from salt://_outputters, for base)
      [INFO    ] Caching directory '_outputters/' for environment 'base'
      [INFO    ] Loading fresh modules for state activity
      [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://myfile/init.sls'
      [INFO    ] Running state [/root/myfile-renotest.txt] at time 14:23:24.228722
      [INFO    ] Executing state file.managed for /root/myfile-renotest.txt
      [INFO    ] File /root/myfile-renotest.txt is in the correct state
      [INFO    ] Completed state [/root/myfile-renotest.txt] at time 14:23:24.679833
      local:
      ----------
      	  ID: /root/myfile-renotest.txt
      Function: file.managed
        Result: True
       Comment: File /root/myfile-renotest.txt is in the correct state
       Changes:   
    
      Summary
      ------------
      Succeeded: 1
      Failed:    0
      ------------
      Total:     1
    
  • 模拟Master01宕机

    stop Master01

      /etc/init.d/salt-master stop
      Stopping salt-master daemon:                               [  OK  ]
    
  • 切换DNS

    将dns绑定到2.2.2.2上

  • 修改minion-conf文件

      cat minion
      master:
        - saltstack.online.net
        - 3.3.3.3
      id: {{ grains['localhost'] }}
      return: mysql_minion_return
    
  • 推送minion-conf文件到各个Minion节点, 并且restart minion

    先看看state文件

      cat init.sls
      get_minion:
        file.managed:
          - source: salt://minion_conf/minion
          - name: /etc/salt/minion
          - user: root
          - group: root
          - mode: 640
          - template: jinja
        cmd.wait:
          - name: /etc/init.d/salt-minion restart
          - watch:
          - file: get_minion
    

    这里minion会watch minion-conf文件, 如果本地minion配置文件更新, 则cmd.wait来restart salt-minion

  • Master02推送:

      salt '<minion-id>' state.highstate 
      <minion-id>:
      ----------
            ID: /root/myfile-renotest.txt
      Function: file.managed
        Result: True
       Comment: File /root/myfile-renotest.txt is in the correct state
       Changes:   
    
      Summary
      ------------
      Succeeded: 1
      Failed:    0
      ------------
      Total:     1
    
  • Master03推送:

      salt '<minion-id>' state.highstate 
      <minion-id>:
      ----------
            ID: /root/myfile-renotest.txt
      Function: file.managed
        Result: True
       Comment: File /root/myfile-renotest.txt is in the correct state
       Changes:   
    
      Summary
      ------------
      Succeeded: 1
      Failed:    0
      ------------
      Total:     1
    
  • Minion拉取:

      salt-call state.highstate --master 2.2.2.2
      [INFO    ] Loading fresh modules for state activity
      [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
      [INFO    ] Syncing modules for environment 'base'
      [INFO    ] Loading cache from salt://_modules, for base)
      [INFO    ] Caching directory '_modules/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
      [INFO    ] Syncing states for environment 'base'
      [INFO    ] Loading cache from salt://_states, for base)
      [INFO    ] Caching directory '_states/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
      [INFO    ] Syncing grains for environment 'base'
      [INFO    ] Loading cache from salt://_grains, for base)
      [INFO    ] Caching directory '_grains/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
      [INFO    ] Syncing renderers for environment 'base'
      [INFO    ] Loading cache from salt://_renderers, for base)
      [INFO    ] Caching directory '_renderers/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
      [INFO    ] Syncing returners for environment 'base'
      [INFO    ] Loading cache from salt://_returners, for base)
      [INFO    ] Caching directory '_returners/' for environment 'base'
      [INFO    ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
      [INFO    ] Syncing outputters for environment 'base'
      [INFO    ] Loading cache from salt://_outputters, for base)
      [INFO    ] Caching directory '_outputters/' for environment 'base'
      [INFO    ] Loading fresh modules for state activity
      [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://myfile/init.sls'
      [INFO    ] Running state [/root/myfile-renotest.txt] at time 14:23:24.228722
      [INFO    ] Executing state file.managed for /root/myfile-renotest.txt
      [INFO    ] File /root/myfile-renotest.txt is in the correct state
      [INFO    ] Completed state [/root/myfile-renotest.txt] at time 14:23:24.679833
      local:
      ----------
      	  ID: /root/myfile-renotest.txt
      Function: file.managed
        Result: True
       Comment: File /root/myfile-renotest.txt is in the correct state
       Changes:   
    
      Summary
      ------------
      Succeeded: 1
      Failed:    0
      ------------
      Total:     1
    

可以看到无论是Master推送还是Minion来拉去都是ok的

如果没有及时切换dns会导致minion可能拉取到fail状态的master, 从而auth失败:

	salt-call state.highstate    
	[WARNING ] Attempted to authenticate with master salt.dbfree.tbsite.net and failed
	[WARNING ] Attempted to authenticate with master 10.194.232.8 and failed
	local:
	    Data failed to compile:
	----------
	    The function "state.highstate" is running as PID 32237 and was started at 2014, Sep 03 15:38:11.202456 with jid 20140903153811202456

切换dns后minion的拉去是没问题的; 状态ok的master推送也是正常的, 但是可能return log会有多个, 影响日志入库后校验结果.


先整理到这里吧! 先撤, 回家休息..

最近白天事情多, 晚上还要机房演练, 感觉有点累, i need short rest -😃

posted @ 2015-08-12 20:20  Renolei  阅读(1318)  评论(0编辑  收藏  举报