Burrow 服务的安装部署

　　　　　　　　　　　　　　　　　　Burrow 服务的安装部署

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

　　最近协助开发的同时帮忙把10个topic的数据使用5个topic的来工作。结果发现数据flume在手机数据时存在延迟消费的问题。因此遇到了不少坑，要是有个专门监控kafka consumer lag的开源软件就好了！当然我们通过kafka-consumer-groups.sh这个命令也可以查看到相应的参数，但每次都需要敲击命令行。无意间听网友说有LinkedIn公司开源有一款叫做Burrow都软件，专门是对kafka消费这延迟检查。

 

一.Burrow概述

1>.什么是Burrow

　　Burrow是Apache Kafka的监控伴侣，它将消费者滞后检查作为服务提供，无需指定阈值。它监控所有消费者的承诺抵消额，并根据需要计算这些消费者的状态。提供HTTP端点以按需请求状态，以及提供其他Kafka群集信息。还有可配置的通知程序可以通过电子邮件或HTTP调用将状态发送到另一个服务。

2>.Burrow特征 

没有门槛！通过滑动窗口评估组。

多个Kafka群集支持

使用Kafka提交的偏移自动监视所有使用者

可配置支持Zookeeper提交的偏移量

可配置支持Storm-committed offsets

消费者组状态的HTTP端点，以及代理和消费者信息

可配置的电子邮件程序，用于发送特定组的警报

可配置的HTTP客户端，用于向所有组的另一个系统发送警报

3>.先决条件

　　Burrow是用Go编写的，所以在开始之前，你应该安装并设置Go。

  

二.安装Golang环境

1>.下载 go 的安装包 

[root@node101.yinzhengjie.org.cn ~/Downloads]# wget https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz
--2019-03-28 16:09:35--  https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz
Resolving dl.google.com (dl.google.com)... 203.208.41.70, 203.208.41.72, 203.208.41.65, ...
Connecting to dl.google.com (dl.google.com)|203.208.41.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 127906702 (122M) [application/octet-stream]
Saving to: ‘go1.12.1.linux-amd64.tar.gz’

100%[=========================================================================================================================================================================>] 127,906,702 2.27MB/s   in 54s    

2019-03-28 16:10:29 (2.28 MB/s) - ‘go1.12.1.linux-amd64.tar.gz’ saved [127906702/127906702]

[root@node101.yinzhengjie.org.cn ~/Downloads]# 

2>.解压go的安装包到指定目录

[root@node101.yinzhengjie.org.cn ~/Downloads]# tar -zxf go1.12.1.linux-amd64.tar.gz -C /yinzhengjie/softwares/
[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# ll /yinzhengjie/softwares/go/
total 208
drwxr-xr-x.  2 root root  4096 Mar 15 03:43 api
-rw-r--r--.  1 root root 55358 Mar 15 03:43 AUTHORS
drwxr-xr-x.  2 root root  4096 Mar 15 03:46 bin
-rw-r--r--.  1 root root  1339 Mar 15 03:43 CONTRIBUTING.md
-rw-r--r--.  1 root root 78132 Mar 15 03:43 CONTRIBUTORS
drwxr-xr-x.  8 root root  4096 Mar 15 03:43 doc
-rw-r--r--.  1 root root  5686 Mar 15 03:43 favicon.ico
drwxr-xr-x.  3 root root  4096 Mar 15 03:43 lib
-rw-r--r--.  1 root root  1479 Mar 15 03:43 LICENSE
drwxr-xr-x. 13 root root  4096 Mar 15 03:43 misc
-rw-r--r--.  1 root root  1303 Mar 15 03:43 PATENTS
drwxr-xr-x.  6 root root  4096 Mar 15 03:46 pkg
-rw-r--r--.  1 root root  1607 Mar 15 03:43 README.md
-rw-r--r--.  1 root root    26 Mar 15 03:43 robots.txt
drwxr-xr-x. 46 root root  4096 Mar 15 03:43 src
drwxr-xr-x. 21 root root 12288 Mar 15 03:43 test
-rw-r--r--.  1 root root     8 Mar 15 03:43 VERSION
[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# 

3>.配置go到环境变量 

[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# mkdir /yinzhengjie/goproject
[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# tail -5 /etc/profile
#ADD GO_PATH
export GO_HOME=/yinzhengjie/softwares/go
export PATH=$PATH:$GO_HOME/bin
export GOPATH=/yinzhengjie/goproject

[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# 

[root@node101.yinzhengjie.org.cn ~/Downloads]# source /etc/profile 
[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/yinzhengjie/goproject"
GOPROXY=""
GORACE=""
GOROOT="/yinzhengjie/softwares/go"
GOTMPDIR=""
GOTOOLDIR="/yinzhengjie/softwares/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build685793832=/tmp/go-build -gno-record-gcc-switches"
[root@node101.yinzhengjie.org.cn ~/Downloads]# 
[root@node101.yinzhengjie.org.cn ~/Downloads]# 

 

三.Burrow安装

1>.Burrow的github地址（https://github.com/linkedin/Burrow）

2>.安装git工具

[root@node101.yinzhengjie.org.cn ~]# yum -y install git
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.8.3.1-20.el7 will be installed
--> Processing Dependency: perl-Git = 1.8.3.1-20.el7 for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Git) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Error) for package: git-1.8.3.1-20.el7.x86_64
--> Running transaction check
---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed
---> Package perl-Git.noarch 0:1.8.3.1-20.el7 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                                 Arch                                          Version                                                Repository                                      Size
===================================================================================================================================================================================================================
Installing:
 git                                                     x86_64                                        1.8.3.1-20.el7                                         updates                                        4.4 M
Installing for dependencies:
 perl-Error                                              noarch                                        1:0.17020-2.el7                                        base                                            32 k
 perl-Git                                                noarch                                        1.8.3.1-20.el7                                         updates                                         55 k
 perl-TermReadKey                                        x86_64                                        2.30-20.el7                                            base                                            31 k

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 4.5 M
Installed size: 22 M
Downloading packages:
(1/4): perl-Error-0.17020-2.el7.noarch.rpm                                                                                                                                                  |  32 kB  00:00:00     
perl-TermReadKey-2.30-20.el7.x FAILED                                          
http://mirrors.cloud.aliyuncs.com/centos/7/os/x86_64/Packages/perl-TermReadKey-2.30-20.el7.x86_64.rpm: [Errno 14] curl#6 - "Could not resolve host: mirrors.cloud.aliyuncs.com; Unknown error"   0 B  --:--:-- ETA 
Trying other mirror.
(2/4): perl-TermReadKey-2.30-20.el7.x86_64.rpm                                                                                                                                              |  31 kB  00:00:00     
perl-Git-1.8.3.1-20.el7.noarch FAILED                                          
http://mirrors.cloud.aliyuncs.com/centos/7/updates/x86_64/Packages/perl-Git-1.8.3.1-20.el7.noarch.rpm: [Errno 14] curl#6 - "Could not resolve host: mirrors.cloud.aliyuncs.com; Unknown error"   0 B  --:--:-- ETA 
Trying other mirror.
(3/4): git-1.8.3.1-20.el7.x86_64.rpm                                                                                                                                                        | 4.4 MB  00:00:01     
(4/4): perl-Git-1.8.3.1-20.el7.noarch.rpm                                                                                                                                                   |  55 kB  00:00:01     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                              2.1 MB/s | 4.5 MB  00:00:02     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:perl-Error-0.17020-2.el7.noarch                                                                                                                                                               1/4 
  Installing : perl-TermReadKey-2.30-20.el7.x86_64                                                                                                                                                             2/4 
  Installing : git-1.8.3.1-20.el7.x86_64                                                                                                                                                                       3/4 
  Installing : perl-Git-1.8.3.1-20.el7.noarch                                                                                                                                                                  4/4 
  Verifying  : perl-Git-1.8.3.1-20.el7.noarch                                                                                                                                                                  1/4 
  Verifying  : 1:perl-Error-0.17020-2.el7.noarch                                                                                                                                                               2/4 
  Verifying  : git-1.8.3.1-20.el7.x86_64                                                                                                                                                                       3/4 
  Verifying  : perl-TermReadKey-2.30-20.el7.x86_64                                                                                                                                                             4/4 

Installed:
  git.x86_64 0:1.8.3.1-20.el7                                                                                                                                                                                      

Dependency Installed:
  perl-Error.noarch 1:0.17020-2.el7                                    perl-Git.noarch 0:1.8.3.1-20.el7                                    perl-TermReadKey.x86_64 0:2.30-20.el7                                   

Complete!
[root@node101.yinzhengjie.org.cn ~]# 

2>.

 

 

3>.

 

4>.

 

5>.