vector日志采集与接收

vector安装：

#在/usr/local 目录下解压vector_0.24.1.tar.gz
cd /usr/local
tar -zxf vector_0.24.1.tar.gz

#创建目录 /var/lib/vector/ ，用于存储日志采集变化配置文件
mkdir -p /var/lib/vector/

#创建目录 /etc/vector ，用于放置vector配置文件
mkdir /etc/vector
cp /usr/local/vector/config/vector.toml /etc/vector/

ln -s /usr/local/vector/bin/vector /usr/bin/vector
cp /usr/local/vector/etc/systemd/vector.service  /usr/lib/systemd/system/

systemctl daemon-reload
systemctl enable vector
systemctl start vector
systemctl status  vector -l

注：/data/log/下的haproxy日志，只有root账户能读取，需要修改vim /usr/lib/systemd/system/vector.service文件，修改服务运行账户

ExecStart=/usr/bin/vector "--config-dir" "/etc/vector"
User=root
Group=root

 

采集客户端日志，并将其发送到远端服务器（通过vector接收）

客户端/etc/vector/vector.toml配置文件如下：

[sources.haproxy_http]  #读取特定目录下的haproxy http.log
type = "file"
include = ["/data/log/haproxy_http.log"]
ignore_older = 86400

[sources.haproxy_tcp] #读取特定目录下的haproxy tcp.log，vector运行账户必须要对所读取的日志具有读取权限
type = "file"
include = ["/data/log/haproxy_tcp.log"]
ignore_older = 86400

[transforms.add_host_http] #对日志进行转换，添加本地计算机名称
type = "remap"
inputs = ["haproxy_http"]
source = '''
.host = "haserver01"   #自定义本地计算机名称
.log_type = "http"
'''

[transforms.add_host_tcp]
type = "remap"
inputs = ["haproxy_tcp"]
source = '''
.host = "haserver02"
.log_type = "tcp"
'''

[sinks.to_vecserver]  #将日志发送到远端vector服务器
type = "vector"
inputs = ["add_host_http", "add_host_tcp"]
address = "123.123.123.123:9000"

 

服务端/etc/vector/vector.toml配置文件如下：

[sources.vec_servers]
type = "vector"
address = "0.0.0.0:9000" #开放9000端口用于接收客户端日志

[transforms.route_logs]  #配置路由，将不同的日志存储到不同的目录下
type = "route"
inputs = ["vec_servers"]
route.http = '.log_type == "http"'
route.tcp = '.log_type == "tcp"'

[sinks.http_logs] #保存haproxy http日志到特定目录下，并添加客户端计算机名称
type = "file"
inputs = ["route_logs.http"]
path = "/data/log/vector/haproxy/http/{{host}}_haproxy_http-%Y-%m-%d.log"  #vector运行账号需要对该目录（/data/log/vector/haproxy）具有读写权限，否则无法保存日志到本地
encoding.codec = "text"                                                                                                                                                                                                                    

[sinks.tcp_logs] #保存haproxy tcp日志到特定目录下，并添加客户端计算机名称
type = "file"
inputs = ["route_logs.tcp"]
path = "/data/log/vector/haproxy/tcp/{{host}}_haproxy_tcp-%Y-%m-%d.log"
encoding.codec = "text"

 

如果采集不到日志，使用以下命令查看vector是否监听到了日志变化：

cat /var/lib/vector/haproxy_http/checkpoints.json | jq .

 

如果使用vector账户运行vector服务，则要求如下：

groupadd vector
useradd vector -g vector -r -s /sbin/nologin
#vector账号需要对/var/lib/vector具有读写权限
chown -R vector:vector /var/lib/vector/