Logstash安装及传输日志

Logstash安装

• 按照官方文档安装

启动logstash

方法1

• 启动命令

systemctl start logstash

方法2

查看logstash启动配置

• 先执行命令查看启动文件位置，可看出启动文件位置为/etc/systemd/system/logstash.service

systemctl status logstash

• 查看/etc/systemd/system/logstash.service文件内容，可知使用的logstash位于/usr/share/logstash/bin/logstash, 配置文件所在的目录路径为/etc/logstash

• 最终启动命令

/usr/share/logstash/bin/logstash --path.settings /etc/logstash

测试日志经Filebeat传输到Logstash

新增配置文件

• 新增配置文件first-pipeline.conf

input {
    beats {
        port => "5044"
    }
}
output {
    stdout { codec => rubydebug }
}

指定配置文件启动

• 先启动Filebeat

./filebeat -e -c filebeat.yml -d "publish"

• 再启动Logstash

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/first-pipeline.conf

结果显示

Filebeat打印日志

Logstash打印日志

优化filter配置

修改配置文件first-pipeline.conf

• 增加filter,先将字符串转为json
• 去掉多余的字段，减少传输字节数

input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
    json {
        source => "message"
        target => "doc"
    }
    mutate {
        remove_field => [ "agent", "message", "host" ]
    }
}
output {
#    stdout { codec => json_lines }
    elasticsearch {
        hosts => ["192.168.1.19:9200", "192.168.1.19:9201"]
        index => "nginx-%{+YYYY.MM.dd}"
    }
}

• Logstash控制台输出

测试输出Elasticsearch

• 修改配置文件first-pipeline.conf

input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
    json {
        source => "message"
        target => "doc"
    }
    mutate {
        remove_field => [ "agent", "message", "host" ]
    }
}
output {
#    stdout { codec => json_lines }
    elasticsearch {
        hosts => ["192.168.1.19:9200", "192.168.1.19:9201"]
        index => "nginx-%{+YYYY.MM.dd}"
    }
}

• 查看Elasticsearch-head

FAQ

1.配置完成后，执行如下命令报错

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/first-pipeline.conf --config.test_and_exit

提示指定目录无权限，将目录所属权限原先为logstash用户，修改为启动用户

2.输出到控制台时，codec => json_pretty配置，提示插件不识别

检查插件安装情况，logstash-pluguin所在路径/usr/share/logstash/bin已配置到~/.bashrc文件的环境变量PATH中

logstash-plugin list | grep json

将output配置修改为

stdout { codec => json_lines }

参考

• Getting Started with Logstash
• Filter plugins