zabbix监控logstash日志

准备工作

1. zabbix:192.168.0.6
2. logstash:192.168.0.7

server1

安装zabbix

1. # yum -y install zabbix-server zabbix-web zabbix-server-mysql zabbix-web-mysql zabbix-agent zabbix-get mariadb mariadb-server httpd

登录mysql创建mysql

1. create database zabbix charset utf8;
2. grant all on zabbix.* to zabbix@'localhost' identified by 'zabbix';
3. flush privileges;

初始化zabbix数据库

1. zcat /usr/share/doc/zabbix-server-mysql-4.2.8/create.sql.gz | mysql -uzabbix -pzabbix zabbix

上传simkai.ttf字体到/usr/share/zabbix/assets/fonts/并修改字体

1. cd /usr/share/zabbix/assets/fonts/
2. mv simkai.ttf graphfont.ttf

修改时区

1. # vim /etc/php.ini
2. date.timezone = Asia/Shanghai # 878行

开启zabbix

1. # systemctl start zabbix-server zabbix-agent httpd

访问zabbix

浏览器打开192.168.0.6/zabbix

-> 添加主机(192.168.0.7)

-> 创建模板

-> 创建应用集

-> 创建监控项(选择采集器,归属到刚才创建的应用集)

-> 监测

-> 最新数据(主机选择l92.168.0.7)

-> 历史记录

server2

安装zabbix-agent

1. # yum -y install zabbix-agent

修改zabbix-agent配置

1. # vim /etc/zabbix/zabbix_agent.conf

1. Server=192.168.0.6 # 98行 zabbix-server ip
2. ServerActive=192.168.0.6 # 139行 zabbix-server ip

开启zabbix-agent

1. # systemctl start zabbix-agent

zabbix-server主机测试zabbix-agent是否正常

1. zabbix-get -s 192.168.0.7 -k agent.ping

1 # 测试成功

安装jdk

1. rpm -ivh jdk-8u131-linux-x64_.rpm

安装logstash

1. # unzip logstash-7.9.2.zip
2. # mv logstash-7.9.2 /usr/local/logstash

安装插件

1. # yum -y install wget unzip
2. # /usr/local/logstash/bin/logstash-plugin install logstash-output-zabbix
3. # /usr/local/logstash/bin/logstash-plugin install logstash-integration-jdbc
4. # wget https://github.com/logstash-plugins/logstash-input-beats/archive/master.zip -O /usr/local/logstash/master.zip
5. # unzip /usr/local/logstash/master.zip

配置logstash

1. # vim /usr/local/logstash/config/zabbix.conf

1. input {
2. file {
3. path => ["/var/log/secure"] # 文件位置
4. type => "system" # 类型
5. add_field => [ "[zabbix_key]", "oslogs" ] # zabbix键值
6. add_field => [ "[zabbix_host]", "192.168.0.7" ] # logstash主机
7. start_position => "beginning"
8. }
9. }
10. output{
11. stdout { codec => rubydebug } # 调试信息
12. zabbix {
13. zabbix_host => "[zabbix_host]"
14. zabbix_key => "[zabbix_key]"
15. zabbix_server_host => "192.168.0.6" # zabbix-server ip
16. zabbix_server_port => "10051" # zabbix-server port
17. zabbix_value => "message" # 日志字段
18. }
19. }

测试:

1. /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/zabbix.conf

开启成功提示下方信息

1. Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
2. [2020-10-08T20:37:47,334][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +indy +jit [linux-x86_64]"}
3. [2020-10-08T20:37:47,923][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
4. [2020-10-08T20:37:50,204][INFO ][org.reflections.Reflections] Reflections took 42 ms to scan 1 urls, producing 22 keys and 45 values
5. [2020-10-08T20:37:51,436][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/zabbix.conf"], :thread=>"#<Thread:0x2ef7b133 run>"}
6. [2020-10-08T20:37:52,520][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.06}
7. [2020-10-08T20:37:52,766][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/local/logstash/data/plugins/inputs/file/.sincedb_730aea1d074d4636ec2eacfacc10f882", :path=>["/var/log/secure"]}
8. [2020-10-08T20:37:52,830][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
9. [2020-10-08T20:37:52,921][INFO ][filewatch.observingtail ][main][5ffcc74b3b6be0e4daa892ae39a07dc20fdbc1d05bd5cedc4b4290930274f61e] START, creating Discoverer, Watch with file and sincedb collections
10. [2020-10-08T20:37:52,963][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
11. [2020-10-08T20:37:53,369][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

使用ssh命令连接logstash主机,故意输错密码查看输出

1. {
2. "type" => "system",
3. "path" => "/var/log/secure",
4. "host" => "localhost.localdomain",
5. "message" => "Oct 8 20:43:31 localhost sshd[112359]: Accepted passwh2",
6. "zabbix_key" => "oslogs",
7. "@timestamp" => 2020-10-09T00:43:56.940Z,
8. "zabbix_host" => "192.168.0.7",
9. "@version" => "1"
10. }
11. {
12. "type" => "system",
13. "path" => "/var/log/secure",
14. "host" => "localhost.localdomain",
15. "message" => "Oct 8 20:43:31 localhost sshd[112359]: pam_unix(sshd:(uid=0)",
16. "zabbix_key" => "oslogs",
17. "@timestamp" => 2020-10-09T00:43:56.966Z,
18. "zabbix_host" => "192.168.0.7",
19. "@version" => "1"
20. }
21. {
22. "type" => "system",
23. "path" => "/var/log/secure",
24. "host" => "localhost.localdomain",
25. "message" => "Oct 8 20:43:36 localhost sshd[112359]: pam_systemd(ssterrupted system call",
26. "zabbix_key" => "oslogs",
27. "@timestamp" => 2020-10-09T00:43:56.967Z,
28. "zabbix_host" => "192.168.0.7",
29. "@version" => "1"
30. }
31. ...

正常输出,CTRL+C退出

如果日志中存在报错:

[2020-10-08T09:06:42,311][WARN ][logstash.outputs.zabbix ][main][630c433ba0be0739e8ee72ca91d03f00695f05873b64e12c7488b8f2c32a8e05] Zabbix server at 192.168.0.6 rejected all items sent. {:zabbix_host=>"192.168.0.6"}

表示zabbix主机拒绝了所有的连接

解决方法:

设置防火墙规则

查看配置文件zabbix地址与logstash地址

[2020-10-08T20:41:45,154][WARN ][logstash.outputs.zabbix ][main][cf6b448e829beca8b4ffbd64e71c6e510108015eec5933f7b4675d79d5f09f03] Field referenced by 192.168.0.7 is missing

缺少引用的字段

解决方法:

添加带有message字段的信息到secure日志中

展示:

关闭输出并后台运行

1. /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/zabbix.conf &> /dev/null &

生成日志信息并使用zabbix查看

-> 监测

-> 最新数据

-> 历史记录