28.9买的 ELK（V6.6.2）
1.简介
ELK日志收集分析系统，由Elasticsearch和Logstash及Kibana组成。
Elasticsearch负责存储数据和分析数据。
Logstash负责数据采集以及将数据发送到Elasticsearch。优点：功能强大。支持过滤。缺点：需要JDK环境，占用系统资源多。
Kibana由于数据的web展示。
Filebeat轻量级的数据采集工具，将采集的数据发送至Elatsticsearch或Logstash。优点：轻量级，无需安装java,占用系统资源少。缺点：功能单一，不支持过滤。
Redis用于数据存储，Logstash发生故障时，保证filebeat传输的数据不会丢失。
2.原理图
3.安装Elasticsearch
3.1 安装java环境。
yum install java-1.8.0-openjdk-devel -y3.2 获取elasticsearch安装包，安装。
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.2.tar.gz tar zxf elasticsearch-6.6.2.tar.gz -C /application/3.3 配置。
egrep -v "^$|^#" /application/elasticsearch-6.6.2/config/elasticsearch.yml cluster.name: Mycluster node.name: node1 path.data: /data/data path.logs: /data/logs network.host: 0.0.0.0 http.port: 9200 egrep "Xms|Xmx" /application/elasticsearch-6.6.2/config/jvm.options -Xms1g -Xmx1g3.4 创建运行elasticsear的用户。elasticsearch不允许使用root用户运行。
useradd -u 9200 elk mkdir /data/{data,logs} -p chown -R elk:elk /application/elasticsearch-6.6.2 chown -R elk:elk /data/*3.5 切换到elk用户，运行elasticsearch。
su - elk /application/elasticsearch-6.6.2/bin/elasticsearch -d # -d为后台运行3.6 报错解决
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] 解决1： vim /etc/security/limits.conf * soft nofile 65536 * hard nofile 65536 查看生效 ulimit -Hn ulimit -Sn 解决2: vim /etc/sysctl.conf vm.max_map_count=262144 查看生效 sysctl -p3.7 测试,使用浏览器打开。
http://172.168.1.61:9200
4.安装Kibana
4.1 获取安装包，安装。
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.2-linux-x86_64.tar.gz tar zxf kibana-6.6.2-linux-x86_64.tar.gz -C /application/4.2 配置
vim /application/kibana-6.6.2-linux-x86_64/config/kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.hosts: ["http://localhost:9200"]4.3 启动
/application/kibana-6.6.2-linux-x86_64/bin/kibana &4.4 访问
http://172.168.1.61:56014.5 概念
index索引 => 数据库databases tyep类型 => 表table Document文档 => 行row Field字段 => 列column4.6 开发工具
创建index索引。类似创建数据库。 put /yy 查看索引。类似查看库信息。 GET /yy/_settings 存入文档，类似于存入一行数据。 POST /yy/student/1 { "name":"zhangsan", "age":18, "email":"zhangsan@qq.com" } 获取索引，类似获取库信息。 GET /yy 获取个字段。 GET /yy/student/1?_source=age 获取一行的所有字段。 GET /yy/student/1/_source 查看响应头部，类似于http请求状态。 HEAD /yy/student/1 删除文档，类似于删除一天纪录。 DELETE /yy/student/1
5.安装Logstash
5.1.获取安装包，安装。
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.2.tar.gz tar zxf logstash-6.6.2.tar.gz -C /application/5.2 配置。
vim /application/logstash-6.6.2/config/jvm.options -Xms512m -Xmx512m5.3 测试。
/application/logstash-6.6.2/bin/logstash -e "input{ stdin{} } output{ stdout {}}"5.4 脚本方式。
vim config/logstash.conf input{ file{ path => "/var/log/maessages" type => "system" start_position => "beginning" } } filter { } output{ stdout{ } }5.5 运行
./bin/logstash -f config/logstash.conf5.6 将logstash收集的数据，发送至elasticsearch。
vim logstash-es.conf input{ file{ path => "/var/log/messages" type => "system" start_position => "beginning" } } filter{ } output{ elasticsearch{ hosts => ["172.168.1.61:9200"] index => "msg-%{+YYYY.MM.dd}" } } 运行 ./bin/logstash -f config/logstash-es.conf5.7 监控nginx日志。
cat /application/logstash-6.6.2/config/logstash-nginx.conf input{ file{ path => "/var/log/nginx/access.log" type => "nginx-access" start_position => "beginning" } } filter { grok{ match => { "massage" => "%{HTTPD_COMBINEDLOG}" } } } output{ elasticsearch{ hosts => ["127.0.0.1:9200"] index => "nginx-%{+YYYY.MM.dd}" } } 启动 /application/logstash-6.6.2/bin/logstash -f /application/logstash-6.6.2/config/logstash-nginx.conf
6.安装Filebeat
6.1 获取安装包，安装。
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.2-linux-x86_64.tar.gz tar zxf filebeat-6.6.2-linux-x86_64.tar.gz -C /application/6.2 配置
egrep -v "#|^$" filebeat.yml filebeat.inputs: - type: log enabled: true #开启模块 backoff: "1s" #检查文件更新一秒 tail_files: false #从头开始读 paths: - /var/log/nginx/access.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: output.elasticsearch: hosts: ["localhost:9200"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~6.3 启动
./filebeat -e -c filebeat.yml /applicatin/filbeat-6.6.2-linux-x86_64/data/* #此文件纪录上一次fliebeat读取日志文件的位置。
7.集群（filebeat+logstaah+elaasticsearch+kibana）
7.1 fliebeat相关配置。
cat fields.yml filebeat.inputs: - type: log enabled: true backoff: "1s" tail_files: false paths: - /var/log/nginx/access.log output.logstash: enabled: true hosts: ["localhost:5044"]7.2 logstash相关配置。
cat logstash-filebeat.conf input{ beats { host => "0.0.0.0" port => 5044 } } filter { grok{ match => { "massage" => "%{HTTPD_COMBINEDLOG}" } } date{ match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target => "@timestamp" } } output{ elasticsearch{ hosts => ["127.0.0.1:9200"] index => "nginx-%{+YYYY.MM.dd}" } } 去掉不需要收集的字段。在grok配置项中配置。 remove_filed => ["beat","offset","tags","prospector"]7.3 启动
/application/logstash-6.6.2/bin/logstash -f /application/logstash-6.6.2/config/logstash-filebeat.conf & /application/filebeat-6.6.2-linux-x86_64/filebeat -e -c /application/filebeat-6.6.2-linux-x86_64/fields.yml
8.json数据收集
8.1 配置nginx的日志格式。
log_format log_json '{"remote_adder":"$remote_addr",' '"ident":"-",' '"user":"remote_user",' '"timestamp":"$time_local",' '"request":"$request",' '"status":"$status",' '"bytes":"body_bytes_sent",' '"referer":"$http_referer",' '"agent":"$http_user_agent",' '"x_forwarded":"$http_x_forwarded_for"' '}'; access_log /var/log/nginx/access_json.log log_json;8.2 filebeat配置
cat fields.yml filebeat.inputs: - type: log enabled: true backoff: "1s" tail_files: false paths: - /var/log/nginx/access_json.log output.logstash: enabled: true hosts: ["localhost:5044"]
8.3 logstash配置。
cat logstash-json.conf input{ beats { host => "0.0.0.0" port => 5044 } } filter { json{ source => "message" remove_field => ["beat","offset","tags","prospector"] } date{ match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target => "@timestamp" } } output{ elasticsearch{ hosts => ["127.0.0.1:9200"] index => "nginx-%{+YYYY.MM.dd}" } }8.4 启动
/application/logstash-6.6.2/bin/logstash -f /application/logstash-6.6.2/config/logstash-json.conf /application/filebeat-6.6.2-linux-x86_64/filebeat -e -c /application/filebeat-6.6.2-linux-x86_64/filebeat_json.yml
9.filebeat收集多个日志
9.1 配置filebeat
cat /application/filebeat-6.6.2-linux-x86_64/filebeats.yml filebeat.inputs: - type: log enabled: true backoff: "1s" tail_files: false paths: - /var/log/nginx/access_json.log fields: filetype: logjson fields_under_root: true - type: log enabled: true backoff: "1s" tail_files: false paths: - /var/log/messages fields: filetype: logsystem fields_under_root: true output.logstash: enabled: true hosts: ["localhost:5044"]
9.2 logstash配置
cat /application/logstash-6.6.2/config/logstash-files.conf input { beats { host => "0.0.0.0" port => 5044 } } filter { if [filetype] == "logjson" { json { source => "message" remove_field => ["beat","offset","tags","prospector"] } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target => "@timestamp" } } } output { if [filetype] == "logjson" { elasticsearch { hosts => ["127.0.0.1:9200"] index => "nginx-%{+YYYY.MM.dd}" } } else if [filetype] == "logsystem" { elasticsearch { hosts => ["127.0.0.1:9200"] index => "msg-%{+YYYY.MM.dd}" } } }9.3 启动
/application/logstash-6.6.2/bin/logstash -f /application/logstash-6.6.2/config/logstash-files.conf /application/filebeat-6.6.2-linux-x86_64/filebeat -e -c /application/filebeat-6.6.2-linux-x86_64/filebeats.yml
10 增加Redis（filebeat+redis+logstash）
10.1 启动Redis
/application/redis-4/bin/redis-server /application/redis-4.0.14/redis.conf10.2 filbeat配置
cat /application/filebeat-6.6.2-linux-x86_64/filebeat-redis.yml filebeat.inputs: - type: log enabled: true backoff: "1s" tail_files: false paths: - /var/log/nginx/access_json.log fields: filetype: nginxjson fields_under_root: true output.redis: enabled: true hosts: ["127.0.0.1:6379"] port: 6379 key: nginx db: 0 datatype: list10.3 logstash配置
cat /application/logstash-6.6.2/config/logstash-redis.conf input{ redis { host => "127.0.0.1" port => 6379 key => "nginx" data_type => "list" db => 0 } } filter { json{ source => "message" remove_field => ["beat","offset","tags","prospector"] } date{ match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target => "@timestamp" } } output{ elasticsearch{ hosts => ["127.0.0.1:9200"] index => "nginx-%{+YYYY.MM.dd}" } }10.4 启动
/application/filebeat-6.6.2-linux-x86_64/filebeat -e -c /application/filebeat-6.6.2-linux-x86_64/filebeat-redis.yml /application/logstash-6.6.2/bin/logstash -f /application/logstash-6.6.2/config/logstash-redis.conf /application/redis-4/bin/redis-server /application/redis-4.0.14/redis.conf
11.限制访问
11.1 生成密码加密串。
openssl passwd -apr1 12345611.2 配置用户名密码。
cat /etc/nginx/passwd yy:$apr1$b749/TXa$fNk596y9Tg2yFtzsuvsxU.11.3 配置nginx
cat /etc/nginx/nginx.conf auth_basic "kibana auth"; auth_basic_user_file /etc/nginx/passwd; proxy_pass http://127.0.0.1:5601;11.4 nginx重新加载配置文件。
nginx -s reload
12.Elasticsearch集群
12.1 环境
node1 172.168.1.61 node2 172.168.1.6212.2 node1配置
grep -v "#" config/elasticsearch.yml cluster.name: fuerda node.name: node1 path.data: /data/data path.logs: /data/logs network.host: 0.0.0.0 http.port: 9200 transport.tcp.port: 9300 #单薄 node.master: true #在集群中可以成为主节点 node.data: true #在集群中可以存储数据 discovery.zen.ping.unicast.hosts: ["172.168.1.61:9300", "172.168.1.62:9300"] discovery.zen.minimum_master_nodes: 2 #投票机制。节点数除以2加112.3 node2配置
grep -v "#" elasticsearch.yml cluster.name: fuerda node.name: node2 path.data: /data/data path.logs: /data/logs network.host: 0.0.0.0 http.port: 9200 transport.tcp.port: 9300 node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["172.168.1.61:9300", "172.168.1.62:9300"] discovery.zen.minimum_master_nodes: 213.4 启动
su - elk /application/elasticsearch-6.6.2/bin/elasticsearch13.5 查看集群状态
http://172.168.1.61:9200/_cluster/state?pretty=true13.6 配置kibana
grep -v "#" /application/kibana-6.6.2-linux-x86_64/config/kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.hosts: ["http://172.168.1.61:9200","http://172.168.1.62:9200"]
14.IK
14.1 切换用户
su - elk14.2.创建ik目录，进入目录，下载安装包并解压。
mkdir /application/elasticsearch-6.6.2/plugins/ik cd /application/elasticsearch-6.6.2/plugins/ik wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.6.2/elasticsearch-analysis-ik-6.6.2.zip unzip elasticsearch-analysis-ik-6.6.2.zip14.3 重启Elasticsearch
/application/redis-4/bin/redis-server /application/redis-4.0.14/redis.conf
浙公网安备 33010602011771号