elk部署
https://www.cnblogs.com/jsonhc/p/7562412.html
参考链接: https://www.elastic.co/guide/en/kibana/current/rpm.html
参考2
https://www.cnblogs.com/cheesebar/p/9126171.html
导入GPG key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
yum update curl#如果报错执行一下这个
新建文件: /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
安装
安装目录: /usr/share/elasticsearch
yum install elasticsearch
配置自动启动
chkconfig --add elasticsearch
创建数据目录
/data/elasticsearch/lib
/data/elasticsearch/log
chmod 777 -R /data/elasticsearch
修改配置文件/etc/elasticsearch/elasticsearch.yml
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch/lib
#
# Path to log files:
#
path.logs: /data/elasticsearch/log
修改配置文件/etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
修改配置文件/etc/elasticsearch/jvm.options
-Xms8g
-Xmx8g
内存设置成8G
启动
service elasticsearch start
如果启动报错
tail -F /var/log/messages
which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
解决办法：
vi /etc/sysconfig/elasticsearch
JAVA_HOME=/usr/local/jdk1.8.0_211
service elasticsearch restart
验证安装
curl -X GET "localhost:9200/"
端口配置
iptables -I INPUT -p tcp --dport 9200 -j ACCEPT
service iptables save
service iptables restart
Logstash
导入GPG key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
新建repo文件
vi /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
安装
yum install logstash
配置
参考链接: https://www.cnblogs.com/jsonhc/p/7562412.html
mkdir -p /usr/share/logstash/config/
ln -s /etc/logstash/* /usr/share/logstash/config
131.2 tomcat日志读取并发送到ES
创建配置文件
vi nohup ./bin/logstash -f ./config/conf.d/tomcat-log.conf
文件内容:
input{
file {
path => ["/usr/local/tomcat/cast-robot-server/logs/*.log"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
file {
path => ["/usr/local/tomcat/push-server/logs/*.out"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
file {
path => ["/usr/local/tomcat/apache-tomcat-server/logs/*.out"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
file {
path => ["/usr/local/tomcat/apache-tomcat-8.5,30/logs/*.out"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
}
filter {
grok {
match => {
"message" => "(?<year>[\d]{4})([-]{1})(?<month>[\d]{2})([-]{1})(?<day>[\d]{2})([\ ]{1})(?<hour>[\d]{2})([:]{1})(?<minute>[\d]{2})([:]{1})(?<second>[\d]{2})([,]{1})(?<ms>[\d]{3})([\ ]{1,})(?<thread>[\s\S]*[\]]{1})([\ ]{1,})(?<level>[\S]{1,6})(?<logger_name>[ ]{1,}[\S]{1,}[\ ]{1,})(?<line_number>[\d]{1,})(?<message>[\s\S]*)"
}
overwrite => ["message"]
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
创建启动脚本
vi /usr/share/logstash/tomcat-log-to-es.sh
文件内容:
nohup ./bin/logstash -f ./config/conf.d/tomcat-log.conf &
sh /usr/share/logstash/bin/logstash -f /root/file.conf &
131.7 tomcat日志读取并发送到ES
创建配置文件
vi nohup ./bin/logstash -f ./config/conf.d/tomcat-log.conf
文件内容:
input{
file {
path => ["/data/logs/app-interface/*.log"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
file {
path => ["/data/logs/app-interface-01/*.log"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
}
filter {
grok {
match => {
"message" => "(?<year>[\d]{4})([-]{1})(?<month>[\d]{2})([-]{1})(?<day>[\d]{2})([\ ]{1})(?<hour>[\d]{2})([:]{1})(?<minute>[\d]{2})([:]{1})(?<second>[\d]{2})([,]{1})(?<ms>[\d]{3})([\ ]{1,})(?<thread>[\s\S]*[\]]{1})([\ ]{1,})(?<level>[\S]{1,6})(?<logger_name>[ ]{1,}[\S]{1,}[\ ]{1,})(?<line_number>[\d]{1,})(?<message>[\s\S]*)"
}
overwrite => ["message"]
}
}
output {
elasticsearch {
hosts => ["104.243.131.2:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
创建启动脚本
vi /usr/share/logstash/tomcat-log-to-es.sh
文件内容:
nohup ./bin/logstash -f ./config/conf.d/tomcat-log.conf &
131.8 tomcat日志读取并发送到ES
创建配置文件
vi nohup ./bin/logstash -f ./config/conf.d/tomcat-log.conf
文件内容:
input{
file {
path => ["/data/tomcat/8081/*.log"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
file {
path => ["/data/tomcat/8080/*.log"]
start_position => "end"
codec => multiline {
pattern => "^[\d]{4}[-]{1}[\d]{2}[-]{1}[\d]{2} "
negate => true
what => "previous"
}
}
}
filter {
grok {
match => {
"message" => "(?<year>[\d]{4})([-]{1})(?<month>[\d]{2})([-]{1})(?<day>[\d]{2})([\ ]{1})(?<hour>[\d]{2})([:]{1})(?<minute>[\d]{2})([:]{1})(?<second>[\d]{2})([,]{1})(?<ms>[\d]{3})([\ ]{1,})(?<thread>[\s\S]*[\]]{1})([\ ]{1,})(?<level>[\S]{1,6})(?<logger_name>[ ]{1,}[\S]{1,}[\ ]{1,})(?<line_number>[\d]{1,})(?<message>[\s\S]*)"
}
overwrite => ["message"]
}
}
output {
elasticsearch {
hosts => ["104.243.131.2:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
创建启动脚本
vi /usr/share/logstash/tomcat-log-to-es.sh
文件内容:
nohup ./bin/logstash -f ./config/conf.d/tomcat-log.conf &
sh /usr/share/logstash/bin/logstash -f /root/file.conf &
安装Kibana
参考链接: https://www.elastic.co/guide/en/kibana/current/rpm.html
导入GPG Key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
新建repo文件: /etc/yum.repos.d/kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
安装
yum install kibana
配置: /etc/kibana/kibana.yml
server.host: 0.0.0.0
server.port: 56001
Nginx配置
server {
listen 443;
server_name facecast-back-stage-api.facecast.xyz;
location /kibana-platform {
proxy_pass http://localhost:56001/;
}
location /app {
proxy_pass http://localhost:56001/app;
}
location /ui {
proxy_pass http://localhost:56001/ui;
}
location /bundles {
proxy_pass http://localhost:56001/bundles;
}
location /api {
proxy_pass http://localhost:56001/api;
}
location /plugins {
proxy_pass http://localhost:56001/plugins;
}
location /elasticsearch {
proxy_pass http://localhost:56001/elasticsearch;
}
}
启动
nohup sh /usr/share/kibana/bin/kibana &
sh /usr/share/kibana/bin/kibana &
浙公网安备 33010602011771号