ELK介绍
1.什么是ELK
ELK是三个软件
1.E：elasticsearch java程序 存储，查询日志
2.L: logstash java程序 收集、过滤日志
3.K: kibana java程序 提供web服务，将数据页面化
4.F: filebeat go 收集、过滤日志
2.ELK作用
1.收集： 收集所有服务器的日志
2.传输： 把日志稳定的传输到ES或者其他地方
3.存储： ES能有效快速的存储日志数据
4.分析： 通过web页面分析数据
5.监控： 监控集群架构
3.ELK优点
1.处理方式灵活：elasticsearch是实时全文索引，具有强大的搜索功能
2.配置相对简单：elasticsearch全部使用JSON 接口，logstash使用模块配置，kibana的配置文件部分更简单。
3.检索性能高效：基于优秀的设计，虽然每次查询都是实时，但是也可以达到百亿级数据的查询秒级响应。
4.集群线性扩展：elasticsearch和logstash都可以灵活线性扩展
5.前端操作绚丽：kibana的前端设计比较绚丽，而且操作简单
4.为什么使用ELK
### ELK 集群服务版本必须一致
#收集所有的日志
web服务日志
业务服务日志
系统日志
#统计、分析：
1.统计访问量
2.统计访问量前10的IP
3.站点访问次数最多的URL
4.查询一上午以上三个值
5.查询一下午以上三个值
6.对比一下上下午用户访问量
7.对比这一周，每天用户增长还是减少
环境准备
实现功能
1. tomcat部署jpress项目
2. nginx代理jpress
3. 实时获取tomcat json格式日志
4. 实时获取nginx json格式日志
5. 实现 redis 消息队列
| 主机名 |
IP |
服务 |
环境要求 |
内存 |
| jojo01 |
10.0.0.20 |
tomcat nginx logstash MySQL Redis |
JDK |
4G |
| jojo02 |
10.0.0.21 |
es kibana |
JDK |
2G |
| jojo03 |
10.0.0.22 |
es |
JDK |
2G |
ES集群部署
1.时间同步
yum install -y ntpdate
ntpdate time1.aliyun.com
2.安装Java环境
rz jdk-8u181-linux-x64.rpm
rpm -ivh jdk-8u181-linux-x64.rpm
3.安装ES
rz elasticsearch-6.6.0.rpm
#下载地址：https://www.elastic.co/downloads/elasticsearch
rpm -ivh elasticsearch-6.6.0.rpm
# 根据提示继续操作
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
4.配置ES
# 配置文件
vim /etc/elasticsearch/elasticsearch.yml
# jojo02
cluster.name: escluster
node.name: es1
path.data: /service/es/data
path.logs: /service/es/logs
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
network.host: 10.0.0.21,127.0.0.1
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: [10.0.0.21","10.0.0.22"] # 集群ip
# jojo02
cluster.name: escluster
node.name: es1
path.data: /service/es/data
path.logs: /service/es/logs
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
network.host: 10.0.0.21,127.0.0.1
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: [10.0.0.21","10.0.0.22"] # 集群ip
# 创建数据目录
mkdir /service/es/{data,logs} -p
chown -R elasticsearch.elasticsearch /service/es/
5.配置启动文件中内存锁
vim /usr/lib/systemd/system/elasticsearch.service
[Service]
LimitMEMLOCK=infinity
5.启动ES
systemctl daemon-reload
systemctl start elasticsearch.service
Kibana 部署(jojo02)
1. 上传代码包
rz kibana-6.6.0-x86_64.rpm
rpm -ivh kibana-6.6.0-x86_64.rpm
2. 配置kibana
vim /etc/kibana/kibana.yml
#进程的端口
server.port: 5601
#监听地址
server.host: "10.0.0.21"
#指定ES的地址
elasticsearch.hosts: ["http://10.0.0.21:9200"]
#kibana也会创建索引
kibana.index: ".kibana"
3. 启动kibana
systemctl start kibana.service
4. 访问页面
http://10.0.0.20:5601
Tomcat 部署(jojo01)
1. 时间同步
yum install -y ntpdate
ntpdate time1.aliyun.com
2. 安装Java环境
rz jdk-8u181-linux-x64.rpm
rpm -ivh jdk-8u181-linux-x64.rpm
3.tomcat
yum install tomcat tomcat-webapps tomcat-admin-webapps -y
4.挂载wer包
# 下载地址
https://gitee.com/fuhai/jpress/blob/alpha/wars/jpress-web-newest.war
cd /usr/share/tomcat/webapps/ rz
4.启动
systemctl start tomcat
# 修改tomcat 默认页面
cp -rp /usr/share/tomcat/webapps/ /usr/share/tomcat/webapps-jpress
cd /usr/share/tomcat/webapps-jpress
rm -rf ROOT/*
mv jpress-v3.2.5/* ROOT/
# 修改默认地址及json格式
vim /usr/share/tomcat/conf/server.xml
<Service name="webapps-jpress">
<Engine name="webapps-jpress" resourceName="UserDatabase"/>
<Host name="localhost" appBase="webapps-jpress"
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat_access_json" suffix=".log"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
部署MySQL(jojo01)
1. 安装依赖
yum install -y ncurses-devel libaio-devel gcc gcc-c++ glibc cmake autoconf openssl openssl-devel
2.上传
rz
tar xf mysql-5.6.46-linux-glibc2.12-x86_64.tar.gz
3.创建目录
mkdir /service
4.软链接
mv mysql-5.6.46-linux-glibc2.12-x86_64 /service/
ln -s /service/mysql-5.6.46-linux-glibc2.12-x86_64 /service/mysql
5.创建用户
useradd mysql -s /sbin/nologin -M
6.拷贝配置文件和启动脚本
cd /service/mysql/support-files/
cp my-default.cnf /etc/my.cnf
cp: overwrite '/etc/my.cnf'? y
# 主库配置文件
vim /etc/my.cnf
[mysqld]
basedir = /service/mysql
datadir = /service/mysql/data
port=mysql
server_id=1
skip_name_resolve
log_err=/service/mysql/data/mysql.err
log_bin=/service/mysql/data/mysql-bin
cp mysql.server /etc/init.d/mysqld
7. 初始化
cd /service/mysql/scripts/
./mysql_install_db --user=mysql --basedir=/service/mysql --datadir=/service/mysql/data
8.system管理
vim /usr/lib/systemd/system/mysqld.service
[Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=https://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/service/mysql/bin/mysqld --defaults-file=/etc/my.cnf
LimitNOFILE = 5000
systemctl daemon-reload
systemctl start mysqld
9.添加环境变量
vim /etc/profile.d/mysql.sh
export PATH=/service/mysql/bin:$PATH
source /etc/profile
10.操作数据库
mysql
create database php;
grant all on php.* to 'lyw' identified by '123';
Nginx 部署(jojo01)
1. 更换官方源
vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
2. 安装nginx
yum install -y nginx
3.修改json格式日志
vim /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
access_log /var/log/nginx/access.log json;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
4. 代理配置
vim /etc/nginx/conf.d/wp.conf
server {
listen 80;
server_name jpress.com;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
5. 启动
systemctl start nginx
Logstash 部署(jojo01)
1. 时间同步
yum install -y ntpdate
ntpdate time1.aliyun.com
2. 安装Java环境
rz jdk-8u181-linux-x64.rpm
rpm -ivh jdk-8u181-linux-x64.rpm
3. 安装Logstash
rz logstash-6.6.0.rpm
rpm -ivh logstash-6.6.0.rpm
4. 授权
chown -R logstash.logstash /usr/share/logstash/
# nginx
/var/log/nginx/
access.log # json 格式
# tomcat
/usr/local/tomcat/logs/
tomcat_access # json 格式
vim /etc/logstash/conf.d/jpress_json.conf
input {
file {
type => "tomcat_access_json"
path => "/usr/share/tomcat/logs/tomcat_access_json*.log"
start_position => "beginning"
codec => "json"
}
}
input {
file {
type => "nginx_log_json"
path => "/var/log/nginx/access.log"
start_position => "beginning"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "%{type}_%{+YYYY-MM-dd}"
codec => "json"
}
}
# 启动
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/jpress_json.conf &
实现 Reids 消息队列
yum install -y redis
vim /etc/redis.conf
bind 172.16.1.20 # redis 网络端口设置
systemctl start redis
vim /etc/logstash/conf.d/redis_json.conf
input {
file {
type => "nginx_log"
path => "/var/log/nginx/access.log"
start_position => "beginning"
codec => "json"
}
file {
type => "tomcat_log"
path => "/usr/share/tomcat/logs/tomcat_access_json.*.log"
start_position => "beginning"
codec => "json"
}
}
output {
if [type] == "nginx_log" {
redis {
host => "172.16.1.20"
port => "6379"
data_type => "list"
db => "0"
key => "nginx_log"
}
}
if [type] == "tomcat_log" {
redis {
host => "172.16.1.20"
port => "6379"
data_type => "list"
db => "1"
key => "tomcat_log"
}
}
}
# 启动
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat_redis.conf &
# 查看
172.16.1.20:6379> SELECT 1
OK
172.16.1.20:6379[1]> KEYS *
1) "tomcat_log"
172.16.1.20:6379[1]> KEYS *
1) "tomcat_log"
172.16.1.20:6379[1]> SELECT 0
OK
172.16.1.20:6379> KEYS *
1) "nginx_log"
实现 Reids 消息队列 到es集群
vim /etc/logstash/conf.d/redis_to_es.conf
input {
redis {
host => "172.16.1.20"
port => "6379"
db => "0"
data_type => "list"
key => "nginx_log"
}
redis {
host => "172.16.1.20"
port => "6379"
db => "1"
data_type => "list"
key => "tomcat_log"
}
}
output {
if [type] == "nginx_log" {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "nginx_log_%{+YYYY-MM-dd}"
}
}
if [type] == "tomcat_log" {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "tomcat_log_%{+YYYY-MM-dd}"
}
}
}
# 启动多实例
mkdir /data/logstash/redis_es
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis_to_es.conf --path.data=/data/logstash/redis_es &
# 查看 (瞬间被消费)
172.16.1.20:6379> KEYS *
(empty list or set)
172.16.1.20:6379>
浙公网安备 33010602011771号