Elastic Stack 集群搭建

Elasticsearch相关软件安装

1.1 Elasticsearch相关软件下载

这里提供了下载大礼包

https://www.elastic.co/cn/downloads/past-releases/kibana-7-6-1
https://www.elastic.co/cn/downloads/past-releases/elasticsearch-7-6-1
https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-1
https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1
https://nodejs.org/zh-cn/download/
https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz

注意：elasticsearch-7版本以后不许要单独安装JDK环境，elasticsearch二进制安装启动需要普通用户，root用户启动会报错

1.2 Elasticsearch单机部署

1.启动需要使用普通用户并创建相关文件目录

groupadd es
useradd es -g es
mkdir -p /usr/local/elasticsearch/
tar xf elasticsearch-7.6.1-linux-x86_64.tar.gz -C /usr/local/elasticsearch/

2.修改文件权柄

vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535

3.修改虚拟内存区域

cat >>/etc/sysctl.conf <<EOF
vm.max_map_count = 262144
EOF

sysctl -p

4.修改java环境变量

#修改java环境变量
vim /usr/local/elasticsearch/elasticsearch-7.6.1/bin/elasticsearch-env
#elasticsearch-env脚本文件中添加JAVA环境变量，本环境变量目录为elasticsearch下的JDK
JAVA_HOME={安装目录下的}elasticsearch/jdk

5.修改配置文件

vim /usr/local/elasticsearch/elasticsearch-7.6.1/conf/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.0.200"]         
cluster.initial_master_nodes: ["10.0.0.200"]
#跨域请求配置(为了让类似head的第三方插件可以请求es)
http.cors.enabled: true
http.cors.allow-origin: "*"

6.jav.options配置文件

vim /usr/local/elasticsearch/elasticsearch-7.6.1/conf/jvm.options
#由于UseConcMarkSweepGC修改外G1GC
8-13:-XX:+UseG1GC

7.授权目录

chown -R es. /usr/local/elasticsearch/

8.启动并测试

elasticsearch-7.6.1/bin/elasticsearch -d
curl 10.0.0.200:9200

1.3 Elasticsearch集群部署

1.需要修改配置文件，其他的和单机一样

vim /usr/local/elasticsearch/elasticsearch-7.6.1/conf/elasticsearch.yml
# 集群名称
cluster.name: elk
# 节点名称和cluster.initial_master_nodes 这里的名字需要对应
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.0.200", "10.0.0.201", "10.0.0.202"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
discovery.zen.ping_timeout: 60s

#跨域请求配置(为了让类似head的第三方插件可以请求es)
http.cors.enabled: true
http.cors.allow-origin: "*"

2.查看集群状态

 http://10.0.0.200:9200/_cluster/health?pretty

1.4 安装可视化工具elasticsearch head插件

1.下载node

https://nodejs.org/zh-cn/download/ 官网网站,可以选择版本

wget https://nodejs.org/dist/v14.17.6/node-v14.17.6-linux-x64.tar.xz

2.解压并添加环境变量

tar xf node-v14.17.6-linux-x64.tar.xz -C /usr/local/node/
cat >> /etc/profile <<EOF
export NODE_HOME=/usr/local/node/node-v14.17.6-linux-x64
export PATH=$PATH:$NODE_HOME/bin
EOF

source /etc/profile

3.测试命令并安装

node -v
npm -v
npm install -g cnpm --registry=https://registry.npm.taobao.org

4.创建插件目录

mkdir -p /usr/local/elasticsearch/plugins
cd /usr/local/elasticsearch/plugins

5.拉取elasticsearch-head代码

git clone git://github.com/mobz/elasticsearch-head.git

6..进入目录修改配置文件,添加参数

vim Gruntfile.js
#搜索:server
options: {
            hostname:'*',     #添加此处即可,不让无法访问
            port: 9100,
            base: '.',
            keepalive: true

7 安装依赖

cd /usr/local/elasticsearch/plugins/elasticsearch-head
cnpm install

8.启动

cnpm run start &

1.5 kibana部署

1.下载kibana,建议RPM安装

https://www.elastic.co/cn/downloads/past-releases/kibana-7-6-1 下载相应版本

2.创建kibana目录并解压

mkidr -p /usr/local/kibana
tar xf kibana-7.6.1-linux-x86_64.tar.gz  -C /usr/local/kibana

3.修改配置

vim kibana.yaml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.0.0.200:9200","http://10.0.0.201:9201","http://10.0.0.202:9202"]

4启动

cd /usr/local/kibana/kibana-7.6.1-linux-x86_64/bin       #进入bin目录
./kibana --allow-root

1.6 filebeat部署

1.下载kibana,建议RPM安装

https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1

2.安装

rpm -ivh https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1

1.7 filebeat-ES架构部署

1.架构图

2.自定义索引配置

vim filebeat.yml
  index: "nginx-%{[agent.version]}-%{+yyyy.MM}"
setup.ilm.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: tru

3.Nginx的json格式日志理想状态

1.可以把日志所有字段拆分出来
{
	$remote_addr : 192.168.12.254
	- : -
	$remote_user : -
	[$time_local]: [10/Sep/2019:10:52:08 +0800]
	$request: GET /jhdgsjfgjhshj HTTP/1.0
	$status : 404
	$body_bytes_sent : 153
	$http_referer : -
	$http_user_agent :ApacheBench/2.3
	$http_x_forwarded_for:-
}

4.修改nginx配置文件使日志转换成json

log_format json '{ "time_local": "$time_local", '
                          '"remote_addr": "$remote_addr", '
                          '"referer": "$http_referer", '
                          '"request": "$request", '
                          '"status": $status, '
                          '"bytes": $body_bytes_sent, '
                          '"agent": "$http_user_agent", '
                          '"x_forwarded": "$http_x_forwarded_for", '
                          '"up_addr": "$upstream_addr",'
                          '"up_host": "$upstream_http_host",'
                          '"upstream_time": "$upstream_response_time",'
                          '"request_time": "$request_time"'

5.清除旧日志

> /var/log/nginx/access.log
nginx -t 
systemctl restart nginx

6.修改filebeat配置文件支持json解析

通过查看发现,虽然nginx日志变成了json,但是es里还是存储在message里仍然不能拆

#filebeat配置文件
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /application/nginx/logs/access*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /application/nginx/logs/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.200:9200","10.0.0.201:9200","10.0.0.202:9200"]
  indices:
    - index: "nginx-access%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"
  #username: "elastic"
  #password: "changeme"

processors:
  - drop_fields:
      fields: ["ecs","log"]

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false


logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

setup.template.settings:
  index.number_of_shards: 1

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: true
setup.template.enabled: false
setup.ilm.enabled: false
EOF

7.ES配置

cat > /usr/local/elasticsearch/elasticsearch-7.6.1/conf/elasticsearch.yml <<EOF
# 集群名称
cluster.name: elk
# 节点名称和cluster.initial_master_nodes 这里的名字需要对应
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.0.200", "10.0.0.201", "10.0.0.202"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
discovery.zen.ping_timeout: 60s

#跨域请求配置(为了让类似head的第三方插件可以请求es)
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF

1.8 filebeat按照服务类型拆分索引

1.第一种写法

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /application/nginx/logs/access*.log
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /application/nginx/logs/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.200:9200"]
  indices:
    - index: "nginx-access%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"
  #username: "elastic"
  #password: "changeme"

processors:
  - drop_fields:
      fields: ["ecs","log"]

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

setup.template.settings:
  index.number_of_shards: 1

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: true
setup.template.enabled: false
setup.ilm.enabled: false
EOF

2.第二种写法：

使用filebeat自带的模块，

模块方式不需要input，可以用 fileset.name: 字段来区分access

确保Nginx的日志为普通格式

但不建议这样做，会加大ES的压力，尽量使用将日志转换成JSON格式

确认Nginx日志为普通格式

检查filebeat配置文件里是否包含模块相关参数

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

激活filebeat模块并查看

filebeat modules --list 
filebeat enable nginx

filebeat.yml最终配置文件

cat >/etc/filebeat/filebeat.yml<<EOF 
output.elasticsearch:
  hosts: ["10.0.0.200:9200"]
  indices:
    - index: "nginx-access%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.name: "access"
    - index: "nginx-error%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.name: "error"
  #username: "elastic"
  #password: "changeme"

processors:
  - drop_fields:
      fields: ["ecs","log"]

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s


logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

setup.template.settings:
  index.number_of_shards: 1

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: true
setup.template.enabled: false
setup.ilm.enabled: false

1.9 Filebeat收集tomcat的json日志

1.修改tomcat日志格式 prefix="localhost_access_log" suffix=".txt"下方添加

2.配置filebeat配置文件

1.修改日志格式

[root@web01 ~]# /opt/tomcat/bin/shutdown.sh
[root@web01 ~]# vim /opt/tomcat/conf/server.xml 
	       pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>

2 启动tomcat

/opt/tomcat/bin/startup.sh

3.配置filebeat

cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/tomcat/logs/localhost_access_log.*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat"]

output.elasticsearch:
  hosts: ["10.0.0.200:9200"，"10.0.0.201:9200"，"10.0.0.202:9200"]
  index: "tomcat_access-%{[agentt.version]}-%{+yyyy.MM}"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644


setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true
setup.ilm.enabled: false
EOF

4.启动 filebeat

# 前台启动：利于查看配置文件错误
filebeat -e -d "publish"
# 后台启动
systemctl restart filebeat

2.0 filebeat收集java多行匹配模式

2.1filbeat使用模块收集nginx日志

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

2.2激活filebeat模块并查看

filebeat modules --list 
filebeat enable nginx

2.3filebeat 配置配置文件

output.elasticsearch: 
  hosts: ["10.0.0.200"]
  index: "JAVA-%{[agent.version]}-%{+yyyy.MM}"
filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "JAVA"
setup.template.pattern: "JAVA-*

2.4进入kinaba页面进行测试

2.1 使用redis缓存服务来缓解ES压力

2.1.1 架构图

架构流程：filebeat------>>redis<-<-----logstash---->>ES----->>kibana

注意：此架构中，redis只能当做高可用，无法实现高性能，只有当redis服务器出现问题，备用redis才会进行工作

2.1.2 部署logstash

注意事项:

1.需要提前部署好JDK环境,否则无法安装!

2.尽量不要安装在有其他JDK环境下,否则容易出问题

1.下载logstash

https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-1 下载相应版本

2.安装logstash

rpm -ivh logstash-7.6.1.rpm

3.启动

#前台启动：利于查看报错
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf

#后台启动：systemctl start logstash

2.1.3 配置filebeta文件

一下配置为多服务配置文件，其中包含nginx日志文件、多行JAVA日志文件、Tomcat日志文件

配置方法说明：

配置filebeat.inputs 日志文件路径，并增加tags标记

配置output.redis 指向redis服务器，一定注意，filebeat不支持redis集群或者Cluster模式

output.redis: key 指定日志标记log，用于logstash抓取日志时候的匹配项！次配置方法是将三个服务的日志都存入log中发送给redis

logstash input去redis中获取标记log的日，并进行处理！

疑问？如何区分那个是nginx 、tomcat、java他们的日志

解答：logstash在发送给ES之前会进行处理，它会根据filebeat input中配置的tags: ["nginx-access"] 标记去区日志服务的类型

数据进行处理后发送给ES ，最终ES发送kibana进行数据展示

总结：以上配置的好处：如果需要增加日志，只需要增加2处配置即可

增加filebeat.inputs: 配置中的日志路径

增加logstash配置中的output，匹配filebeat.inputs中配置的tags 标签即可

1.filebeat配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /application/nginx/logs/access*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx-access"]

- type: log
  enabled: true
  paths:
    - /application/nginx/logs/error*.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  tags: ["es-error"]

- type: log
  enabled: true
  paths:
    - /usr/local/tomcat/logs/localhost_access_log*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat-access"]

output.redis:
  hosts: ["10.0.0.92:7000"]
  key: "log"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

setup.template.settings:
  index.number_of_shards: 1

setup.template.enabled: false
setup.ilm.enabled: false

2.logstash 配置文件

input {
  redis {
    host => "10.0.0.92"
    port => "7000"
    db => "0"
    key => "log"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "nginx-access" in [tags] {
      elasticsearch {
        hosts => ["http://10.0.0.200:9200","http://10.0.0.200:9201","http://10.0.0.200:9202"]
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "es-error" in [tags] {
      elasticsearch {
        hosts => ["http://10.0.0.200:9200","http://10.0.0.200:9201","http://10.0.0.200:9202"]
        manage_template => false
        index => "es_error-%{+yyyy.MM}"
      }
    }
    if "tomcat-access" in [tags] {
      elasticsearch {
        hosts => ["http://10.0.0.200:9200","http://10.0.0.200:9201","http://10.0.0.200:9202"]
        manage_template => false
        index => "tomcat_access-%{+yyyy.MM}"
      }
    }
}

2.2 使用kafka作为缓存

疑问？为什么使用kafka替换redis ？

解答：

由于filebeat output 不支持redis集群，无法实现高性能！如日志量非常大，可能会倒是处理不及时

kafka处理数据的性能要比redis强大~

kafuka可以真正实现高可用、高性能，支持分布式架构

缺点：

配置复杂

需要引入第三方插件模块 zookeeper

2.2.1 架构图

2.2.2 部署Zookeeper

1.配置hosts和密钥

cat >/etc/hosts<<EOF
10.0.0.200 es1
10.0.0.201 kafka-1
10.0.0.202 kafka-2
10.0.0.203 kafka-3
EOF

ssh-keygen
ssh-copy-id 10.0.0.202
ssh-copy-id 10.0.0.203

2.安装配置zookeeper

kafka-1操作

# 提前三台设备都创建application目录
mkdir /application  

tar xf apache-zookeeper-3.6.3-bin.tar.gz -C /application/
ln -s /application/zookeeper-3.6.3/ /application/zookeeper
mkdir -p /data/zookeeper
cp /application/zookeeper/conf/zoo_sample.cfg /application/zookeeper/conf/zoo.cfg

# 修改配置文件
cat >/application/zookeeper/conf/zoo.cfg<<EOF
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=10.0.0.201:2888:3888
server.2=10.0.0.202:2888:3888
server.3=10.0.0.203:2888:3888

# 查看ID
echo "1" > /data/zookeeper/myid
cat /data/zookeeper/myid

# 同步到其他2台机子
rsync -avz /application/zookeeper* 10.0.0.202:/application/
rsync -avz /application/zookeeper* 10.0.0.203:/application/

kafka-2操作

mkdir -p /data/zookeeper
echo "2" > /data/zookeeper/myid
cat /data/zookeeper/myid

kafka-3操作

mkdir -p /data/zookeeper
echo "3" > /data/zookeeper/myid
cat /data/zookeeper/myid

3.所有节点启动zookeeper

/application/zookeeper/bin/zkServer.sh start

4.每个节点都检查

/application/zookeeper/bin/zkServer.sh status

5.测试zookeeper

在一个节点上执行,创建一个频道

/application/zookeeper/bin/zkCli.sh -server 10.0.0.51:2181
create /test "hello"

在其他节点上看能否接收到

/application/zookeeper/bin/zkCli.sh -server 10.0.0.52:2181
get /test

2.2.3.安装部署kafka

kafka-1操作

# 提前三台设备都创建application目录
mkdir /application

tar zxf kafka_2.11-1.0.0.tgz -C /application/
ln -s /application/kafka_2.11-1.0.0/ /opt/kafka
mkdir /application/kafka/logs

# 修改配置文件
cat >/application/kafka/config/server.properties <<EOF
broker.id=1
listeners=PLAINTEXT://10.0.0.201:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
EOF

# 同步其他2台设备
rsync -avz /application/kafka* 10.0.0.52:/applicaton/
rsync -avz /application/kafka* 10.0.0.53:/applicaton/

kafka-2操作

sed -i "s#10.0.0.201:9092#10.0.0.202:9092#g" /application/kafka/config/server.properties
sed -i "s#broker.id=1#broker.id=2#g" /application/kafka/config/server.properties

kafka-3操作

sed -i "s#10.0.0.201:9092#10.0.0.203:9092#g" /application/kafka/config/server.properties
sed -i "s#broker.id=3#broker.id=3#g" /application/kafka/config/server.properties

2.2.5.前台启动测试

/application/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties

2.2.6.验证进程

jps

2.2.8.测试创建topic

# 老版本
/application/kafka/bin/kafka-topics.sh --create  --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181 --partitions 3 --replication-factor 3 --topic kafkatest

# 新版本
./kafka-topics.sh --create --bootstrap-server k1.cn:9092 --replication-factor 1 --partitions 1 --topic xxoo

2.2.9.测试获取toppid

# 老版本查看
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181 --topic kafkatest

# 新版本
./kafka-topics.sh --describe --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181  --topic xxoo

2.2.9.测试删除topic

/opt/kafka/bin/kafka-topics.sh --delete --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181 --topic kafkatest

2.2.10.kafka测试命令发送消息

创建命令

/application/kafka/bin/kafka-topics.sh --create --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181 --partitions 3 --replication-factor 3 --topic  messagetest

测试发送消息

/application/kafka/bin/kafka-console-producer.sh --broker-list  10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092 --topic  messagetest

其他节点测试接收

/application/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181 --topic messagetest --from-beginning

2.2.11.测试获取所有的频道

# 老版本
/application/kafka/bin/kafka-topics.sh  --list --zookeeper 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181

# 新版本
./kafka-topics.sh --list --bootstrap-server 10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092

# 新版本查看消费组的的列表
./kafka-consumer-groups.sh  --list  --bootstrap-server 192.168.100.11:9092

2.2.12.测试成功之后,可以放在后台启动

/applicaton/kafka/bin/kafka-server-start.sh  -daemon /opt/kafka/config/server.properties

2.2.13.修改filebeat 配置文件

对比 redis缓存 ,只需要output进行了修改

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /application/nginx/logs/access*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx-access"]

- type: log
  enabled: true
  paths:
    - /application/nginx/logs/error*.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  tags: ["es-error"]

- type: log
  enabled: true
  paths:
    - /usr/local/tomcat/logs/localhost_access_log*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat-access"]

output.kafka:
  hosts: ["10.0.0.201:9092", "10.0.0.202:9092", "10.0.0.203:9092"]
  topic: "filebeat"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

setup.template.settings:
  index.number_of_shards: 1

setup.template.enabled: false
setup.template.overwrite: true
setup.ilm.enabled: false

2.2.14.修改logstash配置文件

对比redis缓存,只需要修改 input 内容

[root@mha1 ~]# more /etc/logstash/conf.d/kafka.conf
input {
  kafka{
    bootstrap_servers=>["10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092"]
    topics=>["filebeat"]
    #group_id=>"logstash"
    codec => "json"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "nginx-access" in [tags] {
      elasticsearch {
        hosts => ["http://10.0.0.200:9200","http://10.0.0.200:9201","http://10.0.0.200:9202"]
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "es-error" in [tags] {
      elasticsearch {
        hosts => ["http://10.0.0.200:9200","http://10.0.0.200:9201","http://10.0.0.200:9202"]
        manage_template => false
        index => "es_error-%{+yyyy.MM}"
      }
    }
    if "tomcat-access" in [tags] {
      elasticsearch {
        hosts => ["http://10.0.0.200:9200","http://10.0.0.200:9201","http://10.0.0.200:9202"]
        manage_template => false
        index => "tomcat_access-%{+yyyy.MM}"
      }
    }
}

2.2.15 启动logstash并测试

前台启动

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf

后台启动

systemctl start logstash

2.2.16 logstash移除不需要的字段

在filter区块里添加remove_field字段即可

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
    remove_field => [ "beat" ]
  }
}

Kafka新老版本命令行参数对比详解

对比详解转载地址： https://blog.csdn.net/knight_zhou/article/details/105029521

新老版本Kafka启动方式（两者一致）：

bin/zookeeper-server-start.sh config/zookeeper.properties    # 启动内置zk
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties   # 使用守护进程启动
 
 
bin/kafka-server-start.sh onfig/server.properties    # 前台启动Kafka
bin/kafka-server-start.sh -daemon config/server.properties      # daemon方式启动Kafka

新版本Kafka命令行参数（kafka_2.11-2.3.0 为例)：

##创建topic
./kafka-topics.sh --create --bootstrap-server k1.cn:9092 --replication-factor 1 --partitions 1 --topic xxoo
 
#查看topic
./kafka-topics.sh --list --bootstrap-server k1.cn:9092
 
# topic 描述
./kafka-topics.sh --describe --zookeeper k1.cn:2181  --topic xxoo
 
# producer(控制台向topic生产数据)
./kafka-console-producer.sh --broker-list k1.cn:9092 --topic xxoo
>this is a message
>this is another message
 
##consumer(控制台消费topic的数据2)
./kafka-console-consumer.sh --bootstrap-server k1.cn:9092 --topic xxoo --from-beginning
this is a message
this is another message
 
## 查看某一个topic对应的消息数量
./kafka-run-class.sh  kafka.tools.GetOffsetShell --broker-list k1.cn:9092 --topic xxoo --time -1
 
## 新版本的消费者组名和它要消费的那个topic的offset信息就会被记录在broker服务器上,老版本存在zookeeper上
 
./kafka-consumer-groups.sh --bootstrap-server k1.cn:9092 --list
 
./kafka-consumer-groups.sh --bootstrap-server  kafka01.qq.cn:9092,kafka02.qq.cn:9092,kafka03.qq.cn:9092 --list
 
 
##删除消费组
./kafka-consumer-groups.sh --bootstrap-server 192.168.100.11:9092 --delete --group py-test
 
##查看消费组的的列表
./kafka-consumer-groups.sh  --list  --bootstrap-server 192.168.100.11:9092
或者
./kafka-consumer-groups.sh --bootstrap-server k1.cn:9092 xxoo --list
 
## 查看特定消费组的情况
./kafka-consumer-groups.sh --bootstrap-server 192.168.100.11:9092 --group py-test --describe

旧版本Kafka命令行参数

# 查看topic
./kafka-topics.sh --list --zookeeper k1.cn:2181
 
## topic描述
./kafka-topics.sh --describe --zookeeper k1.cn:2181  --topic xxoo
 
## 创建topic
./kafka-topics.sh --create --zookeeper k1.cn:2181 --replication-factor 1 --partitions 1 --topic xxoo
 
## consumer(控制台消费topic的数据2)
./kafka-console-consumer.sh --bootstrap-server k1.cn:9092 --topic xxoo --from-beginning
 
# 指定消费组消费
./kafka-console-consumer.sh --bootstrap-server k1.cn:9092 --topic xxoo --group xx-group
 
###生产数据
./kafka-console-producer.sh --broker-list k1.cn:9092 --topic xxoo
 
##查看某一个topic对应的消息数
./kafka-run-class.sh  kafka.tools.GetOffsetShell --broker-list k1.cn:9092 --topic xxoo --time -1

posted @ 2021-09-27 10:20 mingge1110 阅读(149) 评论(0) 收藏举报

刷新页面返回顶部

mingge1110

Elastic Stack 集群搭建

Elasticsearch相关软件安装

1.1 Elasticsearch相关软件下载

1.2 Elasticsearch单机部署

1.启动需要使用普通用户并创建相关文件目录

2.修改文件权柄

3.修改虚拟内存区域

4.修改java环境变量

5.修改配置文件

6.jav.options配置文件

7.授权目录

8.启动并测试

1.3 Elasticsearch集群部署

1.需要修改配置文件，其他的和单机一样

2.查看集群状态

1.4 安装可视化工具elasticsearch head插件

1.下载node

2.解压并 添加环境变量

3.测试命令并安装

4.创建插件目录

5.拉取elasticsearch-head代码

6..进入目录修改配置文件,添加参数

7 安装依赖

8.启动

1.5 kibana部署

1.下载kibana,建议RPM安装

2.创建kibana目录并解压

3.修改配置

4启动

1.6 filebeat部署

1.下载kibana,建议RPM安装

2.安装

1.7 filebeat-ES架构部署

1.架构图

2.自定义索引配置

3.Nginx的json格式日志理想状态

4.修改nginx配置文件使日志转换成json

5.清除旧日志

6.修改filebeat配置文件支持json解析

7.ES配置

1.8 filebeat按照服务类型拆分索引

1.第一种写法

2.第二种写法：

1.9 Filebeat收集tomcat的json日志

1.修改日志格式

2 启动tomcat

3.配置filebeat

4.启动 filebeat

2.0 filebeat收集java多行匹配模式

2.1filbeat使用模块收集nginx日志

2.2激活filebeat模块并查看

2.3filebeat 配置配置文件

2.4进入kinaba页面进行测试

2.1 使用redis缓存服务来缓解ES压力

2.1.1 架构图

2.1.2 部署logstash

2.1.3 配置filebeta文件

1.filebeat配置文件

2.logstash 配置文件

2.2 使用kafka作为缓存

2.2.1 架构图

2.2.2 部署Zookeeper

1.配置hosts和密钥

2.安装配置zookeeper

3.所有节点启动zookeeper

4.每个节点都检查

5.测试zookeeper

2.2.3.安装部署kafka

kafka-1操作

kafka-2操作

kafka-3操作

2.2.5.前台启动测试

2.2.6.验证进程

2.2.8.测试创建topic

2.2.9.测试获取toppid

2.2.9.测试删除topic

2.2.10.kafka测试命令发送消息

创建命令

测试发送消息

2.解压并添加环境变量