docker 部署 elk

docker 部署 elasticsearch + logstash + kibana  ELK

这里按照 elasticsearch 7.17.28 这个 LTS 版本为例

一 准备工作

1 首先拉取镜像:

docker pull elasticsearch:7.1.28
docker pull logstash:7.17.28
docker pull kibana:7.17.28 

2 若要把 数据 ,配置文件, log 挂载在容器之外, 则需要配置一些目录

mkdir -p /data/elasticsearch/
mkdir -p /data/kibana
mkdir -p /data/logstash

elasticsearch 的数据, 配置, 日志, 插件 目录

mkdir -p /data/elasticsearch/config
mkdir -p /data/elasticsearch/data
mkdir -p /data/elasticsearch/logs
mkdir -p /data/elasticsearch/plugins
chmod -rf 777 /data/elasticsearch/config/
chmod -rf 777 /data/elasticsearch/data/
chmod -rf 777 /data/elasticsearch/logs/
chmod -rf 777 /data/elasticsearch/plugins/

3 设置 .yml 文件 用来配置容器内的es

vim /data/elasticsearch/config/elasticsearch.yml

cluster.name: es-cluster
node.name: es-node-1
network.host: 0.0.0.0
discovery.type: single-node
http.port: 9200
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

保存,然后设置权限到可执行

chmod +x /data/elasticsearch/config/elasticsearch.yml

 

4 安装 logstash,首先配置 logstash :  创建logstash 的配置文件 .conf, 其中,elasticsearch 的 密码,我以  Elastic1@345  举例

cd /data/logstash
touch logstash.conf
vim logstash.conf

    input {
      tcp {
        mode => "server"
        host => "0.0.0.0"
        port => 4560
        codec => json_lines
        type => "debug"
      }
      tcp {
        mode => "server"
        host => "0.0.0.0"
        port => 4561
        codec => json_lines
        type => "error"
      }
      tcp {
        mode => "server"
        host => "0.0.0.0"
        port => 4562
        codec => json_lines
        type => "business"
      }
      tcp {
        mode => "server"
        host => "0.0.0.0"
        port => 4563
        codec => json_lines
        type => "record"
      }
    }
    filter{
      if [type] == "record" {
        mutate {
          remove_field => "port"
          remove_field => "host"
          remove_field => "@version"
        }
        json {
          source => "message"
          remove_field => ["message"]
        }
      }
    }
    output {
      elasticsearch {
        hosts => "es:9200"
        index => "%{project}-%{service}"
        user => "elastic"
        password => "Elastic1@345"
      }
    }

保存

5 配置 logstash.yml

cd /data/logstash
touch logstash.yml
vim logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: Elastic1@345
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]

 

6 配置 kibana

cd /data/kibana
touch kibana.yml
vim kibana.yml

server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.username: "elastic"
elasticsearch.password: "Elastic1@345"
i18n.locale: "zh-CN"

 

二 操作

在准备工作完成后, 各个工具的配置文件, 挂载目录,创建好,修改好,即可创建 docker 容器来实施

1 创建 elasticsearch 容器

docker run -p 9200:9200  --name elasticsearch \
-e "discovery.type=single-node" \
-e "ELASTIC_PASSWORD=Elastic1@345" \
-e "ES_JAVA_OPTS=-Xms512m -Xmx1024m" \
-v /etc/localtime:/etc/localtime \
-v /data/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /data/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-v /data/elasticsearch/data:/usr/share/elasticsearch/data \
-v /data/elasticsearch/logs:/usr/share/elasticsearch/logs \
--restart=always \
-d elasticsearch:7.17.28

2 创建 logstash 容器

docker run --name logstash -p 4560:4560 -p 4561:4561 -p 4562:4562 -p 4563:4563 \
--link elasticsearch:es \
-v /etc/localtime:/etc/localtime \
-v /data/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml \
-v /data/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
--restart=always \
-d logstash:7.17.28

3 创建 kibana 容器

docker run --name kibana -p 5601:5601 \
--link elasticsearch:es \
-e "elasticsearch.hosts=http://es:9200" \
-v /etc/localtime:/etc/localtime \
-v /data/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml \
--restart=always \
-d kibana:7.17.28

4 插件准备

如果要安装分词器,可以去 https://release.infinilabs.com/analysis-ik/stable/ 这里查找对应版本的分词器,下载下来,解压,放到 /data/elasticsearch/plugins 里面去,

   1  先进入插件目录

cd /data/elasticsearch/plugins

   2 创建分词插件目录

mkdir elasticsearch-analysis-ik

　　

然后重新 启动 elasticsearch 容器

 

三 遇到的问题分析

1  因虚拟内存小而导致 容器反复重启, 现象就是,docker ps -a 查看服务的时候, 一直在重启,说明容器没有启动,可能的原因是内部错误,也可能是 内存小导致无法启动

     修改虚拟内存区域大小，否则会因为过小而无法启动

sysctl -w vm.max_map_count=262144