elk基础搭建，单服务器模拟

1、官网下载软件，并解压 tar -zxvf xxx.tar.gz

　　https://www.elastic.co/cn/downloads/past-releases（elk各版本）

　　本次搭建6.8.7版本，对应jdk8.

　　

 2、安装&配置elasticsearch

　　注意！elasticsearch不能用root启动需要单独的elasticsearch用户

　　(1)创建elasticsearch账户

　　groupadd elasticsearch                  #新建elasticsearch组

　　useradd elasticsearch -g elasticsearch -p elasticsearch  #新建一个elasticsearch用户

　　chown -R elasticsearch:elasticsearch  ./elasticsearch-6.8.7    #指定elasticsearch-7.4.1所属elasticsearch组

　　(2)切换账户启动elasticsearch

　　从root用户切换为一般用户，不需要输入elasticsearch用户的密码，直接使用su elasticsearch命令即可进入到elasticsearch用户中。 

　　在终端输入exit或使用快捷方式ctrl+d，可以退回到root用户，前提：我们是使用root用户从终端切换进入到了一般用户。

　　su elasticsearch #切换账户

　　./elasticsearch-6.8.7/bin/elasticsearch #启动elasticsearch

　　启动报错：我的Linux版本是centos6 版本内核不支持elasticsearch 的插件（要求centos 版本内核为3.5以上）所以报出以下警告⚠️
　　 

 　　解决问题：

 　　修改elasticsearch.yml文件，添加如下配置，禁用插件即可：

　　 bootstrap.system_call_filter: false

　　

 　　再次启动：不再报出异常信息

[elasticsearch@aliyun config]$ cd ../
[elasticsearch@aliyun elasticsearch-6.8.7]$ ./bin/elasticsearch
[2020-03-18T10:21:52,581][INFO ][o.e.e.NodeEnvironment    ] [70KIC_Z] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [32.1gb], net total_space [39.2gb], types [rootfs]
[2020-03-18T10:21:52,584][INFO ][o.e.e.NodeEnvironment    ] [70KIC_Z] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-03-18T10:21:52,586][INFO ][o.e.n.Node               ] [70KIC_Z] node name derived from node ID [70KIC_ZeTN2xW0J8OujWRQ]; set [node.name] to override
[2020-03-18T10:21:52,587][INFO ][o.e.n.Node               ] [70KIC_Z] version[6.8.7], pid[8850], build[default/tar/c63e621/2020-02-26T14:38:01.193138Z], OS[Linux/2.6.32-696.6.3.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_231/25.231-b11]
[2020-03-18T10:21:52,587][INFO ][o.e.n.Node               ] [70KIC_Z] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6056993285147514667, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/local/elk/elasticsearch-6.8.7, -Des.path.conf=/usr/local/elk/elasticsearch-6.8.7/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [aggs-matrix-stats]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [analysis-common]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [ingest-common]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [ingest-geoip]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [ingest-user-agent]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [lang-expression]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [lang-mustache]
[2020-03-18T10:21:54,674][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [lang-painless]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [mapper-extras]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [parent-join]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [percolator]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [rank-eval]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [reindex]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [repository-url]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [transport-netty4]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [tribe]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-ccr]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-core]
[2020-03-18T10:21:54,675][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-deprecation]
[2020-03-18T10:21:54,676][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-graph]
[2020-03-18T10:21:54,676][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-ilm]
[2020-03-18T10:21:54,676][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-logstash]
[2020-03-18T10:21:54,676][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-ml]
[2020-03-18T10:21:54,676][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-monitoring]
[2020-03-18T10:21:54,677][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-rollup]
[2020-03-18T10:21:54,677][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-security]
[2020-03-18T10:21:54,677][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-sql]
[2020-03-18T10:21:54,677][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-upgrade]
[2020-03-18T10:21:54,677][INFO ][o.e.p.PluginsService     ] [70KIC_Z] loaded module [x-pack-watcher]
[2020-03-18T10:21:54,678][INFO ][o.e.p.PluginsService     ] [70KIC_Z] no plugins loaded
[2020-03-18T10:21:58,704][INFO ][o.e.x.s.a.s.FileRolesStore] [70KIC_Z] parsed [0] roles from file [/usr/local/elk/elasticsearch-6.8.7/config/roles.yml]
[2020-03-18T10:21:59,589][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [70KIC_Z] [controller/8916] [Main.cc@109] controller (64 bit): Version 6.8.7 (Build f3f0534e9623b3) Copyright (c) 2020 Elasticsearch BV
[2020-03-18T10:22:00,224][DEBUG][o.e.a.ActionModule       ] [70KIC_Z] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-03-18T10:22:00,470][INFO ][o.e.d.DiscoveryModule    ] [70KIC_Z] using discovery type [zen] and host providers [settings]
[2020-03-18T10:22:01,388][INFO ][o.e.n.Node               ] [70KIC_Z] initialized
[2020-03-18T10:22:01,389][INFO ][o.e.n.Node               ] [70KIC_Z] starting ...
[2020-03-18T10:22:01,551][INFO ][o.e.t.TransportService   ] [70KIC_Z] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2020-03-18T10:22:01,623][WARN ][o.e.b.BootstrapChecks    ] [70KIC_Z] max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]
[2020-03-18T10:22:01,623][WARN ][o.e.b.BootstrapChecks    ] [70KIC_Z] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2020-03-18T10:22:04,683][INFO ][o.e.c.s.MasterService    ] [70KIC_Z] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {70KIC_Z}{70KIC_ZeTN2xW0J8OujWRQ}{X_GFOQbCRAqrA8JwHtHAdg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4153831424, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2020-03-18T10:22:04,688][INFO ][o.e.c.s.ClusterApplierService] [70KIC_Z] new_master {70KIC_Z}{70KIC_ZeTN2xW0J8OujWRQ}{X_GFOQbCRAqrA8JwHtHAdg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4153831424, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {70KIC_Z}{70KIC_ZeTN2xW0J8OujWRQ}{X_GFOQbCRAqrA8JwHtHAdg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4153831424, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2020-03-18T10:22:04,755][INFO ][o.e.h.n.Netty4HttpServerTransport] [70KIC_Z] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2020-03-18T10:22:04,761][INFO ][o.e.n.Node               ] [70KIC_Z] started
[2020-03-18T10:22:05,004][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [70KIC_Z] Failed to clear cache for realms [[]]
[2020-03-18T10:22:05,053][INFO ][o.e.l.LicenseService     ] [70KIC_Z] license [707a9552-1c29-4936-adb7-0ca7496591be] mode [basic] - valid
[2020-03-18T10:22:05,064][INFO ][o.e.g.GatewayService     ] [70KIC_Z] recovered [0] indices into cluster_state

　　完成elasticsearch启动，可以看到9200、9300端口已经启用。

 3、安装&配置logstash

　　先到config目录中新建一个logstash.conf配置文件。

　　touch logstash.conf

　　

 　　编辑logstash.conf文件，增加如下两块内容

input {
    file {
        type => "log"
        path => "/logs/*.log"
        start_position => "beginning"
    }
}

output {
  stdout {
  codec => rubydebug { }
  }

  elasticsearch {
    hosts => "127.0.0.1"
    index => "log-%{+YYYY.MM.dd}"
  }
}

　　logstash.conf主要有 input ，filter，output三大块：

　　input是读取logs文件下的所有log后缀的日志文件；

　　filter是一个过滤函数（可以先不配置，后续增加详细配置文档）；

　　output是配置导入到elasticsearch中，每天一个索引。

　　配置文件设置为conf/logstash.conf ，然后到bin目录下启动logstash

　　./logstash -f ../config/logstash.conf

 

 　　出现如上错误，说是Java环境变量错误

 　　检查Java环境变量配置，在JAVA_HOME后边增加“/”

 

　　 再次启动一切OK

 

 4、kibana配置&启动

　　kibana需要在kibana.yml文件中指定一下elasticSearch地址和外网访问的bind地址。

　　elasticsearch.url: "http://127.0.0.1:9200"
　　server.host: 0.0.0.0

 启动kibana

./kibana-6.8.7-linux-x86_64/bin/kibana
[root@aliyun elk]# ./kibana-6.8.7-linux-x86_64/bin/kibana
  log   [14:00:09.976] [info][status][plugin:kibana@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.009] [info][status][plugin:elasticsearch@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.012] [info][status][plugin:xpack_main@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.018] [info][status][plugin:graph@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.028] [info][status][plugin:monitoring@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.031] [info][status][plugin:spaces@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.039] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml
  log   [14:00:10.044] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.
  log   [14:00:10.051] [info][status][plugin:security@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.067] [info][status][plugin:searchprofiler@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.069] [info][status][plugin:ml@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.111] [info][status][plugin:tilemap@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.112] [info][status][plugin:watcher@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.124] [info][status][plugin:grokdebugger@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.129] [info][status][plugin:dashboard_mode@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.131] [info][status][plugin:logstash@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.137] [info][status][plugin:beats_management@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.168] [info][status][plugin:apm@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.170] [info][status][plugin:tile_map@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.172] [info][status][plugin:task_manager@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.175] [info][status][plugin:maps@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.179] [info][status][plugin:interpreter@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.186] [info][status][plugin:canvas@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.190] [info][status][plugin:license_management@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.192] [info][status][plugin:cloud@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.194] [info][status][plugin:index_management@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.207] [info][status][plugin:console@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.209] [info][status][plugin:console_extensions@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.212] [info][status][plugin:notifications@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.213] [info][status][plugin:index_lifecycle_management@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.261] [info][status][plugin:infra@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.264] [info][status][plugin:rollup@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.273] [info][status][plugin:remote_clusters@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.279] [info][status][plugin:cross_cluster_replication@6.8.7] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [14:00:10.289] [info][status][plugin:translations@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.299] [info][status][plugin:upgrade_assistant@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.312] [info][status][plugin:uptime@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.314] [info][status][plugin:oss_telemetry@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.324] [info][status][plugin:metrics@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.438] [info][status][plugin:timelion@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:10.921] [info][status][plugin:elasticsearch@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.187] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
  log   [14:00:11.192] [info][status][plugin:xpack_main@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.192] [info][status][plugin:graph@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.194] [info][status][plugin:searchprofiler@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.194] [info][status][plugin:ml@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.194] [info][status][plugin:tilemap@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.194] [info][status][plugin:watcher@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.195] [info][status][plugin:grokdebugger@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.195] [info][status][plugin:logstash@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.195] [info][status][plugin:beats_management@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.196] [info][status][plugin:index_management@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.196] [info][status][plugin:index_lifecycle_management@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.196] [info][status][plugin:rollup@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.197] [info][status][plugin:remote_clusters@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.197] [info][status][plugin:cross_cluster_replication@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.197] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
  log   [14:00:11.205] [info][status][plugin:security@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.206] [info][status][plugin:maps@6.8.7] Status changed from yellow to green - Ready
  log   [14:00:11.267] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
  log   [14:00:11.370] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  log   [14:00:11.375] [info][status][plugin:reporting@6.8.7] Status changed from uninitialized to green - Ready
  log   [14:00:11.838] [info][listening] Server running at http://0.0.0.0:5601
  log   [14:00:11.925] [info][status][plugin:spaces@6.8.7] Status changed from yellow to green - Ready

　　浏览器查看kibana

　　http://127.0.0.1:5601/

posted on 2020-03-18 22:22  XT_小嘎  阅读(248)  评论(0)  编辑  收藏  举报
Copyright © 2024 XT_小嘎
Powered by .NET 8.0 on Kubernetes Powered by: 博客园