elk x-pack

安装 https://www.cnblogs.com/wxw16/p/6150681.html 

x-pack 权限说明https://blog.csdn.net/wu2700222/article/details/90713570

x-pack alert说明 https://blog.csdn.net/gamer_gyt/article/details/53016426

x-pack https://www.jianshu.com/p/23dbe4cc638e

filebeta 安装 https://www.cnblogs.com/yangxiaoyi/p/7240205.html

                     https://www.cnblogs.com/cjsblog/p/9495024.html           

                     https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

            配置  https://arch-long.cn/articles/elasticsearch/FileBeat.html

　　              https://www.dazhuanlan.com/2019/12/10/5deed81d396ad/

                    https://www.cnblogs.com/whych/p/9958188.html

filebeta 配置中文描述 https://www.cnblogs.com/zlslch/p/6622079.html

kibana 配置中文描述 https://segmentfault.com/a/1190000015784224

es 配置说明 https://www.tianmingxing.com/2019/06/20/在ElasticSearch6.8及以上版本开启安全认证功能/

es 集群概念 https://www.cnblogs.com/xidianzxm/p/11775129.html

cerbro https://github.com/lmenezes/cerebro

7.1

1.es 要求用非 root 账号启动，需要新建如下账号并赋予权限来操作

useradd mjyang -g elasticsearch -p /var/lib/elasticsearch

chown -R mjyang:elasticsearch elasticsearch

问题描述：

启动常见错误：https://www.cnblogs.com/zhi-leaf/p/8484337.html

                         https://www.cnblogs.com/hellxz/p/11009634.html

另外一个错误：memory locking requested for elasticsearch process but memory is not locked

解决办法如下：

需要修改
/etc/security/limits.conf 
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

修改：
/etc/sysctl.conf 
vm.swappiness=0

之后重启机器  

sudo systemctl daemon-reload

 

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/setting-system-settings.html

2.默认开启 x-pack 权限安全， elasticsearch.yml 需要开启如下配置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

3.需要创建用于登录es，kibana的用户

es 启动后，执行

bin/elasticsearch-keystore create --先创建keystore文件

bin/elasticsearch-setup-passwords interactive  初始化账号密码

4.为 es  内部使用 ssl 创建安全证书，用于内部安全传输，防止外部节点加入集群，窃取数据

  1. 生成CA证书 bin/elasticsearch-certutil ca，将产生新文件 elastic-stack-ca.p12。该 elasticsearch-certutil 命令还会提示你输入密码以保护文件和密钥，请保留该文件的副本并记住其密码。
  2. 为集群中的每个节点生成证书和私钥 bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12，将产生新文件 elastic-certificates.p12。系统还会提示你输入密码，你可以输入证书和密钥的密码，也可以按Enter键将密码留空。默认情况下 elasticsearch-certutil 生成没有主机名信息的证书，这意味着你可以将证书用于集群中的每个节点，另外要关闭主机名验证。
  3. 将 elastic-certificates.p12 文件复制到每个节点上Elasticsearch配置目录中。例如，/home/es/config/certs。无需将 elastic-stack-ca.p12文件复制到此目录。

     mkdir config/certs
     mv elastic-certificates.p12 config/certs/

4.在 elasticsearch.yml 文件中增加如下配置项

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 

   xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

   如果你在创建证书时输入了密码，那可以通过下面的方法设置。

          bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
          bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

　　 需要重启下 xshell 生效。

        5.为 es 启用 https 协议

　　xpack.security.http.ssl.enabled: true

       xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12

       xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

       如果你在创建证书时输入了密码，那可以通过下面的方法设置。

       bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
       bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

       需要重启下 xshell 生效。 

       6.如果是 kibana 要连接到 es

# 为kibana生成pem
openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elastic-ca.pem

elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.ssl.certificateAuthorities: [ "/Users/yiruan/geektime/kibana-7.1.0/config/certs/elastic-ca.pem" ]
elasticsearch.ssl.verificationMode: certificate

# 为 Kibna 配置 HTTPS
# 生成后解压，包含了instance.crt 和 instance.key
bin/elasticsearch-certutil ca --pem

server.ssl.enabled: true
server.ssl.certificate: config/certs/instance.crt
server.ssl.key: config/certs/instance.key

 5.filebeat 运行 ./filebeat -e -c /etc/filebeat/filebeat.yml

setup.ilm.enabled: false --禁用es默认索引声明周期
setup.template.enabled: true --自动载入模板，相当于执行 setup --dashboards
setup.template.name: "filebeat-%{[agent.version]}"
setup.template.pattern: "filebeat-%{[agent.version]}-*"
setup.template.fields: "/etc/filebeat/fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
index.codec: best_compression
index.routing.allocation.require.box_type: "hot"

另外就是一个 ssl 的传输设置

 

beat 需要验证hostname或ip，而es签发证书的时候如果没指定，这里会出错

setup.kibana:

# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "https://192.168.117.77:5601"
ssl.enabled: true
username: "elastic"
password: "Aa123456"
ssl.certificate_authorities: ["/usr/share/filebeat/certs/elastic-ca.pem"]
ssl.certificate: "/usr/share/filebeat/certs/ca.crt"
ssl.key: "/usr/share/filebeat/certs/ca.key"
ssl.verification_mode: none

output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.117.77:9200"]
#index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"

 indices:

- index: "prod-%{[log_type]}-%{+yyyy-MM-dd}"
when.or:
- equals:
source: 'b2c'
- equals:
source: 'erp'

# Optional protocol and basic auth credentials.
protocol: "https"
username: "elastic"
password: "Aa123456"

ssl.certificate_authorities: ["/usr/share/filebeat/certs/elastic-ca.pem"]
ssl.certificate: "/usr/share/filebeat/certs/ca.crt"
ssl.key: "/usr/share/filebeat/certs/ca.key"
ssl.verification_mode: none --通过指定 node 来禁用验证

7.插件安装

  1. ik： ./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.1.0/elasticsearch-analysis-ik-7.1.0.zip
  2. 拼音： ./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-pinyin/releases/download/v7.1.0/elasticsearch-analysis-pinyin-7.1.0.zip
  3. 同义词：

8.升级

https://www.cnblogs.com/AlienXu/p/11170023.html

9. 安全重启es 

1）禁用分片分配

关闭节点时，分配过程将等待 index.unassigned.node_left.delayed_timeout 1分钟（默认情况下为1分钟），然后开始将该节点上的分片复制到集群中的其他节点，这可能涉及大量I/O。由于该节点不久将要重新启动，因此该I/O是不必要的，通过在关闭节点之前禁用副本分配。

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

设置成primaries，索引的主分片会均分到集群的各个node，副本分片处于unassigined状态。

2）重启es

依次挨个重启一个节点，启动好了后再重启另外节点。

service elasticsearch restart

3）开启分片分配

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}
posted @ 2019-12-11 15:18  mjack  阅读(496)  评论(0)    收藏  举报