基于Kibana的可视化监控报警插件sentinl入门
sentinl是什么
- Kibi/Kibana Alert & Reporting App
- Watching your data, 24/7/365
sentinl是一个免费的kibana预警与报告插件，与付费软件X-Pack功能类似。
Some Examples for illustration:
- HIT COUNT PER HOUR
- QUESTION: How many hits does index X receive hourly?
- WATCHER: query index and return count of hits in last hour
- ACTION: Notify with number of Hits per hour
- METRIC THRESHOLDS
- QUESTION: Is any of my monitored metrics surpassing a certain value?
- WATCHER: query index and type for specific values, aggregated by an arbitrary field.
- ACTION: Notify with aggs bucket details every time a threshold is surpassed or spike anomaly detected.
- BLACKLISTS HITS
- QUESTION: Is any of my users trying to reach blacklisted destinations?
- WATCHER: query firewall logs comparing destination IPs to a blacklist.
- ACTION: Notify admin via email if any IP >= 10 matches returned
- FAILED LOGINS
- QUESTION: Are there recurring failure attempts authenticating users on my network?
- WATCHER: query active directory logs for login failures in last hour and compare to user index. .
- ACTION: Notify admin via webhook if >= 10 matches returned
- LEAK DETECTION (chain)
- QUESTION: Are there any public leaks about my data I was not aware of?
- WATCHER: query for user emails included in published leaks ingested from third parties.
- ACTION: Save hits in secondary result Index. Notify via email if leak was not known in a secondary Watcher
安装
./kibana-plugin install file:./sentinl-v6.0.1.zip
安装完成后，要重启kibana
fuser -n tcp 5601
ps -ef | grep node
kill -9 pid
./kibana &
使用步骤
使用包括5个步骤
- Step 1: New Watcher
give our Watcher a name and choose an execution frequency
- Step 2: Input Query
es的搜索与聚合
- Step 3: Condition
validate if the results received back are worth processing
语法与x-pack script condition语法类似
相当于过滤条件
"condition": {
"script": {
"script": "payload.hits.total>=1" //当报警条件为***出现的次数大于1
}
}
"condition": {
"script": {
"script": "payload.hits.hits[0]._source.responsetime > 0.01" // 检索条件 响应时间大于 0.01秒
}
}
- Step 4: Transform
Our data might need adjustments or post processing. Process our payload using a javascript expression/script
事后处理
- Step 5: Actions
Let's form a notification using the mustache templating language。
可以采用多种方式发送通知。
transform
How to Adapt or Post-Process data
Post Process事后的处理。
The transform script is the wild member of the family and can be used to inject simple or complex logic into the pipeline before delivery to actions using pure javascript.
From converting format types, through generating brand new payload keys and interpolating data, transform is the way up. The script expects a boolean condition to trigger actions. A false condition can be forced to stop the execution. BONUS: Transforms can be saved and used across Watchers! "transform": { "script": { "script": "payload.newvar = payload.aggs.some.values['95.0']" } }
action举例之邮件发送
kibana.yml
logging.verbose: true
sentinl:
settings:
email:
active: true
host: smtp.exmail.qq.com
ssl: false
report:
active: true
tmp_path: /tmp/
上面是官网的，下面是实践已OK
sentinl:
settings:
email:
active: true
user: tanyk@huawangtech.com
password: Dd@2016
host: smtp.exmail.qq.com
ssl: true
timeout: 10000
report:
active: true
tmp_path: /tmp/
先测试
mailx -S smtp=<smtp-server-address> -r <from-address> -s <subject> -v <to-address> < body.txt
yum -y install sendmail
yum install -y sendmail-cf
/etc/init.d/sendmail start
chkconfig sendmail on
yum install -y mailx
vim /etc/mail.rc（optional）
set from=tanyk@mail.com
set smtp=smtp.exmail.qq.com
set smtp-auth-user=tanyk@mail.com
set smtp-auth-password=******
set smtp-auth=login
set nss-config-dir="/etc/pki/nssdb/"
test
echo "This is the message body and contains the message" | mailx -v -r "tanyk@mail.com" -s "This is the subject" -S smtp="smtp.exmail.qq.com" -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user="tanyk@mail.com" -S smtp-auth-password="******" -S ssl-verify=ignore -S nss-config-dir="/etc/pki/nssdb/" tanyk@163.com
