手动编写一个filebeta的module

手动build filebeta module

一、介绍

1.1 filebeta介绍

filebeat是一个elastic公司使用golang编写的一个收集日志的工具,基于beat编写. 已经集成了大多数主流服务日志模块(https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-modules.html).

1.2 本文介绍

本文将会手动build一个codis dashbaord的日志收集模块, 并介绍filebeat的使用

1.3 参考文档

官方文档

• 创建一个新的filebeat模块: https://www.elastic.co/guide/en/beats/devguide/7.x/filebeat-modules-devguide.html
• grok语法: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/grok-processor.html
• filebeat使用环境变量: https://www.elastic.co/guide/en/beats/filebeat/7.10/using-environ-vars.html

其他文档

• 如何创建 filebeat 应用模块: https://zhuanlan.zhihu.com/p/140077079
• grok官方Debug：http://grokdebug.herokuapp.com/
• golang build应用docker中无法启动: https://blog.csdn.net/u013276277/article/details/105797019
• Golang build 填坑笔记: https://blog.csdn.net/u013235478/article/details/105852353/
• Filebeat 模块与配置: https://www.cnblogs.com/cjsblog/p/9495024.html

二、编写一个module

# git clone https://github.com/elastic/beats.git   # beats代码量比较大, 网速慢的可以找个github加速
# cd beats/filebeat
# make create-module MODULE=codis    # 创建codis模块
# make create-fileset MODULE=codis FILESET=d_log    # 在codis模块中创建一个FILESET，命名为d_log
# vim  module/codis/d_log/ingest/pipeline.json
{
  "processors": [
    {
      "set": {
        "field": "event.ingested",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{LOGDATE:codis.date} %{TIME:codis.time} %{operAtionFILE:codis.Ationfile}:%{line:codis.LINE}: \\[%{LOGLEVEL:codis.LOGLEVEL}\\] %{GREEDYDATA} failed\n%{allData:codis.errorInfo}",
          "%{LOGDATE:codis.date} %{TIME:codis.time} %{operAtionFILE:codis.Ationfile}:%{line:codis.LINE}: \\[%{LOGLEVEL:codis.LOGLEVEL}\\] %{GREEDYDATA}:\n%{allData:codis.jsonInfo}",
          "%{LOGDATE:codis.date} %{TIME:codis.time} %{operAtionFILE:codis.Ationfile}:%{line:codis.LINE}: \\[%{LOGLEVEL:codis.LOGLEVEL}\\] sentinel-\\[%{URIHOST:codis.sentinelHost}\\]",
          "%{LOGDATE:codis.date} %{TIME:codis.time} %{operAtionFILE:codis.Ationfile}:%{line:codis.LINE}: \\[%{LOGLEVEL:codis.LOGLEVEL}\\] \\[%{BASE16NUM}\\] API call %{URIPATH:codis.URIPATH} from %{IP:codis.remote_ip}:%{PORT:codis.remote_port} \\[%{IP:codis.client_ip}\\]"
        ],
        "pattern_definitions": {
          "LOGDATE": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}",
          "operAtionFILE": ".*?go",
          "line": "\\d+",
          "LOGLEVEL": "([A-a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)",
          "PORT": "\\d+",
          "allData": "(.|\n)*"
        },
        "ignore_missing": true
      }
    },
    {
      "grok": {
        "field": "log.file.path",
        "patterns": [
          "%{DATA}codis-product/%{DATA:codis.name}/%{DATA}"
        ]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    }
  ]
}


# vim module/codis/d_log/config/d_log.yml
type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
tail_files: true
multiline:
  pattern: ^[[:^digit:]]
  negate: false
  match: after
  timeout: 3

max_bytes: 6291456000
harvester_buffer_size: 65536
scan_frequency: 10s
close_inactive: 5m

fields:
  REGION: ${REGION}  # ${会识别环境变量}
  CUSTOM_RUNTIME_ENV: ${CUSTOM_RUNTIME_ENV}
fields_under_root: true


# vim module/codis/d_log/manifest.yml
module_version: 1.0

var:
  - name: paths
    default:
      - /opt/codis-product/*/log/codis-dashboard.log.*
    os.darwin:
      - /opt/codis-product/*/log/codis-dashboard.log.*
    os.windows:
      - /opt/codis-product/*/log/codis-dashboard.log.*

ingest_pipeline: ingest/pipeline.json
input: config/d_log.yml

# vim filebeat.yml
filebeat.modules:
- module: codis
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0
setup.kibana:
output.elasticsearch:
  hosts: ${ESHOSTS}
  indices:
    - index: "codis-dashboard-%{+yyyy.MM.dd}"

# make create-fields MODULE=codis FILESET=d_log    # 创建filed字段
# make update
# go env -w CGO_ENABLED="0"  # 关闭cgo, 否则使用多阶段构建的容器在使用busybox 或 alpine 作为基础镜像时无法启动.
# make
# ./filebeat setup -modules codis -e  # 初始化
# ./filebeat -e  # 启动
# vim Dockerfile  # 不在docker中使用可以忽略
FROM elastic/filebeat:7.6.0
WORKDIR /usr/share/filebeat
COPY codis-filebeat/filebeat.yml ./filebeat.yml
COPY codis-filebeat/codis ./module/codis
COPY codis-filebeat/codis.yml.disabled ./modules.d/
FROM busybox
COPY --from=0 /usr/share/filebeat /filebeat
COPY codis-filebeat/filebeat /filebeat/
RUN chmod +x /filebeat/filebeat
WORKDIR /filebeat
ENTRYPOINT ["/filebeat/filebeat"]
CMD  ["-e", "--strict.perms=false"]