Graylog处理docker容器的多行日志之过程记录

docker容器虽然支持gelf日志驱动，却不支持合并多行日志为1个message，详情见 log driver should support multiline · Issue #22920 · moby/moby · GitHub

，这导致在graylog查看java应用的报错日志时非常不方便。

 

解决思路：用logstash处理后再发给graylog。

 

docker安装logstash

将 /usr/share/logstash/conf.d/ 目录映射出来，方便编辑配置文件

mkdir -p  /opt/logstash/conf.d/

vi /opt/logstash/logstash.yml

logstash.yml内容如下：

path.config: /usr/share/logstash/conf.d/*.conf
path.logs: /var/log/logstash

 

vi /opt/logstash/conf.d/test.conf

input {
    file{
        path => "/usr/share/logstash/conf.d/test.log"
        start_position => "beginning"
        type=>"runtimelog"
                codec=> multiline {
                        pattern => "^%{TIMESTAMP_ISO8601} "
                        negate => true
                        what => "previous"
                }
	}
}

filter {}

output {
	stdout {
        	codec => rubydebug      
	}
}

 

docker run -d -p 5044:5044 -p 5045:5045 -p 12200:12200/udp --name logstash -v /opt/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml -v /opt/logstash/conf.d/:/usr/share/logstash/conf.d/ logstash:7.16.1

 

进入容器内安装插件

logstash-plugin install logstash-output-gelf
logstash-plugin install logstash-input-gelf

 安装完插件再添加相关conf

vi /opt/logstash/conf.d/app.conf

input {
    gelf {
	port =>12200
	host => "0.0.0.0"
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => "previous"
        }
    }
}

filter {}

output {
    gelf {
	host => "172.17.0.1"
        port => 12201
        protocol => "UDP"
    }
}

测试结果：input类型为file时，multiline编码正常，input类型为gelf时，无效...

参考链接：docker - logstash-5.x gelf input multiline codec doesn't work - Stack Overflow

 

既然logstash行不通，换成fluent-bit试试：

mkdir -p /opt/fluent-bit/

vi /opt/fluent-bit/fluent-bit.conf

[INPUT]
    name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M
    #Multiline	      On
    #Parser_Firstline  multiline_pattern

[OUTPUT]
    Name                    gelf
    Match                   *
    Host                    172.17.0.1
    Port                    12201
    Mode                    udp
    Gelf_Short_Message_Key  log

docker run -d --name fluent -p 24224:24224 -p 24224:24224/udp -v /opt/fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:1.8

很遗憾，fluent-bit的input类型为forward时，也不支持Multiline处理... unknown configuration property 'Multiline'. The following properties are allowed: unix_path, buffer_chunk_size, and buffer_max_size.

参考链接：logging - Does Fluent Bit Input plugin "forward" support multi-line logs processing? - Stack Overflow

 

既然logstash和fluent都不能很好的实现，又有了以下想法：

fluentd将接收到的udp数据保存到文本文件，Logstash处理后再发给Graylog服务器。

然而且不说这个方法有多低效，实际操作后发现fluentd保存下来的文件也不是日至原始的格式了，Logstash并不方便处理。

 

又想到一个方法：自行用golang开发一个软件，实时接收udp数据包，判断是否一个完整的日志，是的话直接转发给Graylog，不是的话存在内存里等待接收完全再将不完整的日志拼接起来，然后发给Graylog。

然而，测试发现golang程序接收到的udp数据包全都是乱码的，一开始以为是不是gelf用的不是utf-8编码，但是看到英文字母都成了乱码，应该跟编码无关。

最终发现了-log-opt有一项是关于是否开启压缩的，-log-opt gelf-compression-type=none  设置完毕，终于不再乱码了...

 

随手敲了几行golang代码，测试发现基本能满足需求，不过没有区分不同tag的消息，不同容器同时发来消息可能将不同容器的gelf信息合并成同一条。代码待完善...

package main

import (
	"encoding/json"
	"fmt"
	"net"
	"time"
)

//用于定义是否有未发的消息
var HasOld bool

type GelfModel struct {
	Version       string    `json:"version"`
	Host          string    `json:"host"`
	ShortMessage  string    `json:"short_message"`
	Timestamp     float64   `json:"timestamp"`
	Level         int       `json:"level"`
	Command       string    `json:"_command"`
	ContainerID   string    `json:"_container_id"`
	ContainerName string    `json:"_container_name"`
	Created       time.Time `json:"_created"`
	ImageID       string    `json:"_image_id"`
	ImageName     string    `json:"_image_name"`
	Tag           string    `json:"_tag"`
}

func SendToGelf(data []byte) {
	conn, err := net.Dial("udp", "127.0.0.1:12201")
	if err != nil {
		fmt.Println("net.Dial err:", err)
		return
	}
	defer conn.Close()

	conn.Write(data)
	//fmt.Println(len(data))
}

func main() {
	HasOld = false
	udpConn, _ := net.ListenUDP("udp", &net.UDPAddr{
		IP:   net.IPv4(0, 0, 0, 0),
		Port: 12200,
	})
	defer udpConn.Close()

	fmt.Println("udp listening ... ")

	gelfModel := &GelfModel{}

	for {
		var data [102400]byte
		n, _, _ := udpConn.ReadFromUDP(data[:]) // 接收数据
		gelfModel_ := &GelfModel{}
		json.Unmarshal(data[:n], &gelfModel_)
		shortMessage := gelfModel_.ShortMessage
		if n > 10 && shortMessage[2] == ':' && shortMessage[4] == ':' && shortMessage[6] == '.' { //判断是否xx:xx:xx.xxx开头
			if !HasOld { //没有的话直接发当前消息
				SendToGelf(data[:])
				HasOld = false
			} else { //有的话发之前的消息再发当前消息
				jsons, _ := json.Marshal(gelfModel) //转换成JSON返回的是byte[]
				SendToGelf(jsons)
				//fmt.Println(string(jsons))
				HasOld = false
				SendToGelf(data[:])
				gelfModel.ShortMessage = ""
			}
		} else {
			HasOld = true
			gelfModel.ShortMessage += shortMessage + "\n"
			gelfModel.Version = gelfModel_.Version
			gelfModel.Host = gelfModel_.Host
			gelfModel.Timestamp = gelfModel_.Timestamp
			gelfModel.Level = gelfModel_.Level
			gelfModel.Command = gelfModel_.Command
			gelfModel.ContainerID = gelfModel_.ContainerID
			gelfModel.ContainerName = gelfModel_.ContainerName
			gelfModel.Created = gelfModel_.Created
			gelfModel.ImageID = gelfModel_.ImageID
			gelfModel.ImageName = gelfModel_.ImageName
			gelfModel.Tag = gelfModel_.Tag
		}
	}
}