Logstash5.3借助临时字段修改@timestamp为北京时间，方便按天生成output文件

$more config/first-pipeline.conf
input {
    beats {
        port => "5044"
    }
}
filter {
    if [type] == "speech" {
       ruby { 
            code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))" 
        }
    }

    if [type] == "speech-en" {
       ruby { 
            code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))" 
       }

    }
 
    if [type] == "client-agent" {
        ruby { 
            code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))" 
        }

    }

    if [type] == "client-agent-en" {
        ruby { 
            code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))" 
        }

    }

    if [type] == "session-manager" {
        ruby { 
            code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))" 
        }
    }
}
output {
    stdout {
        codec => rubydebug
    }
    file{
        codec => line {format => "%{message}"}
        path => "/home/baoshan/elk/data/logstash/%{type}.%{+YYYYMMdd}"
    }
#    elasticsearch {
#        hosts => ["test41:9200","test42:9200","test43:9200"]
#        index => "%{hostabc}"
#        document_type => "%{hostabc}"
#        #protocol: "http"
#        flush_size => 100
#        idle_flush_time => 10
#        user => "elastic"
#        password => "baoshan"
#    }
}

核心代码为ruby中的code语句。（感觉这是最笨的方法，继续探求精简的办法。。。。）

 

特么的找了一天了，现在大部分都还不是logstash5.x的

下面这种方法试烂了都不管用，不知哪里不对，还请高手指教：

    grok {
            match => { 
                "message" => "time%{NUMBER:timestamp}id%{UUID:sn}asr%{NOTSPACE:asr}nlp%{NOTSPACE:nlp}domain%{NOTSPACE:domain}intent%{NOTSPACE:intent}" 
            } 
        }
        date {
            match => ["timestamp", "UNIX_MS"]   #因为我的日志时间戳为UNIX时间戳，毫秒级，后来发现这个时间戳硬生生被ELK改成了UTC时间
            target => "@timestamp"
            locale => "en"
            timezone => "+00:00"
        }

所以有了下面的配置

改配置文件包括两个知识点

1. 不可见字符（ctrl+A，ctrl+B）grok的方法

2. logstash时间戳@timestamp修改为日志中时间字段的方法 

input {
    beats {
        port => "5044"
    }
}
filter {
        grok {
            match => [ # 此处的^A为vim下的CTRL+A
                "message", "time\^B%{INT:timestamp}\^Aid\^B%{NOTSPACE:sn}\^Aasr\^B%{NOTSPACE:asr}\^Anlp\^B%{DATA:nlp}\^Adomain\^B%{JAVACLASS:domain}\^Aintent\^B%{NOTSPACE:intent}"
            ]
        }
        date {
            match => ["timestamp", "UNIX_MS"]
            target => "@timestamp"
        }
        ruby {
            code => "event.set('temp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('temp'))"
        }
}
output {
#    stdout { codec => rubydebug }
    file {
        codec => line {format => "%{message}"}
        path => "/home/admin/data/speech/speech.log.%{+YYYYMMdd}"
    }
    file {
        codec => line {format => "%{+YYYY-MM-dd HH:mm:ss}^A%{sn}^A%{asr}^A%{nlp}^A%{domain}^A%{intent}"}
        path => "/home/admin/data/speech/speech%{+YYYY-MM-dd}"
    }
}

 

各位高手，如果有更好的方法，还请指教