logstash grok 正则匹配之nginx日志（自定义字段）

 

 

 

 

 

 

 

1、nginx access配置：

 

2、nginx原始日志：

192.168.1.148 - - [04/May/2017:16:34:18 +0800] "GET /api/Home/GetConfig?configMD5=&storesysno=0001 HTTP/1.1" 200 816 "-" "Apache-HttpClient/4.5.2 (Java/1.7.0_45)" - 0.005 0.005

3、注意一一匹配，注意空格

 

192.168.1.148  => $remote_addr
-  =>  -
- => $remote_user
[04/May/2017:16:34:18 +0800] =>[$time_local]
"GET /api/Home/GetConfig?configMD5=&storesysno=0001 HTTP/1.1" => "$request"
200=> $status
816 =>$body_bytes_sent
"-"=> "$http_referer"
"Apache-HttpClient/4.5.2 (Java/1.7.0_45)" => "$http_user_agent"
- =>$http_x_forwarded_for
0.005=> $request_time
0.005 =>$upstream_response_time';

 

4、 grok

%{IP:clientip} - (?:%{WORD:user_name}|-) \[%{HTTPDATE:timestamp}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status:int} %{NUMBER:body_sent:int} "(?:%{URI:referrer}|-)" "%{DATA:agent}" %{URIPARAM1:http_x_forwarded_fo} %{NUMBER:request_time:float} %{NUMBER:response_time:float}

某些字段为“-”，可能导致grokfailure，此时我们可以通过(?:%{XX:XX}|-)的方式进行匹配，即为空时显示“-”

实际配置环境：

 

      match => ["message","%{IP:clientip} - (?:%{WORD:user_name}|-) \[%{HTTPDATE:timestamp}\] \"%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:status:int} %{N
UMBER:body_sent:int} \"(?:%{URI:referrer}|-)\" \"%{DATA:agent}\" %{GREEDYDATA:http_x_forwarded_fo} %{NUMBER:request_time:float} %{NUMBER:response_time:float}"
   ] }

OR

match => ["message","%{IP:client_ip} - (?:%{WORD:user_name}|-) \[%{HTTPDATE:timestamp}\] \"(%{WORD:request_action}|-) (%{DATA:request}|-) HTTP/(%{NUMBER:http_version}|-)\" (%{NUMBER:status:int}|-) (%{NUMBER:body_sent:int}|-) \"(?:%{URI:referrer}|-)
\" \"(%{DATA:agent}|-)\" (%{GREEDYDATA:http_x_forwarded_for}|-) (%{NUMBER:request_time:float}|-) (%{NUMBER:response_time:float}|-) (%{IPORHOST:domain}|-) (?:%{HOSTPORT:upstream_addr}|-)"]

########################
NGINX_ERROR (?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage} 

 

ps：（不停报错，倒腾了半个小时）

 

1、match后整个用中括号[] 括起来。

 2、双引号""必须要转移

 

logstash2.2 官方解释：

logstash 5.4:

 

 

 

 

 

 

 

https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

http://grokconstructor.appspot.com/do/match#result

http://blog.csdn.net/yanggd1987/article/details/50486779