利用Filebeat+ELK 来收集.Net Core 微服务日志 (二）

一：建立ELK（这里使用docker 部署）

ELK 在Github 上自己找个docker-compose 来部署，大家喜欢什么版本自行处理

这里接着上一帐，配置好我们的 logstash.conf 文件即可：（目的是按我们的格式 要求，过滤Filebeat 上传的日志，在Kibana 上作展示） 

input {
     beats {
    port => 5044
  }
}

## Add your filters / logstash plugins configuration here
filter {
 if [type] == "filebeat-apps-log" {

    grok {
        match => {           
            "message" => "\[%{DATA:time}\]#%{DATA:ser_name}\#%{DATA:req_url}\#%{DATA:req_method}\#%{DATA:rep_param}\#\s*(?<err_msg>([\s\S]*))"
        }         
        remove_field => ["message"]
        remove_field => ["fields"]
        remove_field => ["input_type"]
        remove_field => ["tags"]
        remove_field => ["count"]
        remove_field => ["host"]
        remove_field => ["@version"]     
        remove_field => ["beat.name"]
        remove_field => ["beat.hostname"]
        remove_field => ["offset"]
    }

   }
}

 
output {
    elasticsearch {
        hosts => "elasticsearch:9200"
    }
}

 ELK 比较简单  logstash  Filter 在线调试工具：http://grok.qiexun.net/

大家可以在 INPUT 输入 ：

[2018-06-26 15:07:22.1562]#API001#/api/get#GET#?id=ad#   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at API001.Controllers.ValuesController.test(String id) in D:\code\study_demo\API001\Controllers\ValuesController.cs:line 26
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at API001.GlobalErrorHandlingMiddleware.<Invoke>d__3.MoveNext() in D:\code\study_demo\API001\GlobalErrorHandlingMiddleware.cs:line 28#

 

Pattern:

\[%{DATA:time}\]#%{DATA:ser_name}\#%{DATA:req_url}\#%{DATA:req_method}\#%{DATA:rep_param}\#\s*(?<err_msg>([\s\S]*))

 

出来的效果如下：