NXlog配置
NXlog文档：
http://nxlog.org/docs/nxlog-ce/nxlog-reference-manual.html
https://nxlog.org/documentation/nxlog-community-edition-reference-manual-v20928
Windows服务器上安装NXlog，然后配置为将Windows服务器上的事件日志发送到Linux syslog服务器（192.168.200.29:514）
<Extension _syslog>
Module xm_syslog #收集事件日志，所有的事件日志默认都被收集
</Extension>
<Input in> #配置input
Module im_msvistalog
# For windows 2003 and earlier use the following:
# Module im_mseventlog
</Input>
<Output out> #配置output
Module om_tcp #配置为使用tcp模式
Host 192.168.200.29 #远端syslog server IP
Port 514 #远端syslog server端口
Exec to_syslog_snare();
</Output>
<Route 1>
Path in => out #源输入对应输出
</Route>
配置完成后，如果nxlog出现如下错误：配置完成后，ERROR Couldn't read next event, corrupted eventlog?; The data is invalid.
则是因为windows8.1下windows事件日志种类过多导致，可以自定义需要保存的事件日志种类，设置如下：
<Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog ReadFromLast FALSE SavePos FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="Security">*</Select>\ #配置为只发送Security Log </Query>\ </QueryList> </Input> <Output out> Module om_tcp Host 192.168.200.29 Port 514 Exec to_syslog_snare(); </Output> <Route 1> Path in => out </Route>
配置为同时接收多个事件日志：
<Input in> Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog ReadFromLast FALSE SavePos FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> </Input>
Linux收集到的日志如下：
每条事件日志对应一个log
如果Linux上某个目录下文件过多，删除方式如下：ls | xargs -n 10 rm -fr ls
## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/docs/ ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog> Module xm_json #日志输出格式为json </Extension> <Input in> Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog ReadFromLast FALSE SavePos FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="Security">*</Select>\ #配置输入源为收集Security Log </Query>\ </QueryList> </Input> <Input systemin> #配置第二个输入源名为 systemin Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog ReadFromLast FALSE SavePos FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="System">*</Select>\ #配置收集System Log </Query>\ </QueryList> </Input> <Output out> #配置第一个输出方式，对应接收input in Module om_file File 'E:\logtest\seclog.json' #输出到seclog.json文件 Exec to_json(); </Output> <Output systemout> #配置第二个输出方式，对应接收input systemin Module om_file File 'E:\logtest\syslog.json' Exec to_json(); </Output> <Route 1> Path in => out #将第一个input in 对应到ouput out </Route> <Route 2> Path systemin => systemout #将第二个input systemin 对应到 output systemout </Route>
结果：
将两个input输出到同一个output：
<Route 1> Path in, systemin => out </Route>
按时间对output file进行rotate（未测试）：
define DIR C:\\logdir <Output out> Module om_file File "%DIR%\\test.log" <Schedule> Every 1 sec Exec out->rotate_to("%DIR%\\test."+ strftime(now(), "%Y%m%d%H%M%S")); </Schedule> </Output>
按大小对output file进行rotate：
<Extension _syslog> Module xm_json </Extension> <Input secin> Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog ReadFromLast FALSE SavePos FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> </Input> <Input systemin> Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog ReadFromLast FALSE SavePos FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="System">*</Select>\ </Query>\ </QueryList> </Input> <Output secout> Module om_file CreateDir TRUE sync FALSE File "e:\logtest\sec_" + $Hostname + "_" + month(now()) + ".json" Exec if secout->file_size() > 20M \ { \ $newfile = "e:\logtest\sec_" + $Hostname + "_" + strftime(now(), "%Y%m%d%H%M%S") + ".json"; \ secout->rotate_to($newfile); \ }; Exec to_json(); </Output> <Output systemout> Module om_file File 'E:\logtest\syslog.json' Exec to_json(); </Output> <Route 1> Path secin => secout </Route> <Route 2> Path systemin => systemout </Route>
output还可以编辑为如下，效果相同：
<Output secout> Module om_file CreateDir TRUE sync FALSE File "e:\logtest\sec_" + $Hostname + "_" + month(now()) + ".json" #此为当前正在写入的文件 <Exec> if secout->file_size() > 20M { $newfile = "e:\logtest\sec_" + $Hostname + "_" + strftime(now(), "%Y%m%d%H%M%S") + ".json"; secout->rotate_to($newfile); } </Exec> Exec to_json(); </Output>
输出如下：
收集IIS Log：
IIS仍为源格式
<Input IIS01> #按照不同站点分为多个Input Module im_file File "d:\\iislog\W3SVC2\\\\u_ex*.log" SavePos TRUE </Input> <Output IIS01>同其他</Output>
IIS log输出为json格式
<Extension w3c> Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> <Input iis01in> Module im_file File 'E:\IISLog\W3SVC1\u_ex*' SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $raw_event = to_json(); \ } </Input> <Output iis01out> Module om_file CreateDir TRUE sync FALSE
Exec $Hostname = 'server01'; File "e:\logtest\iis_" + $Hostname + "_" + month(now()) + ".json" <Exec> if iis01out->file_size() > 200M { $newfile = "e:\logtest\iis_" + $Hostname + "_" + strftime(now(), "%Y%m%d%H%M%S") + ".json"; iis01out->rotate_to($newfile); } </Exec> # Exec to_json(); </Output> <Route iis01> Path iis01in => iis01out </Route>
Using the Query directive
The im_msvistalog has a Query directive which can be used to specify an XML Query that gets passed to the Windows EventLog API in order to read only the selected events. The Windows Event Viewer can help construct such XML queries. The following example will only collect only process creation event records from the Sysmon source.
Query <QueryList> \
<Query Id="0">\
<Select Path="Microsoft-Windows-Sysmon/Operational">*[System[(EventID='1')]]</Select>\
</Query>\
</QueryList>
The event records filtered with the Query directive do not reach NXLog so this might be slightly more efficient than the next native NXLog filtering method.
Filtering with NXLog's log processing language
The NXLog log processing language is available for use by all modules and may be easier to write than the XML query syntax provided by the Windows EventLog API that the im_msvistalog exposes. The following NXLog style filter statement achieves the same as the XML Query above.
Exec if not ($Channel == 'Microsoft-Windows-Sysmon' and $EventID == 1) drop();
The following filtering rule will remove event records that are HTTP network connections to a specific server:
Exec if $SourceName == 'Microsoft-Windows-Sysmon' and $DestinationPort == 80 and $DestinationIp == 10.0.0.1 drop();
nxlog-ce-2.9.1504出现如下报错，可以替换成版本“nxlog-ce-2.9.1347”解决
报错：ERROR if-else failed at line 61, character 312 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 61, character 98 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 15, got 11 in input 'Exchange.asmx &CorrelationID=<empty>;&cafeReqId=bdbb53c3-5227-4601-8dbe-a5a7fb72c7b9; 80 xx\tuliqi 106.3.4.150 AppleExchangeWebServices/806.1+ExchangeSync/121 - 200 0 0 9'
