logstash
logstash作为数据搜集器，主要分为三个部分：input->filter->output 作为pipeline的形式进行处理,支持复杂的操作，如发邮件等
input配置数据的输入和简单的数据转换
filter配置数据的提取，一般使用grok
output配置数据的输出和简单的数据转换
运行：logstash -f /etc/logstash.conf
-f 指定配置文件
-e 只在控制台运行
具体的配置见官网
https://www.elastic.co/products/logstash
Centralize, Transform & Stash Your Data
input
|
Plugin |
Description |
Github repository |
|
Receives events from the Elastic Beats framework |
||
|
Streams events from CouchDB’s |
||
|
Reads query results from an Elasticsearch cluster |
||
|
Streams events from files |
||
|
Reads GELF-format messages from Graylog2 as events |
||
|
Generates random log events for test purposes |
||
|
Reads metrics from the |
||
|
Generates heartbeat events for testing |
||
|
Receives events over HTTP or HTTPS |
||
|
Decodes the output of an HTTP API into events |
||
|
Creates events from JDBC data |
||
|
Reads events from a Kafka topic |
||
|
Reads events over a TCP socket from a Log4j |
||
|
Receives events using the Lumberjack protocl |
||
|
Pulls events from a RabbitMQ exchange |
||
|
Reads events from a Redis instance |
||
|
Streams events from files in a S3 bucket |
||
|
Pulls events from an Amazon Web Services Simple Queue Service queue |
||
|
Reads events from standard input |
||
|
Reads syslog messages as events |
||
|
Reads events from a TCP socket |
||
|
Reads events from the Twitter Streaming API |
||
|
Reads events over UDP |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
|
Plugin |
Description |
Github repository |
|
Pulls events from the Amazon Web Services CloudWatch API |
||
|
Retrieves watchdog log events from Drupal installations with DBLog enabled |
||
|
Pulls events from the Windows Event Log |
||
|
Captures the output of a shell command as an event |
||
|
Reads Ganglia packets over UDP |
||
|
Pushes events to a GemFire region |
||
|
Reads events from a GitHub webhook |
||
|
Streams events from the logs of a Heroku app |
||
|
Reads mail from an IMAP server |
||
|
Reads events from an IRC server |
||
|
Retrieves metrics from remote Java applications over JMX |
||
|
Receives events through an AWS Kinesis stream |
||
|
Captures the output of command line tools as an event |
||
|
Streams events from a long-running command pipe |
||
|
Receives facts from a Puppet server |
||
|
Receives events from a Rackspace Cloud Queue service |
||
|
Receives RELP events over a TCP socket |
||
|
Captures the output of command line tools as an event |
||
|
Creates events based on a Salesforce SOQL query |
||
|
Creates events based on SNMP trap messages |
||
|
Creates events based on rows in an SQLite database |
||
|
Creates events received with the STOMP protocol |
||
|
Reads events over a UNIX socket |
||
|
Reads from the |
||
|
Reads events from a websocket |
||
|
Creates events based on the results of a WMI query |
||
|
Receives events over the XMPP/Jabber protocol |
||
|
Reads Zenoss events from the fanout exchange |
||
|
Reads events from a ZeroMQ SUB socket |
filter
|
Plugin |
Description |
Github repository |
|
Aggregates information from several events originating with a single task |
||
|
Replaces field values with a consistent hash |
||
|
Parses comma-separated value data into individual fields |
||
|
Parses dates from fields to use as the Logstash timestamp for an event |
||
|
Computationally expensive filter that removes dots from a field name |
||
|
Extracts unstructured event data into fields using delimiters |
||
|
Performs a standard or reverse DNS lookup |
||
|
Drops all events |
||
|
Fingerprints fields by replacing values with a consistent hash |
||
|
Adds geographical information about an IP address |
||
|
Parses unstructured event data into fields |
||
|
Parses JSON events |
||
|
Parses key-value pairs |
||
|
Merges multiple lines into a single event |
||
|
Performs mutations on fields |
||
|
Executes arbitrary Ruby code |
||
|
Sleeps for a specified time span |
||
|
Splits multi-line messages into distinct events |
||
|
Parses the |
||
|
Throttles the number of events |
||
|
Replaces field contents based on a hash or YAML file |
||
|
Decodes URL-encoded fields |
||
|
Parses user agent strings into fields |
||
|
Adds a UUID to events |
||
|
Parses XML into fields |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
|
Plugin |
Description |
Github repository |
|
Performs general alterations to fields that the |
||
|
Checks IP addresses against a list of network blocks |
||
|
Applies or removes a cipher to an event |
||
|
Duplicates events |
||
|
Collates events by time or count |
||
|
Calculates the elapsed time between a pair of events |
||
|
Copies fields from previous log events in Elasticsearch to current events |
||
|
Stores environment variables as metadata sub-fields |
||
|
Extracts numbers from a string |
||
|
Removes special characters from a field |
||
|
Serializes a field to JSON |
||
|
Adds arbitrary fields to an event |
||
|
Takes complex events containing a number of metrics and splits these up into multiple events, each holding a single metric |
||
|
Aggregates metrics |
||
|
Parse OUI data from MAC addresses |
||
|
Prunes event data based on a list of fields to blacklist or whitelist |
||
|
Strips all non-punctuation content from a field |
||
|
Checks that specified fields stay within given size or length limits |
||
|
Replaces the contents of the default message field with whatever you specify in the configuration |
||
|
Takes an existing field that contains YAML and expands it into an actual data structure within the Logstash event |
||
|
Sends an event to ZeroMQ |
output
Elastic supported plugins
These plugins are maintained and supported by Elastic.
|
Plugin |
Description |
Github repository |
|
Writes events to disk in a delimited format |
||
|
Stores logs in Elasticsearch |
||
|
Sends email to a specified address when output is received |
||
|
Writes events to files on disk |
||
|
Writes metrics to Graphite |
||
|
Sends events to a generic HTTP or HTTPS endpoint |
||
|
Writes events to a Kafka topic |
||
|
Sends events using the |
||
|
Pushes events to a RabbitMQ exchange |
||
|
Sends events to a Redis queue using the |
||
|
Sends Logstash events to the Amazon Simple Storage Service |
||
|
Prints events to the standard output |
||
|
Writes events over a TCP socket |
||
|
Sends events over UDP |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
|
Plugin |
Description |
Github repository |
|
Sends annotations to Boundary based on Logstash events |
||
|
Sends annotations to Circonus based on Logstash events |
||
|
Aggregates and sends metric data to AWS CloudWatch |
||
|
Sends events to DataDogHQ based on Logstash events |
||
|
Sends metrics to DataDogHQ based on Logstash events |
||
|
Stores logs in Elasticsearch using the |
||
|
Runs a command for a matching event |
||
|
Writes metrics to Ganglia’s |
||
|
Generates GELF formatted output for Graylog2 |
||
|
Writes events to Google BigQuery |
||
|
Writes events to Google Cloud Storage |
||
|
Sends metric data on Windows |
||
|
Writes events to HipChat |
||
|
Writes metrics to InfluxDB |
||
|
Writes events to IRC |
||
|
Writes strutured JSON events to JIRA |
||
|
Pushes messages to the Juggernaut websockets server |
||
|
Sends metrics, annotations, and alerts to Librato based on Logstash events |
||
|
Ships logs to Loggly |
||
|
Writes metrics to MetricCatcher |
||
|
Writes events to MongoDB |
||
|
Sends passive check results to Nagios |
||
|
Sends passive check results to Nagios using the NSCA protocol |
||
|
Sends logstash events to New Relic Insights as custom events |
||
|
Writes metrics to OpenTSDB |
||
|
Sends notifications based on preconfigured services and escalation policies |
||
|
Pipes events to another program’s standard input |
||
|
Sends events to a Rackspace Cloud Queue service |
||
|
Creates tickets using the Redmine API |
||
|
Writes events to the Riak distributed key/value store |
||
|
Sends metrics to Riemann |
||
|
Sends events to Amazon’s Simple Notification Service |
||
|
Stores and indexes logs in Solr |
||
|
Pushes events to an Amazon Web Services Simple Queue Serice queue |
||
|
Sends metrics using the |
||
|
Writes events using the STOMP protocol |
||
|
Sends events to a |
||
|
Sends Logstash events to HDFS using the |
||
|
Publishes messages to a websocket |
||
|
Posts events over XMPP |
||
|
Sends events to a Zabbix server |
||
|
Writes events to a ZeroMQ PUB socket |
