Logstash Pipeline Config Builder

A Logstash pipeline from input to Elasticsearch

A Logstash pipeline is three blocks: where events come from, how they are transformed, and where they go. This tool assembles a common shape — Beats in, Grok and Date and Mutate filters, Elasticsearch out — into a ready pipeline.conf.

Use this builder when you need a production-ready starting point for a Beats-to-Elasticsearch pipeline and do not want to hand-write the block structure from scratch. Copy the output into your conf.d directory and adjust the Grok pattern for your log format.

How it works

The input block opens a beats listener on a port that Filebeat ships to over the lumberjack protocol. Events then flow through the filter block in written order, which is why sequence matters. grok matches a pattern against the raw message field and extracts named fields; for example %{COMBINEDAPACHELOG} turns an access-log line into status, response time, and client IP. Because the timestamp only exists after parsing, the date filter runs next, parsing a field with a Joda format string and writing the result into @timestamp so dashboards align with when events actually happened rather than when Logstash saw them. The mutate filter then tidies up, using remove_field to drop the now-redundant raw message and noisy host fields.

The output block sends each event to Elasticsearch. The index setting uses %{+YYYY.MM.dd} date math to route events into daily indices, which keeps retention manageable since an entire day can be deleted as one index.

Tips and example

Keep Grok before Date, and validate patterns in Kibana’s Grok Debugger before deploying — a mismatch tags events with _grokparsefailure and leaves message unparsed. A minimal pipeline:

input { beats { port => 5044 } }
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] target => "@timestamp" }
}
output {
  elasticsearch {
    hosts => [ "http://localhost:9200" ]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Drop the file into the Logstash conf.d directory and reload; pipeline reloads are supported without a full restart when config.reload.automatic is on.

Common Grok patterns for the filter block

The most useful built-in patterns are:

Build custom patterns with named captures: %{PATTERN:field_name}. If a line does not match, Logstash tags the event with _grokparsefailure and leaves message intact — useful for catching format drift without dropping data.

Pipeline performance considerations

A single-node Logstash instance can typically handle thousands of events per second for simple pipelines, but a few choices sharpen throughput:

• Pipeline workers (pipeline.workers) should match available CPU cores; the default is the number of logical cores.
• Batch size (pipeline.batch.size, default 125) controls how many events are fetched per worker cycle before filters run. Raising it improves throughput but increases memory use and latency.
• Persistent queues (queue.type: persisted) protect against data loss on crash by writing incoming events to disk before they enter the filter stage. Essential for compliance workloads.
• Dead letter queue (DLQ) captures events that fail at the output stage (for example malformed documents that Elasticsearch rejects), so nothing is silently discarded.

Deployment checklist

Before going to production with the generated pipeline:

1. Validate your Grok pattern against ten representative real log lines using Kibana’s Grok Debugger or logstash --config.test_and_exit.
2. Confirm the Date filter’s match format string matches your actual timestamp (for example dd/MMM/yyyy:HH:mm:ss Z for Apache) and that the target field is @timestamp.
3. Review remove_field — ensure you are not accidentally dropping a field you need for correlation downstream.
4. Test the index pattern in Kibana: GET /_cat/indices?v should show a daily index like app-logs-2026.06.21 after the first event lands.