Filebeat

Ship logs from Filebeat to Parseable using the HTTP output.

Overview

Integrate Filebeat with Parseable to:

Lightweight Collection - Minimal resource footprint
Rich Inputs - Collect from files, containers, cloud services
Modules - Pre-built configurations for common applications
Reliable Delivery - At-least-once delivery guarantee

Prerequisites

Filebeat installed
Parseable instance accessible
Log files or inputs configured

Filebeat Configuration

Basic Configuration

Create filebeat.yml:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log
    fields:
      source: filebeat
    fields_under_root: true

output.http:
  hosts: ["http://parseable:8000/api/v1/ingest"]
  method: "POST"
  headers:
    Authorization: "Basic YWRtaW46YWRtaW4="
    X-P-Stream: "filebeat-logs"
    Content-Type: "application/json"
  codec.format:
    string: '[%{[message]}]'

JSON Log Collection

For JSON-formatted logs:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.json
    json.keys_under_root: true
    json.add_error_key: true

processors:
  - timestamp:
      field: timestamp
      layouts:
        - '2006-01-02T15:04:05.000Z'
      test:
        - '2024-01-15T10:30:00.000Z'

output.http:
  hosts: ["http://parseable:8000/api/v1/ingest"]
  method: "POST"
  headers:
    Authorization: "Basic YWRtaW46YWRtaW4="
    X-P-Stream: "app-logs"
  batch.size: 100
  batch.timeout: 5s

Container Logs

Collect Docker container logs:

filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"

output.http:
  hosts: ["http://parseable:8000/api/v1/ingest"]
  method: "POST"
  headers:
    Authorization: "Basic YWRtaW46YWRtaW4="
    X-P-Stream: "container-logs"

Kubernetes Logs

For Kubernetes deployments:

filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

output.http:
  hosts: ["http://parseable:8000/api/v1/ingest"]
  method: "POST"
  headers:
    Authorization: "Basic YWRtaW46YWRtaW4="
    X-P-Stream: "k8s-logs"

Filebeat Modules

Use built-in modules for common applications:

Enable Module

filebeat modules enable nginx
filebeat modules enable mysql

Module Configuration

# modules.d/nginx.yml
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]

Running Filebeat

Docker

docker run -d \
  --name filebeat \
  -v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
  -v /var/log:/var/log:ro \
  docker.elastic.co/beats/filebeat:8.11.0

Docker Compose

version: '3.8'
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

Kubernetes DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    spec:
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:8.11.0
          volumeMounts:
            - name: config
              mountPath: /usr/share/filebeat/filebeat.yml
              subPath: filebeat.yml
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: filebeat-config
        - name: varlog
          hostPath:
            path: /var/log

Configuration Options

Parameter	Description
`output.http.hosts`	Parseable endpoint URL
`output.http.headers`	HTTP headers including auth
`output.http.batch.size`	Events per batch
`output.http.batch.timeout`	Max wait time for batch

Best Practices

Use Batching - Configure appropriate batch sizes
Add Metadata - Include host, container, or k8s metadata
Parse JSON - Use JSON parsing for structured logs
Monitor Filebeat - Enable monitoring endpoints
Handle Backpressure - Configure queue settings

Troubleshooting

Events Not Sending

Test Filebeat configuration: filebeat test config
Test output: filebeat test output
Check Filebeat logs
Verify Parseable endpoint is accessible

Duplicate Events

Check registry file location
Verify clean_removed setting
Check for multiple Filebeat instances

Next Steps

Configure Fluent Bit as alternative
Set up alerts for log patterns
Create dashboards for monitoring

Filebeat

On this page