Skip to main content

Fluent Bit

Fluent Bit is a lightweight, and scalable logging and metrics processor and forwarder. It is one of the most popular choices in cloud and container environments. Fluent Bit agents can be configured to send logs to Parseable with HTTP output plugin and JSON output format.

This document explains how to set up Fluent Bit to ship logs to Parseable. We use Fluent Bit's Memory Metrics Input plugin as the source of logs and send the data to Parseable using the HTTP output plugin.

In this document we'll cover a simple Docker Compose based setup and a Kubernetes based deployment with Helm.

Docker Compose

Please ensure Docker Compose installed on your machine. Then run the following commands to set up Parseable and Fluent Bit.

mkdir parseable
cd parseable
docker-compose up -d

You can now access the Parseable dashboard on http://localhost:8000. You should see a log stream called fluentbitdemo populated with log data generated by the Memory Metrics Input plugin.


Install Fluent Bit

We use the official Fluent Bit Helm chart. But, we'll use a modified values.yaml file, that contains the configuration for Fluent Bit to send logs to Parseable.

helm repo add fluent
helm install fluent-bit fluent/fluent-bit --values values.yaml -n fluentbit --create-namespace

Let's take a deeper look at the Fluent Bit configuration in values.yaml. Here we use the kubernetes filter to enrich the logs with Kubernetes metadata. We then use the http output plugin to send logs to Parseable. Notice the Match section in the http output plugin. We use kube.* to match all logs from Kubernetes filter. With the header X-P-Stream fluentbitdemo, we tell Parseable to send the logs to the fluentbitdemo stream.

  filters: |
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On

outputs: |
Name http
Match kube.*
host parseable.parseable.svc.cluster.local
http_User admin
http_Passwd admin
format json
compress gzip
port 80
header Content-Type application/json
header X-P-META-meta1 value1
header X-P-TAG-tag1 value1
header X-P-Stream fluentbitdemo
uri /api/v1/ingest
json_date_key timestamp
json_date_format iso8601

Batching and Compression

Parseable supports batching and compressing the log data before sending it via HTTP POST. Fluent Bit supports this feature via the compress and buffer_max_size option. We recommend enabling both of these options to reduce the number of HTTP requests and to reduce the size of the HTTP payload.

Check logs in Parseable

If you've not already done so, port-forward Parseable service to access the dashboard with:

kubectl port-forward svc/parseable 8000:80 -n parseable

You can now check the Parseable server fluentbitdemo stream to see the logs from this setup.

Get Updates from Parseable

Subscribe to keep up with latest news, updates and new features on Parseable