Parseable as logging target for MinIO Audit & Debug logs
Learn how to generate audit logs from MinIO, ingest to Parseable and use Parseable to analyse and visualise the audit data.
As new architecture patterns like separation of compute and storage evolve, object storage is increasingly the first choice for applications to store primary (non-archival) data. This is in addition to the old use cases of archival and large blob storage.
MinIO is an open-source, AWS S3-compatible object storage platform, allowing developers to use one of the most common object-based storage options without vendor lock-in and with flexibility.
As production workloads move to MinIO, it’s critical for businesses to audit and track each and every access to their MinIO cluster. Additionally, debugging issues in a production cluster is critical for business continuity.
MinIO exposes both audit logs and its internal server logs via a simple REST API, that you can point to your logging infrastructure. This is perfectly suited to writing to external services optimized for logging, such as Parseable.
This post walks through a live setup of MinIO and Parseable on Kubernetes using helm charts, configuring the MinIO cluster to output audit and server logs to Parseable. Finally, it shows you how to build dashboards on Parseable using the MinIO audit logs. We use an example application that generates random text files of various sizes and saves them to MinIO while also writing logs to a Parseable instance. This allows you to benefit from features such as alerts and comprehensive analytics.
Start Parseable
The preferred way to start Parseable under Kubernetes is using the Helm chart. The Parseable documentation provides more detailed instructions.
Once the Parseable service is running, open the web UI http://localhost:8000, log in with the username and password defined in the parseable-env-secret
file (admin/admin unless you changed anything), and create two new log streams: minio_log
and minio_audit
using the + Create Stream
button.
Log streams are logical (and physical) collections of related log events. This MinIO example posts audit data to one log stream and error log data to another. While Parseable supports dynamic schema, splitting two fundamentally different data sets into different streams for access control, search, and other reasons makes sense.
Start MinIO
Download this Kubernetes YAML file that defines a namespace and pod for MinIO. There are a few values you need to change to suit your setup:
Update kubernetes.io/hostname to match the node running your Kubernetes cluster. For example, docker-desktop.
Create a base64-encoded string of the Parseable username and password for use as a basic authentication header and add it to the values for the
MINIO_LOGGER_WEBHOOK_AUTH_TOKEN_PARSEABLE
andMINIO_AUDIT_WEBHOOK_AUTH_TOKEN_PARSEABLE
environment variables. If you haven’t changed the username and password, this is "Basic YWRtaW46YWRtaW4=".
The MINIO_LOGGER_WEBHOOK_ENDPOINT_PARSEABLE
and MINIO_AUDIT_WEBHOOK_ENDPOINT_PARSEABLE
variables use the two respective log streams you created in the last step, directing the two sources of MinIO log data between the two log streams.
With those changes made, send the configuration to Kubernetes to start MinIO:
kubectl apply -f minio-parseable.yaml
``
And forward the local 9000 port to port 9090 of the MinIO pod:
```shell
kubectl port-forward pod/minio 9000 9090 -n minio-parseable
Open the MinIO Web UI http://localhost:9090 and log in with minioadmin
for the user name and password.
Then, open the Access Keys
section on the right-hand User
menu, and click Create access key +
. Note the Access Key
and Secret Ke
y values and click the Create
button.
Demo Application
As MinIO acts as Amazon S3-compatible storage, you need an application that generates files to connect to it.
Download this JavaScript file that uses the faker library to generate a random number of text files on each run and PUTs
them into a MinIO bucket called js-test-bucket
(which is created on the first run), deleting the local copy of the file.
After this, it GETs
the same file and saves it elsewhere on the local file system.
These two operations simulate a more typical production application so that MinIO sends a reasonable mix of API data to Parseable
Before running the code in the Minio.Client
object, change the endpoint property to match yours and add the access key values created earlier to the accessKey
and secretKeyProperties
.
Run the application several times and open the minio_audit
stream in the Parseable Web UI. Unless the MinIO pod encounters any issues, the minio_log
log stream is probably empty. You can see the JSON structure of each audit entry in the MinIO documentation.
The JSON structure has several objects, which Parseable flattens, for example, api
> name
as api_name
.
Dashboards
The Parseable log stream view offers some ways of analyzing log data, but for quick and convenient views, Parseable also offers user-created dashboards and tiles in the dashboard that can use SQL queries and a variety of visualization options.
Find and create dashboards from the Dashboards tab. Click the + New Dashboard button, give it a name, and set the time range to the last 24 hours.
This example creates four dashboards, showing the following:
The largest files uploaded.
Operation count.
Bucket activity.
Data transferred.
You can find a JSON export of all the tiles and the dashboard in the dashboards folder of the example repository that you can import instead of creating a dashboard from scratch.
You can export dashboards and tiles using the share button on each tile or dashboard. For tiles you can also export PNGs, CSVs, and JSON files of the rendered data.
Largest Files Uploaded
Click the + Add Tile
button, give it a name, and select the minio_audit
stream. Add the following SQL into the Query field, which queries the fields that contain the file name and size, filtering to any that are larger than 20kb, and sorting the result by the size.
Select api_object, api_rx from minio_audit where api_rx > 20000 order by api_rx desc LIMIT 10
Click the Edit Visualization
button. Set the Type to Bar Chart
, the X-Axis to api_object
, and the Y-Axis to api_rx
. Click Done
and then Save Changes
.
Operation Count
Click the + Add Tile
button, give it a name, and select the minio_audit
stream.
Add the following SQL into the Query field, which queries the field that contains the API call, groups them, and returns the total of each group.
select count(*) as count,api_name from minio_audit group by api_name LIMIT 100
Click the Edit Visualization
button. Set the Type to Area Chart
, the X-Axis to api_name
and the Y-Axis to count
. Click Done
and then Save Changes
.
Bucket Activity
Click the + Add Tile
button, give it a name, and select the minio_audit
stream.
Add the following SQL into the Query field, which queries the fields that contain the bucket name and the API operation, returning the count for each operation on the bucket.
select count(*) as count,api_bucket,api_name from minio_audit group by api_bucket,api_name LIMIT 100
Click the Edit Visualization
button. Set the Type to Table
. Click Done
and then Save Changes
.
Data Transferred
Click the + Add Tile
button, give it a name, and select the minio_audit
stream.
Add the following SQL into the Query field, which queries the field that contains the API name and the size for each operation, returning the total size for each operation.
select sum(api_rx) as sum,api_name from minio_audit group by api_name LIMIT 100
Click the Edit Visualization
button. Set the Type to Line Chart
, the X-Axis to api_name
and the Y-Axis to sum
. Click Done
and then Save Changes
.
Summary
MinIO is a powerful and flexible option for object storage that avoids vendor lock-in. As object stores become a more common and important component of a modern application stack, it’s important to ensure you have comprehensive audit logs of the storage as well as quick to see window into the logs. This post covered outputting MinIO audit and error logs to Parseable and creating four dashboards to highlight key information on the log data.