Amazon EKS

Collect and forward logs from Amazon EKS (Elastic Kubernetes Service) to Parseable.

Overview

Integrate Amazon EKS with Parseable to:

Kubernetes Logs - Collect pod and container logs
AWS Integration - Native AWS ecosystem support
Scalable Collection - Handle large cluster deployments
Rich Metadata - Include Kubernetes context

Prerequisites

Amazon EKS cluster
kubectl configured
Helm (recommended)
Parseable instance accessible from EKS

Method 1: Fluent Bit DaemonSet

Deploy Fluent Bit as a DaemonSet for log collection.

Install with Helm

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  --create-namespace \
  --set config.outputs="[OUTPUT]\n    Name http\n    Match *\n    Host parseable.example.com\n    Port 8000\n    URI /api/v1/ingest\n    Format json\n    Header Authorization Basic YWRtaW46YWRtaW4=\n    Header X-P-Stream eks-logs"

Custom Values

Create fluent-bit-values.yaml:

config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

  outputs: |
    [OUTPUT]
        Name http
        Match *
        Host parseable.example.com
        Port 8000
        URI /api/v1/ingest
        Format json
        Header Authorization Basic YWRtaW46YWRtaW4=
        Header X-P-Stream eks-logs
        tls On
        tls.verify Off

tolerations:
  - operator: Exists
    effect: NoSchedule

Install with custom values:

helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  --create-namespace \
  -f fluent-bit-values.yaml

Method 2: AWS for Fluent Bit

Use AWS's optimized Fluent Bit distribution.

DaemonSet Manifest

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: amazon/aws-for-fluent-bit:latest
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: config
              mountPath: /fluent-bit/etc/
          env:
            - name: PARSEABLE_HOST
              value: "parseable.example.com"
            - name: PARSEABLE_AUTH
              valueFrom:
                secretKeyRef:
                  name: parseable-credentials
                  key: auth
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: config
          configMap:
            name: fluent-bit-config
      tolerations:
        - operator: Exists
          effect: NoSchedule

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Parsers_File  parsers.conf

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            cri
        Tag               kube.*
        Refresh_Interval  5
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        K8S-Logging.Parser  On

    [OUTPUT]
        Name            http
        Match           *
        Host            ${PARSEABLE_HOST}
        Port            8000
        URI             /api/v1/ingest
        Format          json
        Header          Authorization Basic ${PARSEABLE_AUTH}
        Header          X-P-Stream eks-logs
        tls             On

  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<dataset>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Method 3: OpenTelemetry Collector

Deploy OTel Collector for comprehensive telemetry.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: logging
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:latest
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: otel-collector-config
        - name: varlog
          hostPath:
            path: /var/log

IRSA Configuration

Use IAM Roles for Service Accounts:

eksctl create iamserviceaccount \
  --name fluent-bit \
  --namespace logging \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \
  --approve

Best Practices

Use DaemonSet - Ensure logs from all nodes
Add K8s Metadata - Include pod, namespace, labels
Configure Tolerations - Run on all nodes
Use IRSA - Secure credential management
Monitor Collector - Watch for backpressure

Troubleshooting

Missing Logs

Check Fluent Bit pod logs
Verify log paths are correct
Check RBAC permissions
Verify Parseable connectivity

High Resource Usage

Adjust buffer limits
Filter unnecessary logs
Increase flush interval

Next Steps

Configure Google GKE for multi-cloud
Set up alerts for Kubernetes events
Create dashboards for cluster monitoring

Amazon EKS

On this page