Amazon ECS

Collect and forward logs from Amazon ECS (Elastic Container Service) to Parseable.

Overview

Integrate Amazon ECS with Parseable to:

Container Logs - Collect logs from ECS tasks
Centralized Observability - Unified view of container logs
AWS Integration - Native AWS log routing
Fargate Support - Works with both EC2 and Fargate

Prerequisites

AWS account with ECS
ECS cluster running
Parseable instance accessible from AWS
IAM permissions for log routing

Method 1: FireLens (Fluent Bit)

Use AWS FireLens with Fluent Bit sidecar for log routing.

Task Definition

{
  "family": "my-app",
  "containerDefinitions": [
    {
      "name": "log-router",
      "image": "amazon/aws-for-fluent-bit:latest",
      "essential": true,
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "config-file-type": "file",
          "config-file-value": "/fluent-bit/configs/parse-json.conf"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/firelens",
          "awslogs-region": "us-east-1",
          "awslogs-dataset-prefix": "firelens"
        }
      },
      "memoryReservation": 50
    },
    {
      "name": "app",
      "image": "my-app:latest",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "http",
          "Host": "parseable.example.com",
          "Port": "8000",
          "URI": "/api/v1/ingest",
          "Format": "json",
          "Header": "Authorization Basic YWRtaW46YWRtaW4=",
          "Header": "X-P-Stream ecs-logs"
        }
      }
    }
  ]
}

Custom Fluent Bit Config

Store in S3 or include in custom image:

[OUTPUT]
    Name http
    Match *
    Host parseable.example.com
    Port 8000
    URI /api/v1/ingest
    Format json
    Header Authorization Basic YWRtaW46YWRtaW4=
    Header X-P-Stream ecs-logs
    tls On
    tls.verify Off

Method 2: CloudWatch to Parseable

Route logs through CloudWatch, then forward to Parseable.

Task Definition with awslogs

{
  "containerDefinitions": [
    {
      "name": "app",
      "image": "my-app:latest",
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/my-app",
          "awslogs-region": "us-east-1",
          "awslogs-dataset-prefix": "app"
        }
      }
    }
  ]
}

Then use AWS CloudWatch integration to forward to Parseable.

Method 3: Sidecar Container

Deploy a log collector sidecar alongside your application.

Task Definition with Sidecar

{
  "family": "my-app-with-collector",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "my-app:latest",
      "essential": true,
      "mountPoints": [
        {
          "sourceVolume": "logs",
          "containerPath": "/var/log/app"
        }
      ]
    },
    {
      "name": "log-collector",
      "image": "fluent/fluent-bit:latest",
      "essential": false,
      "mountPoints": [
        {
          "sourceVolume": "logs",
          "containerPath": "/var/log/app",
          "readOnly": true
        }
      ],
      "environment": [
        {"name": "PARSEABLE_URL", "value": "http://parseable:8000"},
        {"name": "PARSEABLE_AUTH", "value": "YWRtaW46YWRtaW4="}
      ]
    }
  ],
  "volumes": [
    {
      "name": "logs",
      "host": {}
    }
  ]
}

IAM Permissions

Required IAM permissions for FireLens:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:CreateLogGroup",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::your-config-bucket/*"
    }
  ]
}

Best Practices

Use FireLens - AWS-native log routing
Add Metadata - Include task ID, cluster name
Configure Retries - Handle transient failures
Monitor Log Router - Watch sidecar health
Use Secrets - Store credentials in Secrets Manager

Troubleshooting

Logs Not Appearing

Check FireLens container logs
Verify Parseable endpoint is accessible
Check IAM permissions
Verify log configuration syntax

High Memory Usage

Configure buffer limits
Reduce batch sizes
Check for log volume spikes

Next Steps

Configure Amazon EKS for Kubernetes
Set up alerts for container metrics
Create dashboards for monitoring

Amazon ECS

On this page