CircleCI

Collect and analyze CircleCI pipeline logs in Parseable for CI/CD observability.

Overview

Integrate CircleCI with Parseable to:

Pipeline Logs - Collect all job and workflow logs
Debug Failures - Quickly find and analyze failed builds
Track Performance - Monitor build times and trends
Audit Pipelines - Maintain compliance with log retention

Prerequisites

CircleCI project
Parseable instance accessible from CircleCI
CircleCI environment variables configured

Direct Log Shipping

Send logs directly from your CircleCI config.

Config.yml

version: 2.1

executors:
  default:
    docker:
      - image: cimg/base:stable

commands:
  send_to_parseable:
    parameters:
      status:
        type: string
      message:
        type: string
        default: ""
    steps:
      - run:
          name: Send to Parseable
          command: |
            curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
              -H "Authorization: Basic ${PARSEABLE_AUTH}" \
              -H "X-P-Stream: circleci-pipelines" \
              -H "Content-Type: application/json" \
              -d "[{
                \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
                \"project\": \"${CIRCLE_PROJECT_REPONAME}\",
                \"workflow\": \"${CIRCLE_WORKFLOW_ID}\",
                \"job\": \"${CIRCLE_JOB}\",
                \"build_num\": ${CIRCLE_BUILD_NUM},
                \"branch\": \"${CIRCLE_BRANCH}\",
                \"sha\": \"${CIRCLE_SHA1}\",
                \"username\": \"${CIRCLE_USERNAME}\",
                \"status\": \"<< parameters.status >>\",
                \"message\": \"<< parameters.message >>\"
              }]"
          when: always

jobs:
  build:
    executor: default
    steps:
      - checkout
      - send_to_parseable:
          status: "started"
          message: "Build started"
      - run:
          name: Build
          command: |
            echo "Building..."
            npm ci
            npm run build
      - send_to_parseable:
          status: "completed"
          message: "Build completed"

  test:
    executor: default
    steps:
      - checkout
      - run:
          name: Test
          command: npm test
      - send_to_parseable:
          status: "completed"
          message: "Tests passed"

workflows:
  build-and-test:
    jobs:
      - build
      - test:
          requires:
            - build

With Status Detection

commands:
  send_to_parseable_with_status:
    steps:
      - run:
          name: Send to Parseable
          command: |
            if [ "${CIRCLE_JOB_STATUS}" = "success" ]; then
              STATUS="success"
            else
              STATUS="failure"
            fi
            
            curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
              -H "Authorization: Basic ${PARSEABLE_AUTH}" \
              -H "X-P-Stream: circleci-pipelines" \
              -H "Content-Type: application/json" \
              -d "[{
                \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
                \"project\": \"${CIRCLE_PROJECT_REPONAME}\",
                \"job\": \"${CIRCLE_JOB}\",
                \"build_num\": ${CIRCLE_BUILD_NUM},
                \"status\": \"${STATUS}\"
              }]"
          when: always

Orb Integration

Create a reusable orb for Parseable logging.

Orb Definition

# .circleci/orbs/parseable.yml
version: 2.1

description: Send logs to Parseable

commands:
  log:
    parameters:
      status:
        type: string
      message:
        type: string
        default: ""
      dataset:
        type: string
        default: "circleci-pipelines"
    steps:
      - run:
          name: Log to Parseable
          command: |
            curl -s -X POST "${PARSEABLE_URL}/api/v1/ingest" \
              -H "Authorization: Basic ${PARSEABLE_AUTH}" \
              -H "X-P-Stream: << parameters.dataset >>" \
              -H "Content-Type: application/json" \
              -d "[{
                \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
                \"project\": \"${CIRCLE_PROJECT_REPONAME}\",
                \"workflow_id\": \"${CIRCLE_WORKFLOW_ID}\",
                \"job\": \"${CIRCLE_JOB}\",
                \"build_num\": ${CIRCLE_BUILD_NUM},
                \"branch\": \"${CIRCLE_BRANCH}\",
                \"sha\": \"${CIRCLE_SHA1}\",
                \"status\": \"<< parameters.status >>\",
                \"message\": \"<< parameters.message >>\"
              }]" || true
          when: always

Using the Orb

version: 2.1

orbs:
  parseable: your-org/parseable@1.0.0

jobs:
  build:
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      - parseable/log:
          status: "started"
      - run: npm ci && npm run build
      - parseable/log:
          status: "completed"

Webhook Integration

Use CircleCI webhooks for workflow events.

Webhook Receiver

const express = require('express');
const axios = require('axios');
const crypto = require('crypto');

const app = express();
app.use(express.json());

const CIRCLECI_SECRET = process.env.CIRCLECI_WEBHOOK_SECRET;
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;

app.post('/webhook', async (req, res) => {
  // Verify signature
  const signature = req.headers['circleci-signature'];
  // Verification logic here
  
  const event = req.body;
  
  const logEntry = {
    timestamp: new Date().toISOString(),
    event_type: event.type,
    project: event.project?.name,
    workflow_id: event.workflow?.id,
    workflow_name: event.workflow?.name,
    job_name: event.job?.name,
    status: event.job?.status || event.workflow?.status,
    branch: event.pipeline?.vcs?.branch,
    commit: event.pipeline?.vcs?.revision
  };
  
  try {
    await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, [logEntry], {
      headers: {
        'Authorization': `Basic ${PARSEABLE_AUTH}`,
        'X-P-Stream': 'circleci-webhooks',
        'Content-Type': 'application/json'
      }
    });
  } catch (error) {
    console.error('Error:', error);
  }
  
  res.status(200).json({ status: 'received' });
});

app.listen(3000);

Environment Variables

Configure in CircleCI project settings:

Variable	Description
`PARSEABLE_URL`	Parseable instance URL
`PARSEABLE_AUTH`	Base64 encoded `username:password`

Querying CircleCI Logs

-- Recent builds
SELECT timestamp, project, job, status, branch
FROM "circleci-pipelines"
ORDER BY timestamp DESC
LIMIT 100

-- Failed builds
SELECT timestamp, project, job, branch, message
FROM "circleci-pipelines"
WHERE status = 'failure'
ORDER BY timestamp DESC

-- Build success rate
SELECT 
  project,
  COUNT(*) as total,
  SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successful,
  ROUND(SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)::float / COUNT(*) * 100, 2) as success_rate
FROM "circleci-pipelines"
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY project

Best Practices

Use Orbs - Centralize logging logic
Log All Jobs - Track every job in workflow
Include Context - Add branch, commit, and user info
Handle Failures - Use when: always for logging steps

Next Steps

Set up alerts for failed builds
Create dashboards for CI/CD metrics
Configure GitHub Actions for comparison

CircleCI

On this page