GitLab CI

Collect and analyze GitLab CI/CD pipeline logs in Parseable for comprehensive observability.

Overview

Integrate GitLab CI with Parseable to:

Centralize Pipeline Logs - Collect all CI/CD logs in one place
Debug Failures - Quickly find and analyze failed jobs
Track Performance - Monitor pipeline times and trends
Audit Pipelines - Maintain compliance with log retention

Prerequisites

GitLab repository with CI/CD enabled
Parseable instance accessible from GitLab runners
GitLab CI/CD variables configured

Method 1: Direct Log Shipping

Send logs directly from your .gitlab-ci.yml.

Basic Configuration

variables:
  PARSEABLE_URL: ${PARSEABLE_URL}
  PARSEABLE_AUTH: ${PARSEABLE_AUTH}

stages:
  - build
  - test
  - deploy

.send_to_parseable: &send_to_parseable
  - |
    curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
      -H "Authorization: Basic ${PARSEABLE_AUTH}" \
      -H "X-P-Stream: gitlab-pipelines" \
      -H "Content-Type: application/json" \
      -d "[{
        \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
        \"project\": \"${CI_PROJECT_PATH}\",
        \"pipeline_id\": ${CI_PIPELINE_ID},
        \"job_id\": ${CI_JOB_ID},
        \"job_name\": \"${CI_JOB_NAME}\",
        \"stage\": \"${CI_JOB_STAGE}\",
        \"ref\": \"${CI_COMMIT_REF_NAME}\",
        \"sha\": \"${CI_COMMIT_SHA}\",
        \"author\": \"${GITLAB_USER_LOGIN}\",
        \"status\": \"${1}\",
        \"message\": \"${2}\"
      }]"

build:
  stage: build
  script:
    - echo "Building..."
    - npm ci
    - npm run build
  after_script:
    - |
      if [ "$CI_JOB_STATUS" == "success" ]; then
        STATUS="success"
        MESSAGE="Build completed successfully"
      else
        STATUS="failure"
        MESSAGE="Build failed"
      fi
      curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
        -H "Authorization: Basic ${PARSEABLE_AUTH}" \
        -H "X-P-Stream: gitlab-pipelines" \
        -H "Content-Type: application/json" \
        -d "[{
          \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
          \"project\": \"${CI_PROJECT_PATH}\",
          \"pipeline_id\": ${CI_PIPELINE_ID},
          \"job_id\": ${CI_JOB_ID},
          \"job_name\": \"${CI_JOB_NAME}\",
          \"stage\": \"${CI_JOB_STAGE}\",
          \"ref\": \"${CI_COMMIT_REF_NAME}\",
          \"sha\": \"${CI_COMMIT_SHA}\",
          \"author\": \"${GITLAB_USER_LOGIN}\",
          \"status\": \"${STATUS}\",
          \"message\": \"${MESSAGE}\"
        }]"

test:
  stage: test
  script:
    - npm test
  after_script:
    - |
      STATUS="${CI_JOB_STATUS}"
      curl -X POST "${PARSEABLE_URL}/api/v1/ingest" \
        -H "Authorization: Basic ${PARSEABLE_AUTH}" \
        -H "X-P-Stream: gitlab-pipelines" \
        -H "Content-Type: application/json" \
        -d "[{
          \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
          \"project\": \"${CI_PROJECT_PATH}\",
          \"pipeline_id\": ${CI_PIPELINE_ID},
          \"job_id\": ${CI_JOB_ID},
          \"job_name\": \"${CI_JOB_NAME}\",
          \"stage\": \"${CI_JOB_STAGE}\",
          \"status\": \"${STATUS}\"
        }]"

Reusable Template

Create a reusable template for consistent logging:

# templates/parseable.yml
.parseable_logging:
  after_script:
    - |
      STATUS="${CI_JOB_STATUS:-unknown}"
      DURATION="${CI_JOB_DURATION:-0}"
      
      curl -s -X POST "${PARSEABLE_URL}/api/v1/ingest" \
        -H "Authorization: Basic ${PARSEABLE_AUTH}" \
        -H "X-P-Stream: gitlab-pipelines" \
        -H "Content-Type: application/json" \
        -d "[{
          \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
          \"project\": \"${CI_PROJECT_PATH}\",
          \"project_id\": ${CI_PROJECT_ID},
          \"pipeline_id\": ${CI_PIPELINE_ID},
          \"pipeline_url\": \"${CI_PIPELINE_URL}\",
          \"job_id\": ${CI_JOB_ID},
          \"job_name\": \"${CI_JOB_NAME}\",
          \"job_url\": \"${CI_JOB_URL}\",
          \"stage\": \"${CI_JOB_STAGE}\",
          \"ref\": \"${CI_COMMIT_REF_NAME}\",
          \"sha\": \"${CI_COMMIT_SHA}\",
          \"short_sha\": \"${CI_COMMIT_SHORT_SHA}\",
          \"author\": \"${GITLAB_USER_LOGIN}\",
          \"author_email\": \"${GITLAB_USER_EMAIL}\",
          \"status\": \"${STATUS}\",
          \"duration_seconds\": ${DURATION},
          \"runner\": \"${CI_RUNNER_DESCRIPTION}\",
          \"environment\": \"${CI_ENVIRONMENT_NAME:-none}\"
        }]" || true

Use the template:

include:
  - local: templates/parseable.yml

build:
  extends: .parseable_logging
  stage: build
  script:
    - npm ci
    - npm run build

test:
  extends: .parseable_logging
  stage: test
  script:
    - npm test

Method 2: GitLab Webhooks

Use GitLab webhooks to capture pipeline events.

Webhook Receiver

// gitlab-webhook-to-parseable.js
const express = require('express');
const axios = require('axios');

const app = express();
app.use(express.json());

const GITLAB_TOKEN = process.env.GITLAB_WEBHOOK_TOKEN;
const PARSEABLE_URL = process.env.PARSEABLE_URL;
const PARSEABLE_AUTH = process.env.PARSEABLE_AUTH;

app.post('/webhook', async (req, res) => {
  // Verify token
  if (req.headers['x-gitlab-token'] !== GITLAB_TOKEN) {
    return res.status(401).json({ error: 'Invalid token' });
  }
  
  const event = req.headers['x-gitlab-event'];
  const payload = req.body;
  
  let logEntry = null;
  
  if (event === 'Pipeline Hook') {
    logEntry = {
      timestamp: new Date().toISOString(),
      event_type: 'pipeline',
      project: payload.project?.path_with_namespace,
      pipeline_id: payload.object_attributes?.id,
      ref: payload.object_attributes?.ref,
      sha: payload.object_attributes?.sha,
      status: payload.object_attributes?.status,
      duration: payload.object_attributes?.duration,
      created_at: payload.object_attributes?.created_at,
      finished_at: payload.object_attributes?.finished_at,
      user: payload.user?.username
    };
  } else if (event === 'Job Hook') {
    logEntry = {
      timestamp: new Date().toISOString(),
      event_type: 'job',
      project: payload.project?.path_with_namespace,
      pipeline_id: payload.pipeline_id,
      job_id: payload.build_id,
      job_name: payload.build_name,
      stage: payload.build_stage,
      status: payload.build_status,
      duration: payload.build_duration,
      runner: payload.runner?.description,
      user: payload.user?.username
    };
  }
  
  if (logEntry) {
    try {
      await axios.post(`${PARSEABLE_URL}/api/v1/ingest`, [logEntry], {
        headers: {
          'Authorization': `Basic ${PARSEABLE_AUTH}`,
          'X-P-Stream': 'gitlab-webhooks',
          'Content-Type': 'application/json'
        }
      });
    } catch (error) {
      console.error('Error sending to Parseable:', error);
    }
  }
  
  res.status(200).json({ status: 'received' });
});

app.listen(3000);

Configure GitLab Webhook

Go to your project Settings → Webhooks
Enter your webhook URL
Enter a Secret Token
Select events: Pipeline events and Job events
Click Add webhook

Method 3: OpenTelemetry Tracing

Use OpenTelemetry for distributed tracing of pipelines.

variables:
  OTEL_EXPORTER_OTLP_ENDPOINT: ${PARSEABLE_OTEL_ENDPOINT}
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Basic ${PARSEABLE_AUTH},X-P-Stream=gitlab-traces"
  OTEL_SERVICE_NAME: gitlab-ci

build:
  stage: build
  image: node:18
  before_script:
    - npm install -g @opentelemetry/auto-instrumentations-node
  script:
    - node --require @opentelemetry/auto-instrumentations-node npm run build

GitLab CI/CD Variables

Configure these variables in GitLab:

Variable	Type	Description
`PARSEABLE_URL`	Variable	Parseable instance URL
`PARSEABLE_AUTH`	Variable (masked)	Base64 encoded `username:password`
`PARSEABLE_OTEL_ENDPOINT`	Variable	OTEL endpoint (for tracing)

Querying GitLab Logs

Query your GitLab CI/CD logs in Parseable:

-- Get recent pipeline runs
SELECT timestamp, project, pipeline_id, status, duration_seconds
FROM "gitlab-pipelines"
ORDER BY timestamp DESC
LIMIT 100

-- Find failed jobs
SELECT timestamp, project, job_name, stage, status, job_url
FROM "gitlab-pipelines"
WHERE status = 'failed'
ORDER BY timestamp DESC

-- Pipeline success rate by project
SELECT 
  project,
  COUNT(*) as total_pipelines,
  SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successful,
  ROUND(SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)::float / COUNT(*) * 100, 2) as success_rate
FROM "gitlab-pipelines"
WHERE timestamp > NOW() - INTERVAL '7 days'
  AND job_name IS NULL  -- Pipeline-level events only
GROUP BY project
ORDER BY success_rate ASC

-- Average job duration by stage
SELECT 
  stage,
  AVG(duration_seconds) as avg_duration,
  MAX(duration_seconds) as max_duration,
  COUNT(*) as job_count
FROM "gitlab-pipelines"
WHERE timestamp > NOW() - INTERVAL '7 days'
  AND duration_seconds IS NOT NULL
GROUP BY stage
ORDER BY avg_duration DESC

Best Practices

Use Templates - Centralize logging configuration
Include Context - Add project, pipeline, and job details
Log All Stages - Track each stage separately
Handle Failures - Use || true to prevent logging failures from breaking builds
Mask Secrets - Always mask sensitive variables

Troubleshooting

Logs Not Appearing

Verify Parseable URL is accessible from GitLab runners
Check CI/CD variables are properly configured
Verify the dataset exists or auto-creation is enabled
Check job logs for curl errors

Authentication Failures

Verify Base64 encoding of credentials
Check the Parseable user has ingest permissions
Ensure variables are not protected when running on unprotected branches

Next Steps

Set up alerts for failed pipelines
Create dashboards for CI/CD metrics
Configure GitHub Actions for multi-platform CI

GitLab CI

On this page