PagerDuty

Integrate Parseable with PagerDuty for incident management, on-call scheduling, and alert escalation.

Overview

Integrate Parseable with PagerDuty to:

Trigger Incidents - Automatically create PagerDuty incidents from alerts
On-Call Escalation - Route alerts to the right team members
Incident Management - Track and resolve issues with full context
Reduce Alert Fatigue - Use PagerDuty's intelligent grouping

Prerequisites

PagerDuty account with admin access
PagerDuty Integration Key (Events API v2)
Parseable alerts configured or Fluent Bit for log streaming

Setting Up PagerDuty Integration

Create a PagerDuty Service

Log in to your PagerDuty account
Go to Services → Service Directory
Click + New Service
Name your service (e.g., "Parseable Alerts")
Select an escalation policy
Under Integrations, select Events API V2
Click Create Service
Copy the Integration Key (32-character string)

Webhook Integration

Use Parseable's alerting system with a webhook to send incidents to PagerDuty.

Create a Webhook Endpoint

Create a webhook service that transforms Parseable alerts to PagerDuty Events API format:

// webhook-to-pagerduty.js
const express = require('express');
const axios = require('axios');

const app = express();
app.use(express.json());

const PAGERDUTY_ROUTING_KEY = process.env.PAGERDUTY_ROUTING_KEY;
const PAGERDUTY_EVENTS_URL = 'https://events.pagerduty.com/v2/enqueue';

app.post('/webhook', async (req, res) => {
  const alert = req.body;
  
  const pagerdutyEvent = {
    routing_key: PAGERDUTY_ROUTING_KEY,
    event_action: 'trigger',
    dedup_key: `parseable-${alert.name}-${alert.dataset}`,
    payload: {
      summary: `[Parseable] ${alert.name || 'Alert Triggered'}`,
      severity: mapSeverity(alert.severity),
      source: alert.dataset || 'parseable',
      timestamp: new Date().toISOString(),
      custom_details: {
        dataset: alert.dataset,
        query: alert.query,
        threshold: alert.threshold,
        current_value: alert.value,
        message: alert.message
      }
    },
    links: [
      {
        href: `https://your-parseable-instance.com/streams/${alert.dataset}`,
        text: 'View in Parseable'
      }
    ]
  };

  try {
    const response = await axios.post(PAGERDUTY_EVENTS_URL, pagerdutyEvent);
    res.status(200).json({ 
      status: 'sent',
      dedup_key: response.data.dedup_key 
    });
  } catch (error) {
    console.error('Error sending to PagerDuty:', error.response?.data || error);
    res.status(500).json({ error: 'Failed to send to PagerDuty' });
  }
});

// Map Parseable severity to PagerDuty severity
function mapSeverity(severity) {
  const mapping = {
    'critical': 'critical',
    'high': 'error',
    'medium': 'warning',
    'low': 'info',
    'error': 'error',
    'warning': 'warning',
    'info': 'info'
  };
  return mapping[severity?.toLowerCase()] || 'error';
}

// Endpoint to resolve incidents
app.post('/webhook/resolve', async (req, res) => {
  const alert = req.body;
  
  const pagerdutyEvent = {
    routing_key: PAGERDUTY_ROUTING_KEY,
    event_action: 'resolve',
    dedup_key: `parseable-${alert.name}-${alert.dataset}`
  };

  try {
    await axios.post(PAGERDUTY_EVENTS_URL, pagerdutyEvent);
    res.status(200).json({ status: 'resolved' });
  } catch (error) {
    console.error('Error resolving in PagerDuty:', error);
    res.status(500).json({ error: 'Failed to resolve in PagerDuty' });
  }
});

app.listen(3000, () => {
  console.log('PagerDuty webhook server listening on port 3000');
});

Docker Compose Setup

version: '3.8'
services:
  webhook-to-pagerduty:
    build: .
    ports:
      - "3000:3000"
    environment:
      - PAGERDUTY_ROUTING_KEY=your-32-character-integration-key

Configure Parseable Alert

Configure your Parseable alert to send to the webhook endpoint:

{
  "name": "Critical Error Rate",
  "dataset": "production-logs",
  "alertType": "threshold",
  "condition": {
    "field": "level",
    "operator": "equals",
    "value": "critical"
  },
  "threshold": 10,
  "duration": "5m",
  "webhook": {
    "url": "http://webhook-to-pagerduty:3000/webhook",
    "method": "POST"
  }
}

PagerDuty Events API v2

Event Actions

Action	Description
`trigger`	Create a new incident or add to existing
`acknowledge`	Acknowledge an existing incident
`resolve`	Resolve an existing incident

Severity Levels

Severity	Description
`critical`	System is unusable
`error`	A problem that needs immediate attention
`warning`	A problem that should be addressed soon
`info`	Informational message

Deduplication

Use the dedup_key to group related alerts into a single incident:

dedup_key: `parseable-${alert.name}-${alert.dataset}`

This prevents alert storms from creating multiple incidents for the same issue.

Advanced Configuration

Custom Fields

Add custom details to provide more context:

custom_details: {
  dataset: alert.dataset,
  query: alert.query,
  threshold: alert.threshold,
  current_value: alert.value,
  environment: process.env.ENVIRONMENT,
  region: process.env.REGION,
  runbook_url: 'https://wiki.example.com/runbooks/high-error-rate'
}

Priority Mapping

Map Parseable alert priorities to PagerDuty priorities:

const priorityMapping = {
  'P1': { severity: 'critical', urgency: 'high' },
  'P2': { severity: 'error', urgency: 'high' },
  'P3': { severity: 'warning', urgency: 'low' },
  'P4': { severity: 'info', urgency: 'low' }
};

Best Practices

Use Deduplication - Prevent duplicate incidents with consistent dedup keys
Include Runbook Links - Add links to troubleshooting documentation
Set Appropriate Severity - Map alert severity correctly to avoid alert fatigue
Add Context - Include relevant details like dataset, query, and thresholds
Test Integration - Verify the integration works before production deployment

Troubleshooting

Incidents Not Creating

Verify the Integration Key is correct
Check the webhook service logs for errors
Ensure the PagerDuty service is not disabled
Verify network connectivity to PagerDuty

Duplicate Incidents

Ensure dedup_key is consistent for related alerts
Check that the dedup key format matches across trigger/resolve events

Missing Context

Verify custom_details are being populated
Check that the alert payload contains expected fields

Next Steps

Set up Parseable alerts for automated incident creation
Configure Slack for team notifications
Create dashboards for visual monitoring

PagerDuty

On this page