Skip to main content

PagerDuty Integration

Integrate BlackTide with PagerDuty for automated incident creation, on-call escalation, and 24/7 incident management.

Prerequisites

  • PagerDuty account (any plan)
  • Permissions to create integration keys
  • BlackTide account with alert configuration access

Step 1: Create PagerDuty Integration

  1. Log in to PagerDuty
  2. Go to Services → Service Directory
  3. Select an existing service or click New Service
  4. In the service settings, go to Integrations tab
  5. Click Add Integration
  6. Select Events API V2 or Generic Webhook
  7. Name it "BlackTide Monitoring"
  8. Copy the Integration Key

Step 2: Configure BlackTide Channel

  1. Log in to BlackTide
  2. Navigate to Alerts → Alert Channels
  3. Click New Channel
  4. Select PagerDuty
  5. Fill in configuration:
    • Name: On-Call Team
    • Integration Key: Paste from Step 1
    • Severity: critical, error, warning, info
  6. Click Test Channel
  7. Verify incident created in PagerDuty
  8. Save the channel

Configuration Example

{
  "type": "pagerduty",
  "name": "On-Call Engineering",
  "integrationKey": "YOUR_INTEGRATION_KEY",
  "severity": "critical",
  "dedupKey": "monitor-{monitor_id}",
  "autoResolve": true
}

Step 3: Create Alert Rule

  1. Go to Alerts → Alert Rules
  2. Click New Rule
  3. Configure:
    • Name: Production API P0
    • Monitors: Select critical monitors
    • Conditions: 3 consecutive failures
    • Channels: Select PagerDuty channel
  4. Save the rule

Features

Incident Creation

BlackTide automatically creates PagerDuty incidents when monitors fail:

Incident Title: [BlackTide] Production API Down

Details:
Monitor: Production API Health Check
Status: Down
Failures: 3 consecutive
Last Error: Connection timeout after 10s
Time: 2026-02-13 15:30 UTC

[View Monitor] [Dashboard]

Auto-Resolution

When monitors recover, PagerDuty incidents are automatically resolved:

Incident Resolved

Monitor: Production API Health Check
Status: Up (recovered)
Downtime: 5 minutes 32 seconds
Recovery Time: 2026-02-13 15:35:32 UTC

De-duplication

BlackTide uses dedupKey to prevent duplicate incidents:

  • Same monitor failing multiple times → updates existing incident
  • Different monitors failing → creates separate incidents

Severity Mapping

BlackTide PriorityPagerDuty Severity
P0 (Critical)critical
P1 (High)error
P2 (Medium)warning
P3 (Low)info

Advanced Configuration

Custom Routing

// Route different monitor types to different services
{
  "production-api": "integration-key-1",
  "database": "integration-key-2",
  "web3": "integration-key-3"
}

Escalation Policies

In PagerDuty, configure escalation policies:

  1. Level 1: Primary on-call (immediate)
  2. Level 2: Backup on-call (after 5 min)
  3. Level 3: Manager (after 15 min)

Custom Fields

{
  "customFields": {
    "monitor_id": "mon_abc123",
    "environment": "production",
    "team": "backend",
    "runbook": "https://wiki.example.com/runbooks/api-down"
  }
}

Best Practices

1. Reserve for Critical Alerts

Only page for P0/P1 incidents:

  • PagerDuty: Production outages requiring immediate action
  • Slack: Important but not urgent issues
  • Email: Informational alerts

2. Use Schedules

Configure on-call rotations in PagerDuty:

  • Weekly rotations
  • Backup coverage
  • Holiday schedules

3. Test Regularly

Test integration monthly:

  1. Trigger test alert from BlackTide
  2. Verify incident created
  3. Acknowledge and resolve
  4. Confirm auto-resolution works

4. Add Runbooks

Include runbook links in alerts:

Alert message includes:
- What failed: Production API
- Impact: Users cannot access app
- Runbook: https://wiki.example.com/api-down
- Dashboard: https://blacktide.xyz/monitors/...

Troubleshooting

Test Alert Not Creating Incident

Possible Causes:

  • Integration key is incorrect
  • PagerDuty service is disabled
  • API key permissions insufficient

Solution:

  1. Verify integration key matches PagerDuty
  2. Check PagerDuty service is active
  3. Test integration key manually:
    curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' \
  -d '{
    "routing_key": "YOUR_KEY",
    "event_action": "trigger",
    "payload": {
      "summary": "Test from BlackTide",
      "severity": "critical",
      "source": "blacktide"
    }
  }'

Incidents Not Auto-Resolving

Solution:

  • Ensure autoResolve: true in channel config
  • Verify notifyOnRecovery enabled in alert rule
  • Check dedup key is consistent

Pricing Note

PagerDuty charges per user. BlackTide integration does not incur additional costs, but ensure you have sufficient PagerDuty seats for your team.

Next Steps