Heartbeat Monitor

Push-based monitoring for cron jobs, scheduled tasks, and background workers. Your application pings BlackTide; if no ping is received within the threshold, an alert triggers.

What is a Heartbeat Monitor?

A Heartbeat Monitor is the opposite of traditional monitoring. Instead of BlackTide checking your service, your service pings BlackTide at regular intervals.

Traditional vs Heartbeat

Use Cases

• Cron Jobs: Daily backups, data exports, cleanups
• Scheduled Tasks: Report generation, email sending
• Background Workers: Queue processors, ETL pipelines
• Serverless Functions: AWS Lambda, Google Cloud Functions
• Batch Jobs: Data processing, analytics aggregation

Configuration Options

Basic Configuration

How It Works

Setup Flow

1. Create a heartbeat monitor in BlackTide
2. Copy the unique ping URL
3. Add ping to your script/cron job
4. BlackTide waits for pings
5. Alert if ping not received within threshold

Example Timeline

Expected Interval: 24 hours
Grace Period: 15 minutes

00:00 - Last ping received
24:00 - Expecting next ping
24:15 - Grace period ends → ALERT (no ping for 24h 15m)

Configuration Examples

Daily Backup Job

{
  "type": "heartbeat",
  "name": "Daily Database Backup",
  "expectedInterval": "24h",
  "gracePeriod": "15m",
  "timezone": "UTC"
}

Ping URL: https://api.blacktide.xyz/ping/abc123

Hourly ETL Pipeline

{
  "type": "heartbeat",
  "name": "Hourly Data Sync",
  "expectedInterval": "1h",
  "gracePeriod": "5m",
  "timezone": "America/New_York"
}

Weekly Report Generation

{
  "type": "heartbeat",
  "name": "Weekly Sales Report",
  "expectedInterval": "7d",
  "gracePeriod": "1h",
  "timezone": "Europe/London"
}

Sending Pings

Using curl

# At the end of your script
curl -X POST https://api.blacktide.xyz/ping/YOUR_PING_ID

# Or use GET
curl https://api.blacktide.xyz/ping/YOUR_PING_ID

Bash Script

#!/bin/bash
set -e

# Your backup logic
pg_dump mydb > backup.sql
aws s3 cp backup.sql s3://backups/

# Ping BlackTide on success
curl -X POST https://api.blacktide.xyz/ping/YOUR_PING_ID

Python Script

import requests

def send_heartbeat(ping_id):
    url = f"https://api.blacktide.xyz/ping/{ping_id}"
    requests.post(url)

# Your job logic
process_data()

# Ping on success
send_heartbeat("YOUR_PING_ID")

Node.js Script

const fetch = require('node-fetch');

async function sendHeartbeat(pingId) {
  await fetch(`https://api.blacktide.xyz/ping/${pingId}`, {
    method: 'POST'
  });
}

// Your job logic
await processData();

// Ping on success
await sendHeartbeat('YOUR_PING_ID');

Crontab Example

# Run backup daily at 2 AM and ping BlackTide
0 2 * * * /path/to/backup.sh && curl https://api.blacktide.xyz/ping/YOUR_PING_ID

Advanced Features

1. Success/Failure Reporting

Send different pings for success vs failure:

#!/bin/bash

if /path/to/backup.sh; then
  # Success ping
  curl https://api.blacktide.xyz/ping/YOUR_PING_ID/success
else
  # Failure ping
  curl https://api.blacktide.xyz/ping/YOUR_PING_ID/fail
fi

2. Include Runtime Data

Send metadata with your ping:

curl -X POST https://api.blacktide.xyz/ping/YOUR_PING_ID \
  -H "Content-Type: application/json" \
  -d '{
    "duration": 125,
    "recordsProcessed": 1500,
    "status": "success"
  }'

3. Start/End Pings

Track job duration:

# Start of job
curl https://api.blacktide.xyz/ping/YOUR_PING_ID/start

# Your job logic here

# End of job
curl https://api.blacktide.xyz/ping/YOUR_PING_ID/end

Best Practices

1. Ping Only on Success

Only send heartbeat if job completes successfully:

#!/bin/bash
set -e  # Exit on any error

backup_database
upload_to_s3
cleanup_old_files

# Only reaches here if all commands succeed
curl https://api.blacktide.xyz/ping/YOUR_PING_ID

2. Set Appropriate Grace Period

• Fast jobs (under 5 min): 5-10 min grace period
• Medium jobs (5-30 min): 15-30 min grace period
• Long jobs (over 1 hour): 1-2 hour grace period

3. Use Idempotent Scripts

Ensure your job can run multiple times safely:

• Check if backup already exists
• Use unique filenames (timestamps)
• Clean up partial failures

4. Handle Network Failures

Heartbeat ping failures should not fail your job:

# Job logic
process_data

# Ping with timeout, ignore failure
curl -m 10 https://api.blacktide.xyz/ping/YOUR_PING_ID || true

5. Monitor Long-Running Jobs

For jobs that take hours, send periodic pings:

while processing:
    process_batch()
    send_heartbeat()  # Every 10 minutes
    sleep(600)

Common Patterns

Database Backup

#!/bin/bash
PING_URL="https://api.blacktide.xyz/ping/YOUR_PING_ID"
DATE=$(date +%Y%m%d)

# Backup
pg_dump mydb | gzip > backup_$DATE.sql.gz

# Upload
aws s3 cp backup_$DATE.sql.gz s3://backups/

# Ping on success
curl $PING_URL

Log Rotation

#!/bin/bash
find /var/log/app -name "*.log" -mtime +30 -delete
curl https://api.blacktide.xyz/ping/YOUR_PING_ID

Data Sync

import requests

def sync_data():
    # Sync logic
    records = fetch_new_records()
    upload_to_warehouse(records)

    # Ping BlackTide
    requests.post('https://api.blacktide.xyz/ping/YOUR_PING_ID')

if __name__ == '__main__':
    sync_data()

Troubleshooting

Alert Triggered But Job Ran Successfully

Possible Causes:

• Ping failed due to network issue
• Script exited before ping
• Timeout too short

Solution:

• Add timeout to curl: curl -m 30
• Check job logs for errors before ping
• Verify BlackTide API is reachable

No Alert But Job Failed

Cause: Ping sent even when job failed.

Solution:

• Use set -e in bash scripts
• Check exit codes before pinging
• Use try/catch in programming languages

Pings Received But Timing Is Off

Cause: Timezone mismatch.

Solution:

• Verify monitor timezone matches cron timezone
• Use UTC for consistency

Next Steps

• HTTP Monitor: Monitor always-on services
• Configure Alert Rules for heartbeat failures
• Monitors API: Programmatically create heartbeat monitors
• Best Practices for effective monitoring