vaka
CI statusLatest releaseGitHub

Monitoring

Monitors tell you when something that should happen doesn't. Vaka alerts through your existing alert rules (webhook, Slack, email, SMS, GitHub dispatch) when a monitor goes down, and again when it recovers.

Three kinds:

Heartbeat monitors

A heartbeat monitor expects a check-in on a schedule. If the check-in doesn't arrive within the grace period, if the job reports an error, or if a started run doesn't finish within its maximum runtime, the check fails.

Create one with the API (or let the first check-in create it, see below):

curl -X POST https://vaka.example.com/api/v1/projects/web/monitors -H "Authorization: Bearer $VAKA_TOKEN" \
  -H 'Content-Type: application/json' -d '{
    "slug": "nightly-backup",
    "name": "Nightly backup",
    "environment": "production",
    "config": {
      "schedule": {"cron": "0 3 * * *", "timezone": "Europe/Stockholm"},
      "grace_s": 600,
      "max_runtime_s": 3600
    }
  }'
Config Meaning
schedule {"every": "15m"} (at least 1 minute), or {"cron": "0 3 * * *", "timezone": "UTC"}: five fields, ranges, steps, lists, names (mon-fri, jan) and @daily, @hourly, @weekly, @monthly
grace_s How late a check-in may be (default 120 s). Also accepts "10m"
max_runtime_s How long a run reported with in_progress may take (default 1800 s; 0 = no limit)

fail_threshold and recover_threshold (default 1) set how many failed or successful checks in a row change the state.

Check in

Send the project's ingest key, the same one your SDKs use:

# Crontab: report success after the job, failure if it fails.
0 3 * * * /usr/local/bin/backup.sh && curl -fsS "https://vaka.example.com/ingest/v1/monitors/nightly-backup/ok?key=$VAKA_KEY" \
  || curl -fsS "https://vaka.example.com/ingest/v1/monitors/nightly-backup/error?key=$VAKA_KEY"

To measure runtime and catch jobs that hang, report the start too:

curl -X POST https://vaka.example.com/ingest/v1/monitors/nightly-backup/check-ins -H "X-Vaka-Key: $VAKA_KEY" \
  -H 'Content-Type: application/json' -d '{"status": "in_progress", "check_in_id": "run-42"}'
# ... the job runs ...
curl -X POST https://vaka.example.com/ingest/v1/monitors/nightly-backup/check-ins -H "X-Vaka-Key: $VAKA_KEY" \
  -H 'Content-Type: application/json' -d '{"status": "ok", "check_in_id": "run-42", "release": "1.4.0"}'

Or use a helper that does this for you: vaka monitor run nightly-backup -- ./backup.sh in the CLI, withMonitor in Node, @vaka.monitor("slug") in Python and vaka.WithMonitor in Go. They also tag errors from the run, so the incident lists them.

Field Meaning
status in_progress, ok (default) or error
check_in_id Pairs a start with its finish
duration_ms Runtime, if you measured it yourself
release, environment Stored with the check
message Error text for error check-ins, shown in alerts
monitor Create the monitor on its first check-in: {"schedule": {...}, "grace_s": 600, "max_runtime_s": 3600, "name": "..."}

The short form GET or POST /ingest/v1/monitors/{slug}/ok|error|start?key=... takes the same fields as query parameters. Check-ins are limited to one per second per monitor and status.

HTTP monitors

curl -X POST https://vaka.example.com/api/v1/projects/web/monitors -H "Authorization: Bearer $VAKA_TOKEN" \
  -H 'Content-Type: application/json' -d '{
    "name": "Shop API", "kind": "http",
    "config": {"url": "https://shop.example.com/health", "every": "1m", "body_contains": "ok"},
    "secret_headers": {"Authorization": "Bearer ..."}
  }'
Config Meaning
url http or https URL (required)
every Check interval, 30s to 7d (default 1m)
method GET (default) or HEAD
timeout_s Seconds to wait for a response (default 10, at most 60)
expect_status Accepted codes: 200, "2xx" or a list (default 2xx and 3xx)
body_contains Text the response (first 1 MB) must contain
headers Extra request headers, stored and shown as-is
follow_redirects Follow up to 5 redirects (default true); each target is checked like url
tls_warn_days For https: fire tls_expiring this many days before the certificate expires (default 14, 0 = off)

secret_headers (next to config) are stored encrypted with VAKA_SECRET_KEY and only their names are shown again. Updating them replaces the whole set; {} removes them.

An HTTP monitor goes down after 2 failed checks in a row and up after 2 good ones (fail_threshold, recover_threshold). A check fails on a connection error, a timeout, an unexpected status, a missing body_contains or an invalid certificate.

Every check sends a W3C traceparent header. If the target reports traces to Vaka, the incident context has check_trace_id, and get_trace shows where the failed request spent its time.

Run a check right away with POST .../monitors/{slug}/run, the MCP tool run_monitor_check or Check now in the admin UI, for example to confirm a fix. It counts like a scheduled check.

Which addresses checks can reach

Checks only reach public addresses. Private, loopback, link-local and cloud metadata addresses are refused when the monitor is saved and again when each connection is made (after DNS), and redirects to them are refused too. To monitor internal services, the operator lists the ranges in VAKA_MONITOR_ALLOW_CIDRS, for example 10.0.0.0/8,fd00::/8.

When a check can't connect, Vaka first checks its own network by requesting the canary URLs (VAKA_MONITOR_CANARY_URLS, by default Google's and Cloudflare's). If none answer, the check is stored as inconclusive and doesn't count, so a network problem on Vaka's side doesn't page you for every monitor. Set it to off to skip this.

TLS monitors

curl -X POST https://vaka.example.com/api/v1/projects/web/monitors -H "Authorization: Bearer $VAKA_TOKEN" \
  -H 'Content-Type: application/json' -d '{"name": "Shop certificate", "kind": "tls", "config": {"host": "shop.example.com"}}'

Config: host (required), port (443), every (default 6h), tls_warn_days (14). Vaka connects, verifies the certificate and records when it expires. A certificate that is invalid or expired fails the check (and takes the monitor down). Once it is within tls_warn_days of expiring, Vaka fires tls_expiring once for that certificate and opens a tls_expiring incident, which closes when a renewed certificate shows up. HTTP monitors on https URLs do the same, so a separate TLS monitor is only needed for hosts you don't otherwise check.

States, incidents and flapping

A monitor is unknown until its first check, then up or down. Going down opens an incident that closes when the monitor recovers; monitor_up alerts carry the downtime.

A monitor that changes state 4 or more times within 30 minutes is flapping: Vaka sends one alert saying so, then stays quiet until it has been stable for 15 minutes, and sends one more alert with the state it settled in.

Pause a monitor during maintenance with PATCH .../monitors/{slug} and {"paused_until": "2h"} (a duration, an RFC 3339 time or "forever"); {"paused_until": null} resumes it. Check-ins while paused are stored but change nothing.

Alerts

curl -X POST https://vaka.example.com/api/v1/alert-rules -H "Authorization: Bearer $VAKA_TOKEN" \
  -H 'Content-Type: application/json' -d '{
    "project": "web", "trigger": "monitor_down", "channel": "sms", "target": "+46701234567",
    "filters": {"monitors": ["nightly-backup"]}
  }'

Add a second rule with monitor_up to hear about recoveries, and one with tls_expiring for certificates. Webhook payloads contain monitor, incident and details. monitor_kind (heartbeat, http, tls) filters by kind.

Why is it down?

When a monitor goes down, Vaka stores a context snapshot on the incident and puts it in the alert payload:

The MCP tool get_incident returns it, recomputed while the incident is open, so an agent can go from "the backup job is down" to get_fix_context on the issue that broke it.

Admin UI

Monitors on a project page lists monitors with state and uptime. A monitor's page shows its last checks, incidents with their hints, the crontab line to check in (heartbeats) or the target and certificate expiry (HTTP and TLS), Check now, and pause and resume.

API

Endpoint Purpose
GET/POST /api/v1/projects/{p}/monitors List (with uptime for 24 h, 7 d, 30 d) or create
GET/PATCH/DELETE /api/v1/projects/{p}/monitors/{slug} One monitor with recent checks and incidents
GET /api/v1/projects/{p}/monitors/{slug}/checks?since=7d Check results
POST /api/v1/projects/{p}/monitors/{slug}/run Run an HTTP or TLS check now
GET /api/v1/projects/{p}/incidents?status=open Incidents
GET/PATCH /api/v1/incidents/{id} One incident; {"acknowledged": true}

Limits