How we find out something is broken, who gets paged, and how we restore. Alert routing, the rule catalog, on-call, verified backups with monthly restore drills, and the observability stack configuration. Times in UTC.
Alert routing · per tenant, cooldown-deduplicated
Scope Channel Cooldown Last fired Platform (global) PagerDuty Events v2 + Slack #ops-alerts 10 min INC-451 · 22 min ago ✎ edit routing Svenska Bouleförbundet Opsgenie + email 15 min 3 d ago ✎ edit routing Norges Bouleforbund Webhook (custom NOC) 15 min never ✎ edit routing
Alert history keeps every fired/suppressed decision (24 h: 6 fired, 41 suppressed by cooldown). Test-fire available per channel.
Alert rules & on-call
Severity matrix: Sev1 = page immediately, respond 15 min · Sev2 = page business hours, respond 4 h · Sev3 = ticket, next business day.
On-call rotation
Now Patrik W (primary) · Maja L (backup)
Next week Maja L (primary) · Jonas B (backup)
Escalation primary → backup after 10 min unacked → founder
Postmortems (blameless)
PM-2026-06 · INC-448 webhook queue 2 actions open
PM-2026-05 · INC-431 deploy regression closed
Template: timeline · root cause · contributing factors · action list.
Backups & disaster recovery · RPO target 4 h / RTO target 8 h
Monthly restore drills (automated) · temp database from point-in-time restore + smoke queries vs prod ±1 %
Drill Run at RPO measured RTO measured Result 2026-07 monthly restore drill 2026-07-01 04:12 UTC 2 h 10 m 3 h 40 m ● PASS 2026-06 monthly restore drill 2026-06-02 04:09 UTC 2 h 55 m 4 h 05 m ● PASS 2026-05 monthly restore drill 2026-05-04 04:15 UTC 3 h 20 m 5 h 10 m ● PASS (slow smoke)
Latest verified snapshot: 2026-07-02 12:00 UTC (2 h old). A failed weekly backup-verify auto-opens a Sev2. PITR restore is destructive → step-up auth + audit reason required. Drill runs under a least-privilege service principal.
Scenario runbooks
Data corruption
detect → contain → PITR restore → post-mortem
Region down
fail over per runbook · comms via status page
Credential leak
rotate (Stripe / ACS / DB) → revoke → audit sweep
Ransomware
isolate → verified snapshot restore → report
Delete-by-mistake
point-in-time restore of affected tables
Credential rotation procedures
Credential Procedure Last rotated / verified Stripe API keys container-app secret set + update 2026-05-20 Run rotation → ACS connection string regenerate-key + container update 2026-04-11 Run rotation → Key Vault (backup) soft-delete + purge protection · secret backup/restore verified 2026-07-01 Run rotation →
Observability configuration
Structured logging
JSON renderer in prod · request context (request/tenant/user/trace id) bound to every line · sensitive-field masking ON .
Metrics & dashboards
Scrape endpoint live · business metrics (licenses, auth, scores, sync, audit) · 12-panel API overview dashboard provisioned.
Open Grafana ↗ Distributed tracing
OpenTelemetry exporter enabled · API, HTTP clients and data layer auto-instrumented · trace id echoed in the response header.
Health probes
Probe Purpose Latency State /health liveness 4 ms ● OK /ready readiness (DB ping) 11 ms ● OK /startup startup probe (container platform) — ● OK /health/detailed per-dependency: database 9 ms · cache 2 ms 12 ms ● OK
Frontend error capture (Sentry)
DSN served to apps · CSP-safe envelope tunnel through the API · 24 h: 12 frontend errors (0 new signatures).
Continuous profiling
Always-on CPU/memory profiles on the API · flame graphs retained 14 days · overhead budget under 2 %.
States
⏳ Loading drill history…
✓ No alerts firing — on-call is quiet.
⚠ Last restore drill FAILED — Sev2 opened automatically, RPO/RTO baseline not updated.
