Production OP
Platform operations
Jobs, operational controls, incidents, release and continuity for the whole platform. SLO/error budget, public status page and cost live in Operations. Every action here is audited; dangerous actions require a reason.
12
queued jobs
● 2 beyond target duration (SLA widget · F21.08.05)
2
failed runs 24 h
● webhook-fanout-retry ×2
54
registered jobs
cross-tenant scope
Job inventory · schedule, owner, last run, next run · F21.08.01
| Job | Schedule | Owner | Last run | Next run | Manual run | |
|---|---|---|---|---|---|---|
ranking-recompute | cron 03:00 daily | team-sport | OK · 4 m 12 s | in 16 h | yes | Drill in → |
invoice-dunning-sweep | cron 06:00 daily | team-finance | OK · 41 s | in 19 h | no | Drill in → |
audit-worm-seal | hourly | team-security | OK · 8 s | in 21 min | no | Drill in → |
webhook-fanout-retry | every 5 min | team-platform | FAILED · exit 1 | in 2 min | yes | Drill in → |
support-access-grant-expiry | every minute | team-security | OK · 2 s | in 40 s | no | Drill in → |
backup-verify-restore | weekly Sun 04:00 | team-platform | OK · 22 m | Sun | yes | Drill in → |
Per-job status — webhook-fanout-retry · last runs, log tail, 30 d failure trend · F21.08.02
2 Jul 10:35 12 s · exit 1 manual (jonas.holm)
ERROR sendgrid 429 — backoff exhausted 2 Jul 10:30 11 s · exit 1 schedule
ERROR sendgrid 429 2 Jul 10:25 9 s · exit 0 schedule
delivered 214 events Failure trend 30 d (red = failed runs)
Manual run-now · reason + audit · F21.08.03
Jobs registered with manual_run=false render this button disabled (double-fire unsafe).
Webhook failures inbox · replay / suppress / mark handled · F21.08.06
match.finished 2 Jul 10:12 Nordvik Boule → klubbsystem.example/hooks · 410 Gone · 6 attempts
invoice.paid 2 Jul 09:44 Fjordkedjan → erp.fjordkedjan.example · timeout · 3 attempts
Rate-limit overrides · auto-expire · F21.08.07
Cache invalidation · targeted Redis key/pattern purge · throttled · F21.08.10
Global feature flags · platform-wide, audited · F21.08.08
| Flag | State | Rollout | |
|---|---|---|---|
groups_messenger | on | 100% | |
courses | on | 25% rollout | |
ai_plane | off | — | |
new_scoring_engine | on (canary tenants) | 2 tenants |
Database migrations status · F21.08.09
Data-layer init: complete · last migration run 2 Jul 03:10
Pending index changes: 1 —
Schema drift: none (see Dev tools → schema diff)
Pending index changes: 1 —
support_notes(user_id,status)Schema drift: none (see Dev tools → schema diff)
Scheduled maintenance mode · per tenant, user-facing banner · F21.08.11
fresh-auth < 5 min
Active incidents · severity, affected tenants, responder, ETA · F21.09.01
| Id | Severity | Summary | Responder | Status page |
|---|---|---|---|---|
INC-451 | SEV1 | Elevated API latency — postgres pool | Patrik W · started 10:19 | public · toggle → status.petanque.life (F21.09.04) |
INC-450 | SEV2 | Backup verify failed on restore step | Maria E · started 08:02 | internal · toggle → status.petanque.life (F21.09.04) |
INC-448 | SEV4 | Stripe webhook retry queue growing | unassigned · 1 Jul | internal · toggle → status.petanque.life (F21.09.04) |
Create incident · auto-pages on-call (PagerDuty/OpsGenie) · F21.09.02
Timeline — INC-451 · append-only updates · F21.09.03
10:41 · maria.ek · status → mitigating · "pool size doubled, latency recovering"
10:28 · patrik.w · comms sent → status page + banner (sv/en)
10:19 · created SEV1 · paged on-call (ack 2 min)
10:28 · patrik.w · comms sent → status page + banner (sv/en)
10:19 · created SEV1 · paged on-call (ack 2 min)
Impact — INC-451 · F21.09.06–07 · F21.15.10
~1 240
affected users (ApiUsageEvent scan, outage window)
2 tenants
impact-preview: union across affected services, top by MRR
SLA impact report (99.9% target)
| Tenant | Outage min | Credit tier |
|---|---|---|
| Sjöstad PK | 52 | 10% (tiers 0/10/25/50 at 0/45/240/720 min) |
| Nordvik Boule | 31 | 0% |
Deploy status · revision, build, git-sha, health per app · F21.18.01
| App | Revision · sha | Commit | Health | |
|---|---|---|---|---|
| api | rev 2026-07-02.3 · a91f27c | "fix: pool sizing" | healthy | |
| admin | rev 2026-07-01.1 · 88c2e01 | "feat: league play" | healthy | |
| app | rev 2026-07-01.2 · 71b9aa0 | "feat: pickup games" | healthy | |
| sys | rev 2026-06-30.1 · 5efc112 | "feat: escrow view" | healthy | |
| web | rev 2026-06-28.2 · 3d0a998 | "seo: venue pages" | healthy |
Rollback = Container App revision switch · sys_engineer + fresh-auth · every attempt persists a SysRollbackRecord, failures return 502 with the record id.
Canary metrics · canary vs stable · SSE samples · F21.18.02
| Metric | Stable | Canary | Δ |
|---|---|---|---|
| Error rate | 0.31% | 0.29% | −0.02 pp ✓ |
| P95 latency | 238 ms | 251 ms | +13 ms (threshold 50) |
| P99 latency | 590 ms | 640 ms | +50 ms |
Release-notes drafter · PRs between tags, grouped by label · F21.18.05
DR drill history · two-step remediation attach · F21.18.04
Full region failover (dry-run)2026-06-15
passed · RTO 38 min (target 60)
Postgres PITR restore2026-03-12
passed w/ finding: doc gap in runbook — remediated 20 Mar
sys_engineerfresh-auth
Backups · snapshots, PITR, verification · F21.19.01–04
Daily platform snapshot: OK 03:00 · retained 30 d (F21.19.02)
Weekly restore-to-sandbox verify: FAILED Sun → INC-450 (F21.19.04)
Last tenant export: Nordvik Boule · 30 Jun · signed URL, 7 d expiry (F21.19.01)
Weekly restore-to-sandbox verify: FAILED Sun → INC-450 (F21.19.04)
Last tenant export: Nordvik Boule · 30 Jun · signed URL, 7 d expiry (F21.19.01)
Data residency · for regulatory review · F21.19.05
| Data | Region | Regime |
|---|---|---|
| Tenant documents + database | Azure West Europe (NL) | GDPR |
| Backups (GRS) | West Europe + North Europe (IE) | GDPR |
| Media/uploads | West Europe | GDPR |
| E-mail (SendGrid) | EU processing addendum | DPA |
Continuity export pipeline · tenant-controlled S3/SFTP, PGP/age keys · F21.19.06
Daily snapshot + hourly delta: running · Svenska Bouleförbundet → own S3
Weekly fetch-back verify: OK 29 Jun · stop-on-cancellation armed
Weekly fetch-back verify: OK 29 Jun · stop-on-cancellation armed
Source-code escrow · deterministic bundles → NCC Group + Iron Mountain · F21.19.07
Latest deposit v2026.06.2 · 28 Jun · sealed (age multi-recipient) ●
NCC Group quarterly coverage complete · last SFTP 28 Jun ●
Iron Mountain delivery confirmed via HMAC callback ●
Deposit gap monitor 5 d since last release deposit (SEV2 at 90 d) ●
sys_securityfresh-auth
Wind-down readiness · composite view · F21.19.08
YELLOW 1 reserve short · drill due in 18 d
Funded reserve: Svenska BF funded · Fjordkedjan short (74% of 12× monthly cost)
Continuity-export freshness: OK · Escrow drift: none
Clause versions (sv/en/fr/es): hash chain OK · Quarterly drill: tabletop 2026-04-02 (passed)
Continuity-export freshness: OK · Escrow drift: none
Clause versions (sv/en/fr/es): hash chain OK · Quarterly drill: tabletop 2026-04-02 (passed)
sys_engineer gate — all seven tools Audit prefix sys.dev-tools.* · F21.16
Query playground (write) · two-person approval · F21.16.05
Webhook tester · HMAC-SHA-256-signed payload, 10 s timeout · F21.16.03
Seed-data generator · sandbox-by-default · F21.16.04
Read tools · F21.16.01/02/06/07
QA — smoke protocol & route inventory · F21.20.QA
Route inventory: 175 sys routes classified (section, purpose, data source, verdict) — frozen baseline for drift detection
Playwright sweep: all routes, catches 5xx + fatal React-tree crashes · green in CI
Supervised prod-data protocol: token via settings, severity P0–P3, write actions need explicit OK
Playwright sweep: all routes, catches 5xx + fatal React-tree crashes · green in CI
Supervised prod-data protocol: token via settings, severity P0–P3, write actions need explicit OK
Container app metrics · CPU, memory, replicas, rate, latency · allowlisted apps · F21.15.02
| Service | CPU · mem | Replicas | Requests | Latency |
|---|---|---|---|---|
| api | 38% CPU · 1.9 GB | 3 replicas | 412 rps | P95 240 ms · P99 610 ms |
| worker | 22% CPU · 1.1 GB | 2 replicas | — | queue lag 4 s |
| admin | 4% CPU · 300 MB | 1 replica | 9 rps | P95 120 ms |
Error-rate dashboard · 5xx by endpoint · 1/6/24/72 h · F21.15.03
Database throughput · provisioned vs consumed (P95), throttling, hot partitions · F21.15.06
Consumed P95: 64% of provisioned · throttled requests 24 h: 0
Hot partitions:
Hot partitions:
matches(tenant=sbf) · audit(day=2026-07-02) Service dependency graph + impact analysis · F21.15.08–09 · F21.15.11
Force-directed graph of service-graph.yaml enriched with live health + open incidents. Regional badges (SE-only: Bankgirot).
Impact — Stripe degraded: exposed MRR 61 400 SEK · 14 tenants (top-5 by MRR listed) · features: license payment, shop · mitigation: queue + retry.
Historical incidents — Stripe: 3 in window · downtime clamped to window end.
Impact — Stripe degraded: exposed MRR 61 400 SEK · 14 tenants (top-5 by MRR listed) · features: license payment, shop · mitigation: queue + retry.
Historical incidents — Stripe: 3 in window · downtime clamped to window end.
Azure cost breakdown, budget alerts, uptime log and external-dependency probes (F21.15.01/04/05/07) render on Operations.