Postmortem — {{Incident title}}

Copy this file to docs/security/incidents/YYYY-MM-DD-{slug}.md when an incident opens. Fill the timeline as the incident progresses; complete the rest within 5 business days of resolution.

Blameless rule: every “why” answers a system question, never a person question. “The deploy step doesn’t catch this” is fine; “Alice deployed without checking” is not.

Summary

Severity: P0 / P1 / P2 / P3
Started: YYYY-MM-DD HH:MM UTC
Detected: YYYY-MM-DD HH:MM UTC
Resolved: YYYY-MM-DD HH:MM UTC
Total impact window: HHh MMm
Root-cause fix landed: YYYY-MM-DD HH:MM UTC (PR #…)
Tenants affected: N (enterprise: M)
Authors: name, name

One paragraph: what broke, who was affected, how we fixed it. Plain language — this paragraph is the part most readers will read.

Impact

Customer-visible: what did users / tenant admins experience? Errors, slowness, missing data, charged twice, etc. Be specific about surfaces (chat / dashboard / widget / billing).
Data integrity: any data loss, duplication, or cross-tenant exposure? “No” is a valid answer; say it explicitly.
SLO burn: amount of availability or chat-p95 budget consumed (ADR-0050 §3). Calculate against 30-day window.
Financial / contractual: any SLA credit owed (ADR-0051 once active), any refund processed, any sub-processor cost spike.

Timeline (UTC)

Terse. One line per event. Include detection signals, decisions, deploys, comms, escalations, hand-offs.

HH:MM   First Sentry alert: 5xx spike on POST /api/chat
HH:MM   Responder ack'd page; severity set to P1
HH:MM   /admin/tenants/[id]/health — error rate 42% for tnt_abc, 0% elsewhere
HH:MM   Identified: deploy <sha> introduced a typo in scope check
HH:MM   Reverted <sha>, force-deploy via release-please rollback
HH:MM   Sentry rate dropped to baseline; declared resolved
HH:MM   First tenant comms sent

Include the times you paused to make a decision — those are usually where future-us will want to add tooling.

Root cause

The 5-whys, not the surface symptom.

Why did the chat endpoint 5xx? — The scope-check threw on undefined tenant.apiKeyScopeJson.
Why was it undefined? — A migration added the column but didn’t backfill rows created before it.
Why didn’t the deploy catch this? — The migration lint check doesn’t enforce backfill for nullable columns.
Why is backfill not enforced? — The lint rule was added post-migration-0035; nothing reapplied it retroactively.
Why didn’t a test catch it? — Tests exercise the post-migration shape; there’s no regression suite against pre-migration tenants.

The deepest “why” with a system-fixable answer is the root cause. Don’t stop at “human error”.

What worked

What we did well, so we keep doing it. Examples:

Detection time was good — Sentry alerted within 90s
Containment via revert was the right first move — no need to debug in prod
The on-call template-pinning meant the new responder was up and running in under 5 min

What didn’t

What slowed us down or made the impact worse. Examples:

Sentry alert fired but didn’t include tenant_id filter — had to manually narrow scope (action item: add tag-driven alert routing)
The /admin/tenants/[id]/health view is missing the “recent deploys” panel — had to context-switch to git log
Tenant comms drafted while the fix was deploying — should have been pre-staged

Action items

ID	Description	Owner	Due	Status
AI-1	Add tenant-id filter to Sentry alert	name	YYYY-MM-DD	open
AI-2	Migration lint: enforce backfill for nullable columns added by `feat:` commits	name	YYYY-MM-DD	open
AI-3	Pre-stage tenant comms template for P0 / P1	name	YYYY-MM-DD	open

Action items without an owner and a date don’t exist. Track each one to completion in a follow-up issue or ADR; close them out by editing this file when done.

ADRs touched

Did this incident invalidate or amend any ADR? Note here. New ADRs drafted as part of remediation get linked too.

Sign-off

Engineering: name, date
DPO (only if data-related): name, date

This template is part of the incident-response toolchain (ADR-0050 §5). Improvements land as PRs against this file.