Runbook — Incident Response (general)

• Scope: any production incident on the Puccha platform that isn’t a chat-attachment data breach (which has its own runbook).
• First responder: on-call engineer (currently founder — see oncall-rotation.md)
• Postmortem: mandatory for P0 / P1, optional for P2. Use the postmortem template.

This runbook is the general flow. The breach runbook is a P0 specialisation of §2–§5 here, with PDPA/GDPR notification clocks layered on top.

1. Severity matrix

Severity is set by the first responder within five minutes of acknowledging the page. It can be revised up or down once scope is clearer; the original severity stays in the timeline regardless.

Sev	Definition	Acknowledge	First update	Examples
P0	Multi-tenant data exposure, full platform outage, security compromise of secrets or auth	15 min	30 min	Cross-tenant data leak; all-region 5xx ≥5 min; AUTH_SECRET / Stripe webhook secret leaked; attachment breach (→ specialist runbook)
P1	Single-tenant outage, partial multi-tenant degradation, billing-blocking bug, SLO burn at 2× rate	1 hour	2 hours	One enterprise tenant 100% errors; chat latency p95 >10s tenant-wide; Stripe webhook handler crashing
P2	Feature degraded with workaround, non-billing functional bug, SLO burn within budget	1 business day	next business day	Rerank failures (LLM fallback active); widget delivery latency spike; admin dashboard chart broken
P3	Cosmetic, non-urgent, planned regression	next sprint	n/a	Dead link; minor copy issue; deprecated-API warning

Default to the higher severity if you’re between two tiers — downgrading later costs nothing; missing the response clock costs trust.

2. Roles

For P0 / P1 the responder takes on three roles until they hand off:

• Incident commander — owns the response. Decides when to escalate, when to page additional responders, when to declare resolved. Does not type fixes.
• Operator — types the commands. On a one-person rotation the commander and operator are the same human; on N≥2 they should split as soon as a second responder joins.
• Comms — drafts the timeline-style update for stakeholders. Owns the notification cadence below.

For P2 the operator handles all three. P3 is just a normal ticket.

3. Response phases

3.1 Triage (first 15 min for P0, first 60 min for P1)

1. Acknowledge the page in whatever channel raised it.
2. Set severity (§1). Open the incident timeline file: copy docs/security/runbooks/postmortem-template.md to docs/security/incidents/YYYY-MM-DD-{slug}.md and start writing timestamps as you go. Do not optimise the prose — terse HH:MM action/observation lines are the goal.
3. Determine blast radius: how many tenants affected, what surface (chat / dashboard / widget / billing), is data exposure suspected? The answer drives whether the breach runbook takes over.
4. Pull initial evidence: Sentry errors filtered by tenant: and plan: (ADR-0050 §2 tags), /admin/tenants/[id]/health for any reported tenant, recent deploys from git log --oneline -10.

3.2 Containment

Stop the bleed before you understand it. Acceptable containment patterns, in rough preference order:

1. Revert the bad deploy — git revert <sha> && git push is usually the right move when the timeline correlates with a deploy. The release-please workflow handles the next deploy automatically.
2. Disable the affected feature at the route handler with a targeted 503 (the breach runbook §2.1 has the pattern).
3. Rate-limit harder — drop the WAF rate rule for a specific path from CF Dashboard → Security → WAF → Rate Rules. This buys time without code changes.
4. Rotate a secret — if a secret leak is suspected, rotate via wrangler pages secret put immediately. Update the secret in the relevant external system (Stripe, Anthropic, Cohere) in the same change window.

Containment is not the fix. It is the smallest change that stops harm while you investigate. Note the containment action in the timeline with (temporary) so you remember to revert it after the real fix lands.

3.3 Communication cadence

Severity	Internal updates	External (tenant-facing)
P0	Every 30 min until resolved	Every 60 min until resolved, then a written summary within 24h
P1	Every 2 hours until resolved	Within 4 hours; summary within 48h
P2	At resolution	If the affected tenant noticed, a “we saw this and fixed it” note within 1 business day
P3	At resolution	None

External comms route through the founder until Puccha has a customer-success function. Don’t post to public surfaces (status page, social) without sign-off — see ADR-0051 for when and how that surface comes online.

3.4 Resolution

Declared by the commander when:

• The bleed is stopped (containment in place OR root cause fixed)
• A clear next step exists (either “no further action” or “follow-up PR scheduled”)
• The tenant-facing impact is under control (their side of the issue is visible to them with a path forward)

Resolution is not the same as root-cause fix. Plenty of P0/P1 incidents resolve to “containment held; permanent fix lands tomorrow.”

4. After resolution

4.1 Postmortem (mandatory for P0 / P1)

Within 5 business days of resolution. Use the template. Blameless — the question is “what about the system let this happen”, never “who messed up”. Publish to docs/security/incidents/.

4.2 Action items

Each action item in the postmortem gets:

• An owner (a name, not a team)
• A due date (a calendar date, not “soon”)
• A tracking surface (issue number or ADR draft)

Action items without owners and dates don’t exist. Re-read this paragraph during the postmortem.

4.3 SLO accounting

If the incident burned error budget (ADR-0050 §3 — availability or chat latency p95), record the impact:

docs/security/incidents/YYYY-MM-DD-{slug}.md
  ├── SLO impact: 12m of 5xx affecting tenant tnt_abc123
  ├── Availability burn: 12m / 30d budget = ~0.03% (well within)
  └── No external SLA credit owed (ADR-0051 not yet active)

When ADR-0051 ships and SLA credits become contractual, this section also drives the credit calculation.

5. Drills

• Tabletop quarterly: pick a scenario from the matrix, walk through this runbook out loud. 30 minutes. Output: notes on what was unclear, what was missing, what was over-specified. Drives runbook revisions.
• Live drill annually: synthesised P1 against a staging tenant. The on-call responder must reach §3.4 resolution using only this runbook and the tooling listed in §6.
• Drills produce a post-mortem at docs/security/drills/YYYY-MM-DD.md using the same template — they’re not real incidents but they exercise the same muscles.

6. Tooling references

Need	Where
Errors by tenant	Sentry, filter tenant:tnt_xxx (ADR-0050 §2)
Per-tenant health	/admin/tenants/[id]/health (superadmin + MFA)
Recent deploys	git log --oneline -20 on main
D1 inspection	pnpm exec wrangler d1 execute puccha-dev --remote --command "…"
R2 inspection	pnpm exec wrangler r2 object list puccha-dev-uploads
KV inspection	pnpm exec wrangler kv:key list --binding KV_SESSIONS
Lifecycle timeline	tenant_events table (ADR-0050 §6)
Audit trail	audit_log table (ADR-0021)
Cron forcing	curl -H "Authorization: Bearer $CRON_SECRET" https://puccha-dev.hxlab.io/api/cron/retention

7. When this runbook does NOT apply

• Chat attachment data breach → attachment-breach.md takes precedence; PDPA/GDPR clocks override the §3 cadence
• Pre-production incidents (CI failures, dev-deploy regressions) → treat as a normal PR follow-up; no runbook needed
• External provider outages (Cloudflare-wide, Anthropic API down) — acknowledge, log a P2 in our timeline for visibility, but the containment step is “wait for the provider to fix it”; the comms step is “tell affected tenants which provider”

---

This runbook supersedes any verbal or chat-based incident process. Where ambiguity exists, the runbook wins.