Runbook — On-call rotation
Honest status (2026-05-10): Puccha is a one-person engineering team. The “rotation” is a rotation of one. This document captures that reality so it’s planned-for, not pretended-around. When the team hits N≥2 engineers, the §3 rotation matrix gets filled in and this note gets deleted.
1. Current setup (N=1)
Section titled “1. Current setup (N=1)”- Primary on-call: founder (24/7, every day)
- Escalation contact: none; escalation is to self
- Acknowledgement clock for P0: 15 min. With one human this means P0 outside business hours has a hard floor at “founder is awake and checks Sentry within 15 min of the page firing.” Phone push notification is the minimum (ADR-0048 ships this).
- Coverage gaps: travel, sleep, illness. Documented as known. The
three CF-level mitigations that hold even when the on-call is
unreachable:
- CF WAF rate rules (Layer 1, ADR-0017 §4a) — auto-blocks abuse
- KV per-session rate limit (Layer 2) — auto-blocks per-session abuse
- AI Gateway monthly cost cap (Layer 4) — auto-blocks cost runaway
- These don’t recover from a correctness outage. They cap the cost and blast radius until a human is available.
2. Pager surfaces
Section titled “2. Pager surfaces”What pages the on-call:
| Surface | Trigger | Notify via |
|---|---|---|
| Sentry alert | error rate spike, new high-volume issue | Email + ADR-0048 Web Push |
| CF Health Check | /health returns non-200 ≥ 3 consecutive checks |
|
| Stripe webhook failures | repeated webhook delivery failures | Stripe dashboard email |
| External report | tenant emails support, security researcher | Email inbox |
tenant_events quota_exceeded |
enterprise tenant burns daily quota | Future: queue-driven email; today: visible in /admin |
Not yet wired (future work, not blocking on N=1):
- PagerDuty / Grafana OnCall / Discord webhook integration — tool selection deferred until N≥2 (ADR-0050 §“Non-goals”)
- Phone call escalation
- “Acknowledged but no progress in T minutes” auto-escalation
3. Rotation matrix (placeholder for N≥2)
Section titled “3. Rotation matrix (placeholder for N≥2)”When the team grows past one engineer, populate this table:
| Week of | Primary | Secondary | Notes |
|---|---|---|---|
| YYYY-MM-DD | name | name | first week of rotation; primary takes the comms role |
| … | … | … | … |
Defaults to revisit at that point:
- Rotation length: 1 week (industry standard for small teams; longer becomes a burnout risk)
- Hand-off ritual: 15-min sync at week start covering open incidents, any acknowledged-not-resolved alerts, deploy windows in the coming week
- Compensation: TBD with HR — at small scale, expect comp-time or a flat per-week stipend
4. What the on-call is NOT expected to do
Section titled “4. What the on-call is NOT expected to do”Important to write down so the role doesn’t drift:
- Build features. The on-call shift is reactive; feature work happens off-shift.
- Polish runbooks. Improvements land as a normal PR against this directory, not as part of an active shift.
- Tenant support tickets. Those go through the support inbox; the on-call only takes the ones flagged P0 / P1 by support.
- Reach into another team member’s code without coordination — except during an active incident, where the incident-response runbook governs.
5. Hand-off when the on-call goes off-shift (N≥2 future)
Section titled “5. Hand-off when the on-call goes off-shift (N≥2 future)”- Open incidents: documented status in
docs/security/incidents/YYYY-MM-DD-{slug}.md, owner reassigned with a note in the timeline - Acknowledged-but-not-resolved alerts: list in the hand-off sync
- Pre-staged comms (drafts that haven’t been sent): linked from the incident timeline
- Deploys planned for the next week: noted in the team channel so the incoming on-call isn’t surprised
For N=1, the hand-off is mental — no team to hand off to. Write it down anyway during multi-day incidents, future-you will thank present-you.
6. Drills
Section titled “6. Drills”The on-call participates in the quarterly tabletop drill (ADR-0050 §5, incident-response.md §5). At N=1 the drill is a self-walk-through, not a team exercise; the output (notes on what was unclear) is still valuable.
When N≥2: delete the §1 honest-status note, populate §3, and add a “Last rotation reviewed: YYYY-MM-DD” line at the top.