Runbook — Disaster recovery

• Scope: D1 corruption, accidental destructive write, account-level loss. R2 / Vectorize / KV are out of scope (see ADR-0053 §1).
• Source of truth for targets: ADR-0053.
• Drill cadence: quarterly (full restore to puccha-restore-test), annually (full account-migration restore per ADR-0041 §D7).

1. Decide which scenario applies

Before doing anything destructive, decide:

• Live DB reachable, suspected corruption / accidental write within last 30 days → Scenario A (Time Travel restore).
• Live DB unreachable, OR corruption older than 30 days, OR Time Travel restore failed → Scenario B (export-based restore).
• Cloudflare account itself lost / migrating provider → Scenario C (cross-account restore via ADR-0041).

If unsure, default to A first (cheapest, fastest, no data loss). If A doesn’t resolve the issue, escalate to B without re-running A.

2. Pre-flight checks (every scenario)

• Page open with live wrangler access on the right CF account
• Incident declared in /admin/incidents with severity, summary, affected tenants (ADR-0051 Phase 4 — drives the lifecycle bus)
• First responder = incident commander; comms responder named
• CTO + DPO notified for any restore that touches enterprise tenant data

Do NOT begin a restore until the incident is declared. The incident record is the canonical timestamp for “when did the impact start” which feeds the postmortem and any SLA credit calculation.

3. Scenario A — Time Travel restore (RPO ≈ 1 min, RTO ≈ 5 min)

Cloudflare D1 Time Travel keeps point-in-time snapshots for the last 30 days. The live DB is restored to a chosen timestamp.

Time Travel restore is destructive of any live writes since the chosen timestamp. Ensure the impact window is well-understood before running this.

3.1 Identify the target timestamp

The incident timeline (the postmortem you’re filling in alongside this) should have the moment the bad write landed. Subtract a few seconds for safety.

# Example: target = 2026-05-10T14:23:00Z
TARGET="2026-05-10T14:23:00Z"

3.2 Run the restore

cd platform/apps/app
pnpm exec wrangler d1 time-travel restore puccha-db-prod \
  --timestamp "$TARGET" --remote

Wrangler prompts for confirmation; type the DB name to confirm.

3.3 Verify

pnpm exec wrangler d1 execute puccha-db-prod --remote \
  --command "SELECT COUNT(*) FROM tenants"
pnpm exec wrangler d1 execute puccha-db-prod --remote \
  --command "SELECT MAX(ts) FROM audit_log"

The MAX(ts) FROM audit_log should be near the target timestamp. If it’s much later, the restore didn’t apply — re-check the wrangler command and re-run.

3.4 Smoke tests

Run the deploy smoke suite from LAUNCH.md §5 against the live URL. The chat path is the highest-signal smoke test.

4. Scenario B — Export-based restore (RPO 24h, RTO 15 min target)

The daily export at 19:00 UTC (edge cron /api/cron/backup, ADR-0053 Follow-up 3) lives in R2 puccha-backups-prod for 30 days, keyed puccha-db-prod-YYYY-MM-DD.sql.gz. Use this when Time Travel doesn’t help (older than 30 days, or Time Travel itself is failing).

4.1 Find the right backup

Backups are keyed by UTC date. Wrangler 4 has no r2 object list (only get/put), so list what’s present (newest first) with the freshness tool:

bash platform/scripts/backup-status.sh prod

Pick the most recent date that pre-dates the bad event.

4.2 Download + decompress

DATE=2026-05-09  # the date from §4.1
pnpm exec wrangler r2 object get \
  puccha-backups-prod/puccha-db-prod-$DATE.sql.gz \
  --file /tmp/puccha-prod-$DATE.sql.gz --remote
gunzip /tmp/puccha-prod-$DATE.sql.gz

4.3 Create a fresh D1

We restore into a NEW D1, not over the live one — this lets us verify the restored DB before flipping the binding. (The flip is §4.5.)

pnpm exec wrangler d1 create puccha-restore
# Note the database_id from the output — needed in §4.5.

4.4 Apply the SQL dump

pnpm exec wrangler d1 execute puccha-restore --remote \
  --file /tmp/puccha-prod-$DATE.sql

For a million-row tenant this takes ~3–5 min. If the apply fails midway, delete puccha-restore and start over; partial-state recovery is not supported.

4.5 Flip the live binding

Point the prod D1 binding at the restored DB. The durable source of truth is platform/infra/environments.json (the per-env wrangler.prod.toml is generated from it via pnpm gen:wrangler — don’t hand-edit the generated file). Change the prod database_id there, regenerate, then deploy:

[[d1_databases]]
binding = "DB"
database_name = "puccha-db-prod"
database_id = "<old-prod-id>"
database_id = "<puccha-restore-id-from-§4.3>"

Commit the change to a hotfix branch and merge to main — Pages auto-deploys. Do not push to main without the commit message referencing the incident id (fix(dr): restore against incident inc_xxx) — release-please needs the convention.

4.6 Verify + smoke test

Same as §3.3 + §3.4. The smoke tests run against the LIVE URL, which is now backed by puccha-restore. If smoke tests fail, escalate to Scenario C — do not roll back the binding without explicit CTO sign-off.

4.7 After incident close

The old puccha-db-prod DB stays around (in case the restore was imperfect and we need to reach back). Decommission it 14 days after incident resolution if no follow-up issues:

pnpm exec wrangler d1 delete <old-prod-id> --remote

Note the deletion in the postmortem.

5. Scenario C — Cross-account restore

Triggered by: CF account loss, billing dispute, deliberate provider exit. Use the procedure documented in ADR-0041 §D7.

This runbook does NOT duplicate that procedure. The steps there were exercised during the dev/portal account migration; treat them as authoritative.

The pre-flight (§2) still applies. The decommission (§4.7) does not — the old account is the disaster, leave it alone.

6. Known gaps

• Backup-freshness is now checkable via platform/scripts/backup-status.sh <env> (exits non-zero when the newest backup is older than the target age). With --notify it posts a 🔴 to the uptime Google Chat channel (same webhook as infra/uptime/monitor.sh), so it can be scheduled to catch a silently-stopped backup cron, e.g. daily after the 19:00 UTC window:

  bash platform/scripts/backup-status.sh prod --max-age-days 2 --notify

  Still open: pick a scheduler for that line (the local autodeploy daemon is the natural host — it already has repo + wrangler + .env.prod), and note the check only proves a backup exists, not that it restores cleanly — full readability is still only proven by the quarterly drill (§7).

• Schema-version mismatch on restore. If a backup was taken before a destructive migration, restoring it to a NEW D1 produces the OLD schema; current app code expects NEW schema → 500 errors on every endpoint that touches the changed table. Mitigation: apply migrations to puccha-restore BEFORE inserting data whenever the backup pre-dates a recent destructive migration. The migration history is in platform/packages/db/migrations/; pick the migration timestamp that matches the backup’s date.

7. Drills

• Quarterly tabletop (alongside the incident-response runbook drill, ADR-0050): walk through Scenario A end-to-end against puccha-prod test backup. Output: notes on what was unclear, filed as PR comments against this runbook.
• Quarterly live restore to puccha-restore-test: actually run Scenarios A and B against fresh D1 instances. Verify the smoke tests pass. Decommission puccha-restore-test afterward.
• Annual cross-account drill (Scenario C): the harder one, requires coordination with CF support. Skip if the team is < 3 engineers — the drill itself eats a day.

Each drill produces a post-mortem at docs/security/drills/YYYY-MM-DD-dr-{scenario}.md using the postmortem template.

8. Tooling references

Need	Where
List / check backups	bash platform/scripts/backup-status.sh <env> (wrangler 4 has no r2 object list)
Time Travel restore	wrangler d1 time-travel restore <db> --timestamp <iso>
Apply SQL dump	wrangler d1 execute <db> --remote --file <path>
Verify row count	wrangler d1 execute <db> --remote --command "SELECT COUNT(*) FROM tenants"
Decommission D1	wrangler d1 delete <db-id> --remote
Migration history	platform/packages/db/migrations/

---

This runbook supersedes any verbal or chat-based DR process. Where ambiguity exists, the runbook wins.