e5ff798dd4
Adopt a two-level ownership model: a customer owns many sites, with isolation/containers/DB/backup at the site level and grouping (one SFTP login, recursive backup, billing, bulk delete) at the customer level. - 03-naming-conventions: rewritten for customer/site ids and slug = <customer>-<site>; nested ZFS, per-customer SFTP chroot, deployments mirror the nesting. - Propagated paths through docs 01, 04, 05, 06, 07 and deployments/README. - ADR 0005 retitled/updated to per-site datasets nested under customer. - architecture-plan.md: note pointing to doc 03 as authoritative on naming. Clarifies subdomains: same-app subdomains are aliases on one site; a separate-app subdomain is its own isolated site under the same customer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
83 lines
3.9 KiB
Markdown
83 lines
3.9 KiB
Markdown
# 06 — Backup & Disaster Recovery Runbook
|
||
|
||
Backups are **three decoupled streams**, each matched to its data's change
|
||
pattern (see [ADR 0006](adr/0006-decoupled-backup-streams.md)). This keeps
|
||
offsite transfer cheap while preserving per-customer restore.
|
||
|
||
| Stream | Data | Mechanism | Onsite | Offsite |
|
||
|--------|------|-----------|--------|---------|
|
||
| A | Web files | ZFS snapshot + `zfs send` | snapshots on `tank` | incremental `send` to offsite pool |
|
||
| B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<customer>/<site>` | rsync to offsite |
|
||
| C | Logs | Promtail → Loki | Loki store | (per Loki retention; not customer-restore data) |
|
||
|
||
Platform state (Traefik ACME store, Forgejo data, `deployments` repo, Prometheus
|
||
config) is backed up with the same ZFS mechanism from `tank/platform/*`.
|
||
|
||
## 1. Schedule & retention
|
||
|
||
- **Stream A (files):** frequent snapshots (e.g. hourly + daily), `zfs send`
|
||
daily offsite. Retention via a snapshot-pruning policy (keep N hourly, N
|
||
daily, N weekly).
|
||
- **Stream B (DB):** automysqlbackup nightly with its native daily/weekly/monthly
|
||
rotation; rsync offsite right after.
|
||
- **Ordering (consistency):** run the nightly DB dump **immediately before** the
|
||
daily file snapshot so the two are as close in time as possible. The streams
|
||
are **crash-consistent, not atomic** — acceptable for PHP/WordPress.
|
||
|
||
## 2. Offsite targets
|
||
|
||
- **Files:** a second ZFS host/pool receiving incremental `zfs send` streams.
|
||
Incrementals are near-empty for unchanged sites — the reason logs/dumps are
|
||
kept out of the customer dataset.
|
||
- **DB dumps:** rsync to the same or another offsite location (delta transfer).
|
||
- Both offsite copies are the recovery source if the primary host is lost.
|
||
|
||
## 3. Restore — single site (the critical drill)
|
||
|
||
Two coordinated steps:
|
||
|
||
**A. Web files (ZFS)**
|
||
1. Identify the target snapshot: `tank/customers/<customer>/<site>/web@auto-...`.
|
||
2. Restore by clone/rollback (or `zfs receive` from offsite if the host is
|
||
gone) into `tank/customers/<customer>/<site>/web`.
|
||
|
||
**B. Database (dump)**
|
||
3. Pick the matching dump from `db-backups/<customer>/<site>` (or offsite).
|
||
4. Recreate `db_<customer>_<site>` + `u_<customer>_<site>` if needed; import the
|
||
dump into shared MariaDB.
|
||
|
||
**C. Bring up**
|
||
5. `docker compose up -d` in `deployments/<customer>/<site>/`.
|
||
6. Verify site + data; for WordPress confirm site URL / run WP-CLI
|
||
`search-replace` if the domain changed.
|
||
|
||
CLI: `restore <customer> <site> --snapshot <name> --db <dump-file>` wraps A–C.
|
||
|
||
> To restore an entire customer at once, repeat A–C per site, or use the
|
||
> recursive snapshot `tank/customers/<customer>@...` as the file source.
|
||
|
||
## 4. Restore — full host (DR rebuild)
|
||
|
||
1. Provision a fresh host from `platform-infra` Ansible (Docker, ZFS, firewall).
|
||
2. `zfs receive` the platform datasets (Traefik/Forgejo/MariaDB datadir) and all
|
||
`tank/customers/*` from offsite.
|
||
3. Restore DB dumps as needed (or rely on the received MariaDB datadir, then
|
||
reconcile with latest dumps).
|
||
4. Bring platform services up (Traefik, MariaDB, Forgejo, monitoring).
|
||
5. `docker compose up -d` per site from the `deployments` repo.
|
||
6. Repoint DNS if the host address changed.
|
||
|
||
## 5. Verification drills (run regularly, not just once)
|
||
|
||
- [ ] **Single-customer restore** to a scratch location; site + data come back;
|
||
neighbours untouched.
|
||
- [ ] **Unchanged-site incremental is near-empty** — confirms the lean-dataset
|
||
goal ([ADR 0005](adr/0005-zfs-per-customer.md)) is actually holding.
|
||
- [ ] **DB dump imports cleanly** and row counts match expectations.
|
||
- [ ] **Offsite `zfs receive`** of a customer dataset succeeds on the DR target.
|
||
- [ ] **Full-host rebuild** rehearsed at least once end-to-end.
|
||
|
||
## 6. What is NOT backed up here
|
||
- Container images — rebuildable from `site-templates` + registry.
|
||
- Rendered runtime state (`.state/`, container-local `data/`) — regenerated.
|
||
- Loki logs are operational telemetry, not customer-restore data.
|