9f819df4d9
Establish the design foundation for the heleosv2 multi-tenant hosting platform before any implementation code: - Monorepo skeleton: docs/, platform-infra/, site-templates/, deployments/, control-panel/ with orientation READMEs. - docs/: roadmap index, architecture + threat model, naming conventions, site profiles, provisioning workflow, backup & DR runbook, repo/GitOps layout, and the approved architecture plan. - docs/adr/: 9 ADRs recording the rationale for single-host Compose, Traefik edge, nginx+fpm split, shared MariaDB, ZFS-per-customer, decoupled backup streams, Forgejo, CLI-first, and SFTP-only. - Secrets hygiene: .gitignore (only *.enc.* committed) and .gitattributes (LF for scripts/Dockerfiles/YAML run on the Linux host). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
80 lines
3.7 KiB
Markdown
80 lines
3.7 KiB
Markdown
# 06 — Backup & Disaster Recovery Runbook
|
||
|
||
Backups are **three decoupled streams**, each matched to its data's change
|
||
pattern (see [ADR 0006](adr/0006-decoupled-backup-streams.md)). This keeps
|
||
offsite transfer cheap while preserving per-customer restore.
|
||
|
||
| Stream | Data | Mechanism | Onsite | Offsite |
|
||
|--------|------|-----------|--------|---------|
|
||
| A | Web files | ZFS snapshot + `zfs send` | snapshots on `tank` | incremental `send` to offsite pool |
|
||
| B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<slug>` | rsync to offsite |
|
||
| C | Logs | Promtail → Loki | Loki store | (per Loki retention; not customer-restore data) |
|
||
|
||
Platform state (Traefik ACME store, Forgejo data, `deployments` repo, Prometheus
|
||
config) is backed up with the same ZFS mechanism from `tank/platform/*`.
|
||
|
||
## 1. Schedule & retention
|
||
|
||
- **Stream A (files):** frequent snapshots (e.g. hourly + daily), `zfs send`
|
||
daily offsite. Retention via a snapshot-pruning policy (keep N hourly, N
|
||
daily, N weekly).
|
||
- **Stream B (DB):** automysqlbackup nightly with its native daily/weekly/monthly
|
||
rotation; rsync offsite right after.
|
||
- **Ordering (consistency):** run the nightly DB dump **immediately before** the
|
||
daily file snapshot so the two are as close in time as possible. The streams
|
||
are **crash-consistent, not atomic** — acceptable for PHP/WordPress.
|
||
|
||
## 2. Offsite targets
|
||
|
||
- **Files:** a second ZFS host/pool receiving incremental `zfs send` streams.
|
||
Incrementals are near-empty for unchanged sites — the reason logs/dumps are
|
||
kept out of the customer dataset.
|
||
- **DB dumps:** rsync to the same or another offsite location (delta transfer).
|
||
- Both offsite copies are the recovery source if the primary host is lost.
|
||
|
||
## 3. Restore — single customer (the critical drill)
|
||
|
||
Two coordinated steps:
|
||
|
||
**A. Web files (ZFS)**
|
||
1. Identify the target snapshot: `tank/customers/<slug>/web@auto-...`.
|
||
2. Restore by clone/rollback (or `zfs receive` from offsite if the host is
|
||
gone) into `tank/customers/<slug>/web`.
|
||
|
||
**B. Database (dump)**
|
||
3. Pick the matching dump from `db-backups/<slug>` (or offsite).
|
||
4. Recreate `db_<slug>` + `u_<slug>` if needed; import the dump into shared
|
||
MariaDB.
|
||
|
||
**C. Bring up**
|
||
5. `docker compose up -d` in `deployments/<slug>/`.
|
||
6. Verify site + data; for WordPress confirm site URL / run WP-CLI
|
||
`search-replace` if the domain changed.
|
||
|
||
CLI: `restore <slug> --snapshot <name> --db <dump-file>` wraps A–C.
|
||
|
||
## 4. Restore — full host (DR rebuild)
|
||
|
||
1. Provision a fresh host from `platform-infra` Ansible (Docker, ZFS, firewall).
|
||
2. `zfs receive` the platform datasets (Traefik/Forgejo/MariaDB datadir) and all
|
||
`tank/customers/*` from offsite.
|
||
3. Restore DB dumps as needed (or rely on the received MariaDB datadir, then
|
||
reconcile with latest dumps).
|
||
4. Bring platform services up (Traefik, MariaDB, Forgejo, monitoring).
|
||
5. `docker compose up -d` per site from the `deployments` repo.
|
||
6. Repoint DNS if the host address changed.
|
||
|
||
## 5. Verification drills (run regularly, not just once)
|
||
|
||
- [ ] **Single-customer restore** to a scratch location; site + data come back;
|
||
neighbours untouched.
|
||
- [ ] **Unchanged-site incremental is near-empty** — confirms the lean-dataset
|
||
goal ([ADR 0005](adr/0005-zfs-per-customer.md)) is actually holding.
|
||
- [ ] **DB dump imports cleanly** and row counts match expectations.
|
||
- [ ] **Offsite `zfs receive`** of a customer dataset succeeds on the DR target.
|
||
- [ ] **Full-host rebuild** rehearsed at least once end-to-end.
|
||
|
||
## 6. What is NOT backed up here
|
||
- Container images — rebuildable from `site-templates` + registry.
|
||
- Rendered runtime state (`.state/`, container-local `data/`) — regenerated.
|
||
- Loki logs are operational telemetry, not customer-restore data.
|