bart/platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
platform/docs/06-backup-and-dr.md
Bart Van Geyt e5ff798dd4 Refine to nested customer -> site model 
Adopt a two-level ownership model: a customer owns many sites, with
isolation/containers/DB/backup at the site level and grouping (one SFTP
login, recursive backup, billing, bulk delete) at the customer level.

- 03-naming-conventions: rewritten for customer/site ids and slug =
  <customer>-<site>; nested ZFS, per-customer SFTP chroot, deployments
  mirror the nesting.
- Propagated paths through docs 01, 04, 05, 06, 07 and deployments/README.
- ADR 0005 retitled/updated to per-site datasets nested under customer.
- architecture-plan.md: note pointing to doc 03 as authoritative on naming.

Clarifies subdomains: same-app subdomains are aliases on one site; a
separate-app subdomain is its own isolated site under the same customer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 12:37:39 +02:00

3.9 KiB
Raw Blame History

06 — Backup & Disaster Recovery Runbook

Backups are three decoupled streams, each matched to its data's change pattern (see ADR 0006). This keeps offsite transfer cheap while preserving per-customer restore.

Stream Data Mechanism Onsite Offsite
A Web files ZFS snapshot + zfs send snapshots on tank incremental send to offsite pool
B Databases automysqlbackup (per-DB, rotated) tank/platform/db-backups/<customer>/<site> rsync to offsite
C Logs Promtail → Loki Loki store (per Loki retention; not customer-restore data)

Platform state (Traefik ACME store, Forgejo data, deployments repo, Prometheus config) is backed up with the same ZFS mechanism from tank/platform/*.

1. Schedule & retention

  • Stream A (files): frequent snapshots (e.g. hourly + daily), zfs send daily offsite. Retention via a snapshot-pruning policy (keep N hourly, N daily, N weekly).
  • Stream B (DB): automysqlbackup nightly with its native daily/weekly/monthly rotation; rsync offsite right after.
  • Ordering (consistency): run the nightly DB dump immediately before the daily file snapshot so the two are as close in time as possible. The streams are crash-consistent, not atomic — acceptable for PHP/WordPress.

2. Offsite targets

  • Files: a second ZFS host/pool receiving incremental zfs send streams. Incrementals are near-empty for unchanged sites — the reason logs/dumps are kept out of the customer dataset.
  • DB dumps: rsync to the same or another offsite location (delta transfer).
  • Both offsite copies are the recovery source if the primary host is lost.

3. Restore — single site (the critical drill)

Two coordinated steps:

A. Web files (ZFS)

  1. Identify the target snapshot: tank/customers/<customer>/<site>/web@auto-....
  2. Restore by clone/rollback (or zfs receive from offsite if the host is gone) into tank/customers/<customer>/<site>/web.

B. Database (dump) 3. Pick the matching dump from db-backups/<customer>/<site> (or offsite). 4. Recreate db_<customer>_<site> + u_<customer>_<site> if needed; import the dump into shared MariaDB.

C. Bring up 5. docker compose up -d in deployments/<customer>/<site>/. 6. Verify site + data; for WordPress confirm site URL / run WP-CLI search-replace if the domain changed.

CLI: restore <customer> <site> --snapshot <name> --db <dump-file> wraps A–C.

To restore an entire customer at once, repeat A–C per site, or use the recursive snapshot tank/customers/<customer>@... as the file source.

4. Restore — full host (DR rebuild)

  1. Provision a fresh host from platform-infra Ansible (Docker, ZFS, firewall).
  2. zfs receive the platform datasets (Traefik/Forgejo/MariaDB datadir) and all tank/customers/* from offsite.
  3. Restore DB dumps as needed (or rely on the received MariaDB datadir, then reconcile with latest dumps).
  4. Bring platform services up (Traefik, MariaDB, Forgejo, monitoring).
  5. docker compose up -d per site from the deployments repo.
  6. Repoint DNS if the host address changed.

5. Verification drills (run regularly, not just once)

  • Single-customer restore to a scratch location; site + data come back; neighbours untouched.
  • Unchanged-site incremental is near-empty — confirms the lean-dataset goal (ADR 0005) is actually holding.
  • DB dump imports cleanly and row counts match expectations.
  • Offsite zfs receive of a customer dataset succeeds on the DR target.
  • Full-host rebuild rehearsed at least once end-to-end.

6. What is NOT backed up here

  • Container images — rebuildable from site-templates + registry.
  • Rendered runtime state (.state/, container-local data/) — regenerated.
  • Loki logs are operational telemetry, not customer-restore data.