bart/platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
platform/docs/06-backup-and-dr.md
Bart Van Geyt 9f819df4d9 Phase 0: architecture docs, ADRs, and repo scaffold 
Establish the design foundation for the heleosv2 multi-tenant hosting
platform before any implementation code:

- Monorepo skeleton: docs/, platform-infra/, site-templates/,
  deployments/, control-panel/ with orientation READMEs.
- docs/: roadmap index, architecture + threat model, naming conventions,
  site profiles, provisioning workflow, backup & DR runbook, repo/GitOps
  layout, and the approved architecture plan.
- docs/adr/: 9 ADRs recording the rationale for single-host Compose,
  Traefik edge, nginx+fpm split, shared MariaDB, ZFS-per-customer,
  decoupled backup streams, Forgejo, CLI-first, and SFTP-only.
- Secrets hygiene: .gitignore (only *.enc.* committed) and .gitattributes
  (LF for scripts/Dockerfiles/YAML run on the Linux host).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 12:26:00 +02:00

3.7 KiB
Raw Blame History

06 — Backup & Disaster Recovery Runbook

Backups are three decoupled streams, each matched to its data's change pattern (see ADR 0006). This keeps offsite transfer cheap while preserving per-customer restore.

Stream Data Mechanism Onsite Offsite
A Web files ZFS snapshot + zfs send snapshots on tank incremental send to offsite pool
B Databases automysqlbackup (per-DB, rotated) tank/platform/db-backups/<slug> rsync to offsite
C Logs Promtail → Loki Loki store (per Loki retention; not customer-restore data)

Platform state (Traefik ACME store, Forgejo data, deployments repo, Prometheus config) is backed up with the same ZFS mechanism from tank/platform/*.

1. Schedule & retention

  • Stream A (files): frequent snapshots (e.g. hourly + daily), zfs send daily offsite. Retention via a snapshot-pruning policy (keep N hourly, N daily, N weekly).
  • Stream B (DB): automysqlbackup nightly with its native daily/weekly/monthly rotation; rsync offsite right after.
  • Ordering (consistency): run the nightly DB dump immediately before the daily file snapshot so the two are as close in time as possible. The streams are crash-consistent, not atomic — acceptable for PHP/WordPress.

2. Offsite targets

  • Files: a second ZFS host/pool receiving incremental zfs send streams. Incrementals are near-empty for unchanged sites — the reason logs/dumps are kept out of the customer dataset.
  • DB dumps: rsync to the same or another offsite location (delta transfer).
  • Both offsite copies are the recovery source if the primary host is lost.

3. Restore — single customer (the critical drill)

Two coordinated steps:

A. Web files (ZFS)

  1. Identify the target snapshot: tank/customers/<slug>/web@auto-....
  2. Restore by clone/rollback (or zfs receive from offsite if the host is gone) into tank/customers/<slug>/web.

B. Database (dump) 3. Pick the matching dump from db-backups/<slug> (or offsite). 4. Recreate db_<slug> + u_<slug> if needed; import the dump into shared MariaDB.

C. Bring up 5. docker compose up -d in deployments/<slug>/. 6. Verify site + data; for WordPress confirm site URL / run WP-CLI search-replace if the domain changed.

CLI: restore <slug> --snapshot <name> --db <dump-file> wraps A–C.

4. Restore — full host (DR rebuild)

  1. Provision a fresh host from platform-infra Ansible (Docker, ZFS, firewall).
  2. zfs receive the platform datasets (Traefik/Forgejo/MariaDB datadir) and all tank/customers/* from offsite.
  3. Restore DB dumps as needed (or rely on the received MariaDB datadir, then reconcile with latest dumps).
  4. Bring platform services up (Traefik, MariaDB, Forgejo, monitoring).
  5. docker compose up -d per site from the deployments repo.
  6. Repoint DNS if the host address changed.

5. Verification drills (run regularly, not just once)

  • Single-customer restore to a scratch location; site + data come back; neighbours untouched.
  • Unchanged-site incremental is near-empty — confirms the lean-dataset goal (ADR 0005) is actually holding.
  • DB dump imports cleanly and row counts match expectations.
  • Offsite zfs receive of a customer dataset succeeds on the DR target.
  • Full-host rebuild rehearsed at least once end-to-end.

6. What is NOT backed up here

  • Container images — rebuildable from site-templates + registry.
  • Rendered runtime state (.state/, container-local data/) — regenerated.
  • Loki logs are operational telemetry, not customer-restore data.