bart/platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
platform/docs/06-backup-and-dr.md
Bart Van Geyt 0ade1c740f Phase 5: backup/DR automation (backup Ansible role) 
Implements the two decoupled backup streams from docs/06 as an idempotent
Ansible role wired into the host playbook:

- Files: sanoid takes/prunes ZFS snapshots per policy (sanoid_datasets) on
  its packaged timer; syncoid replicates offsite (heleos-zfs-offsite),
  enabled only when zfs_offsite_target is set.
- DB: heleos-db-backup (nightly systemd timer) walks the deployments dir and
  dumps each DB-backed site via `docker exec mariadb-dump` into
  db-backups/<customer>/<site>/{daily,weekly,monthly} with rotation
  (automysqlbackup-style, adapted for the containerized DB; MYSQL_PWD keeps
  the password out of the process list). heleos-db-offsite rsyncs offsite
  when db_offsite_target is set.

Streams and schedules are configured in group_vars/all.yml; offsite is
opt-in via the two target vars. Updates doc 06 (implementation note), the
ansible README, and CLAUDE.md status. YAML + templates validated by render.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 18:04:41 +02:00

4.7 KiB
Raw Permalink Blame History

06 — Backup & Disaster Recovery Runbook

Backups are three decoupled streams, each matched to its data's change pattern (see ADR 0006). This keeps offsite transfer cheap while preserving per-customer restore.

Stream Data Mechanism Onsite Offsite
A Web files ZFS snapshot + zfs send snapshots on tank incremental send to offsite pool
B Databases automysqlbackup (per-DB, rotated) tank/platform/db-backups/<customer>/<site> rsync to offsite
C Logs Promtail → Loki Loki store (per Loki retention; not customer-restore data)

Platform state (Traefik ACME store, Forgejo data, deployments repo, Prometheus config) is backed up with the same ZFS mechanism from tank/platform/*.

1. Schedule & retention

  • Stream A (files): frequent snapshots (e.g. hourly + daily), zfs send daily offsite. Retention via a snapshot-pruning policy (keep N hourly, N daily, N weekly).
  • Stream B (DB): automysqlbackup nightly with its native daily/weekly/monthly rotation; rsync offsite right after.
  • Ordering (consistency): run the nightly DB dump immediately before the daily file snapshot so the two are as close in time as possible. The streams are crash-consistent, not atomic — acceptable for PHP/WordPress.

2. Offsite targets

  • Files: a second ZFS host/pool receiving incremental zfs send streams. Incrementals are near-empty for unchanged sites — the reason logs/dumps are kept out of the customer dataset.
  • DB dumps: rsync to the same or another offsite location (delta transfer).
  • Both offsite copies are the recovery source if the primary host is lost.

2b. Implementation (Phase 5)

Automated by the backup Ansible role (platform-infra/ansible/roles/backup):

  • Files: sanoid takes and prunes snapshots per policy (sanoid.conf, driven by sanoid_datasets), on the packaged sanoid.timer. syncoid replicates offsite (heleos-zfs-offsite), enabled only when zfs_offsite_target is set.
  • DB: heleos-db-backup (systemd timer, nightly) walks the deployments dir, dumps each DB-backed site via docker exec … mariadb-dump into db-backups/<customer>/<site>/{daily,weekly,monthly} with rotation, then heleos-db-offsite rsyncs offsite (enabled only when db_offsite_target set).
  • Toggle streams with backup_snapshots_enabled / db_backup_enabled and the offsite target vars in group_vars/all.yml.

3. Restore — single site (the critical drill)

Two coordinated steps:

A. Web files (ZFS)

  1. Identify the target snapshot: tank/customers/<customer>/<site>/web@auto-....
  2. Restore by clone/rollback (or zfs receive from offsite if the host is gone) into tank/customers/<customer>/<site>/web.

B. Database (dump) 3. Pick the matching dump from db-backups/<customer>/<site> (or offsite). 4. Recreate db_<customer>_<site> + u_<customer>_<site> if needed; import the dump into shared MariaDB.

C. Bring up 5. docker compose up -d in deployments/<customer>/<site>/. 6. Verify site + data; for WordPress confirm site URL / run WP-CLI search-replace if the domain changed.

CLI: restore <customer> <site> --snapshot <name> --db <dump-file> wraps A–C.

To restore an entire customer at once, repeat A–C per site, or use the recursive snapshot tank/customers/<customer>@... as the file source.

4. Restore — full host (DR rebuild)

  1. Provision a fresh host from platform-infra Ansible (Docker, ZFS, firewall).
  2. zfs receive the platform datasets (Traefik/Forgejo/MariaDB datadir) and all tank/customers/* from offsite.
  3. Restore DB dumps as needed (or rely on the received MariaDB datadir, then reconcile with latest dumps).
  4. Bring platform services up (Traefik, MariaDB, Forgejo, monitoring).
  5. docker compose up -d per site from the deployments repo.
  6. Repoint DNS if the host address changed.

5. Verification drills (run regularly, not just once)

  • Single-customer restore to a scratch location; site + data come back; neighbours untouched.
  • Unchanged-site incremental is near-empty — confirms the lean-dataset goal (ADR 0005) is actually holding.
  • DB dump imports cleanly and row counts match expectations.
  • Offsite zfs receive of a customer dataset succeeds on the DR target.
  • Full-host rebuild rehearsed at least once end-to-end.

6. What is NOT backed up here

  • Container images — rebuildable from site-templates + registry.
  • Rendered runtime state (.state/, container-local data/) — regenerated.
  • Loki logs are operational telemetry, not customer-restore data.