platform/docs/06-backup-and-dr.md

# 06 — Backup & Disaster Recovery Runbook

Backups are **three decoupled streams**, each matched to its data's change
pattern (see [ADR 0006](adr/0006-decoupled-backup-streams.md)). This keeps
offsite transfer cheap while preserving per-customer restore.

| Stream | Data | Mechanism | Onsite | Offsite |
|--------|------|-----------|--------|---------|
| A | Web files | ZFS snapshot + `zfs send` | snapshots on `tank` | incremental `send` to offsite pool |
| B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<customer>/<site>` | rsync to offsite |
| C | Logs | Promtail → Loki | Loki store | (per Loki retention; not customer-restore data) |

Platform state (Traefik ACME store, Forgejo data, `deployments` repo, Prometheus
config) is backed up with the same ZFS mechanism from `tank/platform/*`.

## 1. Schedule & retention

- **Stream A (files):** frequent snapshots (e.g. hourly + daily), `zfs send`
  daily offsite. Retention via a snapshot-pruning policy (keep N hourly, N
  daily, N weekly).
- **Stream B (DB):** automysqlbackup nightly with its native daily/weekly/monthly
  rotation; rsync offsite right after.
- **Ordering (consistency):** run the nightly DB dump **immediately before** the
  daily file snapshot so the two are as close in time as possible. The streams
  are **crash-consistent, not atomic** — acceptable for PHP/WordPress.

## 2. Offsite targets

- **Files:** a second ZFS host/pool receiving incremental `zfs send` streams.
  Incrementals are near-empty for unchanged sites — the reason logs/dumps are
  kept out of the customer dataset.
- **DB dumps:** rsync to the same or another offsite location (delta transfer).
- Both offsite copies are the recovery source if the primary host is lost.

## 2b. Implementation (Phase 5)

Automated by the `backup` Ansible role (`platform-infra/ansible/roles/backup`):

- **Files:** [sanoid](https://github.com/jimsalterjrs/sanoid) takes and prunes
  snapshots per policy (`sanoid.conf`, driven by `sanoid_datasets`), on the
  packaged `sanoid.timer`. **syncoid** replicates offsite
  (`heleos-zfs-offsite`), enabled only when `zfs_offsite_target` is set.
- **DB:** `heleos-db-backup` (systemd timer, nightly) walks the deployments dir,
  dumps each DB-backed site via `docker exec … mariadb-dump` into
  `db-backups/<customer>/<site>/{daily,weekly,monthly}` with rotation, then
  `heleos-db-offsite` rsyncs offsite (enabled only when `db_offsite_target` set).
- Toggle streams with `backup_snapshots_enabled` / `db_backup_enabled` and the
  offsite target vars in `group_vars/all.yml`.

## 3. Restore — single site (the critical drill)

Two coordinated steps:

**A. Web files (ZFS)**
1. Identify the target snapshot: `tank/customers/<customer>/<site>/web@auto-...`.
2. Restore by clone/rollback (or `zfs receive` from offsite if the host is
   gone) into `tank/customers/<customer>/<site>/web`.

**B. Database (dump)**
3. Pick the matching dump from `db-backups/<customer>/<site>` (or offsite).
4. Recreate `db_<customer>_<site>` + `u_<customer>_<site>` if needed; import the
   dump into shared MariaDB.

**C. Bring up**
5. `docker compose up -d` in `deployments/<customer>/<site>/`.
6. Verify site + data; for WordPress confirm site URL / run WP-CLI
   `search-replace` if the domain changed.

CLI: `restore <customer> <site> --snapshot <name> --db <dump-file>` wraps A–C.

> To restore an entire customer at once, repeat A–C per site, or use the
> recursive snapshot `tank/customers/<customer>@...` as the file source.

## 4. Restore — full host (DR rebuild)

1. Provision a fresh host from `platform-infra` Ansible (Docker, ZFS, firewall).
2. `zfs receive` the platform datasets (Traefik/Forgejo/MariaDB datadir) and all
   `tank/customers/*` from offsite.
3. Restore DB dumps as needed (or rely on the received MariaDB datadir, then
   reconcile with latest dumps).
4. Bring platform services up (Traefik, MariaDB, Forgejo, monitoring).
5. `docker compose up -d` per site from the `deployments` repo.
6. Repoint DNS if the host address changed.

## 5. Verification drills (run regularly, not just once)

- [ ] **Single-customer restore** to a scratch location; site + data come back;
      neighbours untouched.
- [ ] **Unchanged-site incremental is near-empty** — confirms the lean-dataset
      goal ([ADR 0005](adr/0005-zfs-per-customer.md)) is actually holding.
- [ ] **DB dump imports cleanly** and row counts match expectations.
- [ ] **Offsite `zfs receive`** of a customer dataset succeeds on the DR target.
- [ ] **Full-host rebuild** rehearsed at least once end-to-end.

## 6. What is NOT backed up here
- Container images — rebuildable from `site-templates` + registry.
- Rendered runtime state (`.state/`, container-local `data/`) — regenerated.
- Loki logs are operational telemetry, not customer-restore data.