platform/platform-infra/ansible
Bart Van Geyt 0ade1c740f Phase 5: backup/DR automation (backup Ansible role)
Implements the two decoupled backup streams from docs/06 as an idempotent
Ansible role wired into the host playbook:

- Files: sanoid takes/prunes ZFS snapshots per policy (sanoid_datasets) on
  its packaged timer; syncoid replicates offsite (heleos-zfs-offsite),
  enabled only when zfs_offsite_target is set.
- DB: heleos-db-backup (nightly systemd timer) walks the deployments dir and
  dumps each DB-backed site via `docker exec mariadb-dump` into
  db-backups/<customer>/<site>/{daily,weekly,monthly} with rotation
  (automysqlbackup-style, adapted for the containerized DB; MYSQL_PWD keeps
  the password out of the process list). heleos-db-offsite rsyncs offsite
  when db_offsite_target is set.

Streams and schedules are configured in group_vars/all.yml; offsite is
opt-in via the two target vars. Updates doc 06 (implementation note), the
ansible README, and CLAUDE.md status. YAML + templates validated by render.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 18:04:41 +02:00
..
group_vars Phase 5: backup/DR automation (backup Ansible role) 2026-07-07 18:04:41 +02:00
inventory Phase 1: Ansible host baseline for Ubuntu VM 2026-07-07 12:45:42 +02:00
roles Phase 5: backup/DR automation (backup Ansible role) 2026-07-07 18:04:41 +02:00
ansible.cfg Phase 1: Ansible host baseline for Ubuntu VM 2026-07-07 12:45:42 +02:00
README.md Phase 5: backup/DR automation (backup Ansible role) 2026-07-07 18:04:41 +02:00
requirements.yml Phase 1: Ansible host baseline for Ubuntu VM 2026-07-07 12:45:42 +02:00
site.yml Phase 5: backup/DR automation (backup Ansible role) 2026-07-07 18:04:41 +02:00

Phase 1 — Host baseline (Ansible)

Idempotent host configuration for the heleos platform, targeting Ubuntu 24.04 LTS. Roles: base packages → ZFS pool/datasets → Docker (data-root on ZFS) → nftables firewall + container egress filter → SSH hardening → backup/DR (sanoid snapshots, per-DB dumps, offsite).

Prerequisites

  1. An Ubuntu 24.04 VM you can SSH into as a sudo-capable user.
  2. A dedicated second virtual disk attached to the VM for the ZFS pool (e.g. /dev/sdb or /dev/vdb) — separate from the OS disk.
  3. Ansible installed on your workstation (pipx install ansible or apt).

Configure

cd platform-infra/ansible
ansible-galaxy collection install -r requirements.yml

cp inventory/hosts.yml.example inventory/hosts.yml   # edit host/user (git-ignored)
$EDITOR group_vars/all.yml                            # set zfs_pool_disk, keys, etc.

Key variables in group_vars/all.yml:

Variable Meaning
zfs_pool_disk The dedicated disk for the pool. Its contents will be destroyed.
zfs_pool_force Must be true to create a pool on a non-empty disk (safety gate).
admin_authorized_keys Public keys for the admin — required before disabling passwords.
ssh_disable_password_auth Leave false until key login is verified, then flip to true.
smtp_relay_host Optional; if set, containers may reach SMTP only on this host.

Run

ansible-playbook site.yml --check     # dry run (note: first run can't fully
                                      # check tasks that depend on ZFS/Docker
                                      # not yet present)
ansible-playbook site.yml             # apply

Run a single layer with tags: --tags zfs, --tags docker, --tags firewall, --tags ssh, --tags base.

Safety notes

  • ZFS is destructive: the play refuses to create a pool on a disk that already has a filesystem/partition unless zfs_pool_force: true. Double-check zfs_pool_disk points at the empty spare disk, not the OS disk. For production prefer a stable /dev/disk/by-id/... path over /dev/sdb.
  • SSH lock-out: the play asserts that admin_authorized_keys is non-empty before it will disable password authentication. Verify you can log in with your key before setting ssh_disable_password_auth: true.
  • Firewall coexistence: host inbound rules live in a dedicated inet heleos nftables table and never flush the global ruleset, so Docker's own iptables/nft rules are left intact. Container SMTP egress is blocked via a DOCKER-USER jump applied by the heleos-docker-egress service.

What this sets up

  • ZFS pool tank with platform datasets and the customers parent (see ../../docs/03-naming-conventions.md).
  • Docker Engine + Compose plugin, data-root on tank/platform/docker using the native zfs storage driver; per-site network address pool preconfigured.
  • nftables default-deny inbound (allow SSH/80/443 + established + loopback + ICMP); outbound SMTP blocked from containers.
  • Hardened SSH (key-first, root prohibit-password) and the sftponly group that per-customer SFTP accounts will join in Phase 4.
  • Backup/DR (Phase 5): sanoid snapshot policy + timer, nightly per-database dumps with rotation, and optional offsite zfs send (syncoid) / rsync — enable offsite by setting zfs_offsite_target / db_offsite_target.

Verify after running

zpool status && zfs list
docker info | grep -E 'Storage Driver|Docker Root Dir'
sudo nft list table inet heleos
sudo iptables -S DOCKER-USER