platform/docs/architecture-plan.md

Rebuild: Multi-Tenant Web Hosting Platform (heleosv2)

Note: This is the point-in-time approved plan. Where it refers to the per-customer boundary/naming, the design was later refined to a nested customer → site model — see 03-naming-conventions.md, which is authoritative for naming and paths.

Context

An existing small web-hosting business runs nginx + PHP-FPM + MySQL on a shared host, serving a mix of custom PHP, WordPress, static HTML, and redirect-only sites. The goal is to rebuild on modern best practices with isolation, clean backups, and repeatable provisioning as first principles.

Core philosophy: the customer is the boundary. Isolation (containers), backup (ZFS dataset), and restore all line up on that same boundary, so a compromised or broken site is contained and can be restored without touching neighbours.

This document is an architecture + phased task plan, not a code change. No application code exists yet (greenfield). Implementation happens step-by-step after this plan is approved.

Decisions locked with the user

• Host topology: single bare-metal host → Docker Compose per customer, no orchestrator, Ansible for host config, Terraform not needed initially.
• Control panel: CLI/templating tooling first; customer-facing panel later once the platform is proven.
• Database: one shared MariaDB instance (per-site DB + per-site user). Density over per-instance isolation.
• DevOps stack: lightweight — Forgejo + built-in registry + Forgejo Actions/Woodpecker + Trivy.

Reframing carried into the design

Containers are an isolation boundary, not a hard security boundary (shared kernel). The real protections come from: non-root PHP-FPM, read-only rootfs where possible, per-customer Docker networks, egress filtering, least-privilege DB users, and (later) runtime detection. Docker alone is not the security story.

---

Target Architecture

Edge & routing

• Traefik as the single edge: TLS termination, automatic Let's Encrypt (ACME), dynamic label-based routing. One shared proxy Docker network.

• Traefik cannot speak FastCGI, so each PHP site runs its own small web server behind Traefik:

  Traefik (TLS + routing) → per-site nginx → per-site php-fpm

• Static/redirect sites are served directly by Traefik or a tiny nginx — no PHP.

Per-customer stack (one Docker Compose project per customer/site)

Standard site profiles, each a compose template:

• wordpress — nginx + php-fpm (+ shared-DB credentials)
• custom-php — nginx + php-fpm (+ optional shared-DB credentials)
• static — nginx (or Traefik direct), no PHP, no DB
• redirect — Traefik router rule / tiny nginx, no PHP, no DB

Per-customer isolation:

• Own Docker bridge network; only Traefik joins both proxy and the customer network, so customer containers are unreachable except through Traefik.
• PHP-FPM runs non-root; rootfs read-only where the profile allows; web root is the only writable mount.

Storage — ZFS

• Dataset layout, e.g.: tank/customers/<customer>/web (web root only — kept lean) and tank/platform/{docker,mariadb,db-backups,traefik,forgejo,monitoring}.
• Docker images/layers on their own platform dataset; customer web data bind-mounted from per-customer datasets (clean per-customer snapshots).
• The customer dataset deliberately holds only web files — no logs, no DB dumps — so an unchanged site produces a near-empty incremental and offsite zfs send stays cheap.
• Snapshots per customer on a schedule; zfs send offsite for backup/DR.

Database (shared MariaDB) — decoupled backup stream

• One MariaDB container; each site gets its own database + least-privilege user.
• Live datadir lives on a platform dataset (not inside any customer dataset).
• DB backups are a separate stream from web files (not folded into the customer ZFS dataset — that would make every incremental zfs send ship a fresh dump even for static sites):
  • automysqlbackup produces per-database dumps with daily/weekly/monthly rotation into tank/platform/db-backups/<customer> — per-customer restore granularity for free.
  • rsync those dumps offsite (lean; only changed dumps move). Optional later: put db-backups on its own dataset and zfs send it if checksummed/ immutable offsite is wanted.
• Consistency: schedule the nightly DB dump close to the ZFS snapshot. The two streams aren't atomically consistent, but the small window is harmless for PHP/WordPress (files-on-disk + DB-rows).

File access

• SFTP only (drop FTP — plaintext). Each customer chrooted to their dataset.
• Optional web file manager (e.g. Filebrowser) added with the panel later.

Network security

• Host firewall via nftables. Note explicitly: Docker manipulates iptables and bypasses ufw — firewall rules must account for Docker's chains.
• Only 80/443 (+ SFTP, + admin SSH) exposed. Inter-customer traffic denied.
• Egress filtering: block outbound SMTP except via an approved relay — stops a hacked WordPress becoming a spam source (common real incident).

Observability (phased)

• Phase in: node_exporter + cAdvisor + Traefik metrics → Prometheus + Grafana; Loki + Promtail for logs. Add mysqld_exporter + Alertmanager later.
• Uptime-Kuma for cheap uptime/status pages.

DevOps stack (lightweight)

• Forgejo (git) + built-in container registry + Forgejo Actions (or Woodpecker) for CI.
• Trivy (images/fs/IaC/secrets) + gitleaks in CI. Falco later for runtime detection of compromised containers. No Harbor, no GitLab, no Jenkins.

Provisioning model (CLI-first, GitOps-lite)

A CLI/templating tool takes (customer, site-profile, domain) and:

1. Creates the ZFS dataset(s).
2. Renders docker-compose.yml + .env from the profile template.
3. Creates the shared-DB database + user (for profiles needing DB).
4. Commits rendered config to the deployments repo (audit trail).
5. Runs docker compose up -d in the customer directory. Deprovision/backup/restore are additional CLI subcommands. The customer-facing panel is a later wrapper over this same tooling.

---

Repository Structure

Small number of repos (not a monorepo, not one-per-thing) because secrets and per-customer state have a different lifecycle than platform code:

• platform-infra — Ansible (Docker, ZFS, nftables, host bootstrap), base compose for Traefik / monitoring / Forgejo.
• site-templates — Dockerfiles for standard images (wordpress, php-fpm, static) + compose templates per site profile.
• deployments — one directory per customer; rendered compose + env refs. Secrets via SOPS/age or a vault — never plaintext.
• control-panel — the CLI tooling now; the customer panel later.

Solo/small-team may start as one monorepo with these as top-level dirs, but keep the deployments/secrets boundary clean from day one.

---

Documents to Create Before Code

1. Architecture overview + threat model — boundaries and what they do/don't protect (this doc is the seed).
2. ADRs — one short record per major call: Traefik edge, shared MariaDB, Forgejo-vs-GitLab, CLI-first-vs-panel. Captures the why.
3. Naming & conventions — dataset, network, container, database, domain naming.
4. Site profile spec — exact contents of wordpress / custom-php / static / redirect.
5. Provisioning workflow — signup → live, step by step.
6. Backup & DR runbook — two decoupled streams (web files via ZFS snapshot/send; DB via automysqlbackup + rsync), cadence and near-simultaneous scheduling, offsite targets, single-customer two-step restore drill.
7. Repo layout / GitOps doc.
8. Roadmap / phased task plan (section below).

---

Phased Task Roadmap

Phase 0 — Documents & decisions

• Write the 8 documents above (start from this plan).
• Finalize naming conventions and dataset layout.

Phase 1 — Host baseline (Ansible)

• Provision the bare-metal host: ZFS pool + datasets, Docker, nftables (with Docker-aware rules), admin SSH hardening, egress filtering.
• platform-infra repo with idempotent Ansible.

Phase 2 — Platform services

• Deploy Traefik (ACME/TLS, proxy network, dashboards secured).
• Deploy shared MariaDB (own datadir dataset, tuned buffer pool).
• Deploy Forgejo + registry.
• Smoke test: a hello-world container routed + TLS via Traefik.

Phase 3 — Site templates & images

• Build standard images (php-fpm non-root, nginx, static) in site-templates.
• Trivy + gitleaks scanning in CI on those images.
• Author compose templates for the 4 site profiles.

Phase 4 — Provisioning CLI

• control-panel CLI: provision / deprovision / list, rendering templates, creating ZFS datasets, creating DB+user, committing to deployments, compose up.
• SFTP access per customer (chrooted).

Phase 5 — Backup & DR

• Web files: ZFS snapshot schedule + zfs send offsite (lean incrementals).
• DB: automysqlbackup (per-DB, rotated) → db-backups → rsync offsite, scheduled close to the snapshot.
• Restore drill: restore one test customer end-to-end from offsite — files from ZFS and DB from the dump repo (two coordinated steps).

Phase 6 — Observability

• node_exporter + cAdvisor + Traefik metrics → Prometheus + Grafana; Loki + Promtail. Uptime-Kuma. Basic alerts.

Phase 7 — Migration

• Migrate existing sites profile-by-profile (static/redirect first, then custom PHP, then WordPress). Cut DNS per site after validation.

Phase 8 — Customer panel (later)

• Web panel wrapping the Phase 4 CLI: create/delete site, trigger backup, self-service restore, web file manager. Optionally Falco for runtime detection.

---

Verification / Validation

Because this is infrastructure, "done" is proven by drills, not unit tests:

1. Routing/TLS: provision a test static site → reachable over HTTPS via Traefik with a valid Let's Encrypt cert.
2. Isolation: from inside customer A's container, confirm you cannot reach customer B's containers/network or read B's files; confirm outbound SMTP is blocked.
3. DB least-privilege: confirm customer A's DB user cannot see B's database.
4. Provisioning round-trip: CLI provisions a WordPress site from zero (dataset + DB + compose + route) and it serves; then deprovision cleans up.
5. Backup/restore drill (the critical one): ZFS snapshot + zfs send of web files, automysqlbackup + rsync of the DB, both offsite → restore that single customer onto a clean target (files from ZFS, DB from the dump repo) and verify site + data come back, neighbours untouched. Also verify an unchanged site yields a near-empty ZFS incremental (confirms the lean-dataset goal).
6. CI security gate: Trivy/gitleaks fail a build on a known-vulnerable image or planted secret.