# Rebuild: Multi-Tenant Web Hosting Platform (heleosv2)

## Context

An existing small web-hosting business runs nginx + PHP-FPM + MySQL on a shared
host, serving a mix of custom PHP, WordPress, static HTML, and redirect-only
sites. The goal is to rebuild on modern best practices with **isolation**,
**clean backups**, and **repeatable provisioning** as first principles.

Core philosophy: **the customer is the boundary.** Isolation (containers),
backup (ZFS dataset), and restore all line up on that same boundary, so a
compromised or broken site is contained and can be restored without touching
neighbours.

This document is an **architecture + phased task plan**, not a code change. No
application code exists yet (greenfield). Implementation happens step-by-step
after this plan is approved.

### Decisions locked with the user
- **Host topology:** single bare-metal host → Docker Compose per customer, no
  orchestrator, Ansible for host config, Terraform not needed initially.
- **Control panel:** CLI/templating tooling first; customer-facing panel later
  once the platform is proven.
- **Database:** one **shared MariaDB** instance (per-site DB + per-site user).
  Density over per-instance isolation.
- **DevOps stack:** lightweight — **Forgejo** + built-in registry + Forgejo
  Actions/Woodpecker + Trivy.

### Reframing carried into the design
Containers are an **isolation** boundary, not a hard security boundary (shared
kernel). The real protections come from: non-root PHP-FPM, read-only rootfs
where possible, per-customer Docker networks, egress filtering, least-privilege
DB users, and (later) runtime detection. Docker alone is not the security story.

---

## Target Architecture

### Edge & routing
- **Traefik** as the single edge: TLS termination, automatic Let's Encrypt
  (ACME), dynamic label-based routing. One shared `proxy` Docker network.
- Traefik **cannot speak FastCGI**, so each PHP site runs its own small web
  server behind Traefik:

  `Traefik (TLS + routing) → per-site nginx → per-site php-fpm`

- Static/redirect sites are served directly by Traefik or a tiny nginx — no PHP.

### Per-customer stack (one Docker Compose project per customer/site)
Standard **site profiles**, each a compose template:
- **wordpress** — nginx + php-fpm (+ shared-DB credentials)
- **custom-php** — nginx + php-fpm (+ optional shared-DB credentials)
- **static** — nginx (or Traefik direct), no PHP, no DB
- **redirect** — Traefik router rule / tiny nginx, no PHP, no DB

Per-customer isolation:
- Own Docker bridge network; only Traefik joins both `proxy` and the customer
  network, so customer containers are unreachable except through Traefik.
- PHP-FPM runs **non-root**; rootfs read-only where the profile allows; web
  root is the only writable mount.

### Storage — ZFS
- Dataset layout, e.g.:
  `tank/customers/<customer>/web` (web root **only** — kept lean) and
  `tank/platform/{docker,mariadb,db-backups,traefik,forgejo,monitoring}`.
- **Docker images/layers** on their own platform dataset; **customer web data**
  bind-mounted from per-customer datasets (clean per-customer snapshots).
- The customer dataset deliberately holds **only web files** — no logs, no DB
  dumps — so an unchanged site produces a near-empty incremental and offsite
  `zfs send` stays cheap.
- Snapshots per customer on a schedule; **`zfs send` offsite** for backup/DR.

### Database (shared MariaDB) — decoupled backup stream
- One MariaDB container; each site gets its own database + least-privilege user.
- Live datadir lives on a **platform** dataset (not inside any customer dataset).
- **DB backups are a separate stream from web files** (not folded into the
  customer ZFS dataset — that would make every incremental `zfs send` ship a
  fresh dump even for static sites):
  - **automysqlbackup** produces per-database dumps with daily/weekly/monthly
    rotation into `tank/platform/db-backups/<customer>` — per-customer restore
    granularity for free.
  - **rsync** those dumps offsite (lean; only changed dumps move). Optional
    later: put db-backups on its own dataset and `zfs send` it if checksummed/
    immutable offsite is wanted.
- **Consistency:** schedule the nightly DB dump close to the ZFS snapshot. The
  two streams aren't atomically consistent, but the small window is harmless for
  PHP/WordPress (files-on-disk + DB-rows).

### File access
- **SFTP only** (drop FTP — plaintext). Each customer chrooted to their dataset.
- Optional **web file manager** (e.g. Filebrowser) added with the panel later.

### Network security
- Host firewall via **nftables**. Note explicitly: Docker manipulates iptables
  and bypasses ufw — firewall rules must account for Docker's chains.
- Only 80/443 (+ SFTP, + admin SSH) exposed. Inter-customer traffic denied.
- **Egress filtering:** block outbound SMTP except via an approved relay — stops
  a hacked WordPress becoming a spam source (common real incident).

### Observability (phased)
- Phase in: node_exporter + cAdvisor + Traefik metrics → Prometheus + Grafana;
  Loki + Promtail for logs. Add mysqld_exporter + Alertmanager later.
- Uptime-Kuma for cheap uptime/status pages.

### DevOps stack (lightweight)
- **Forgejo** (git) + built-in container **registry** + **Forgejo Actions** (or
  Woodpecker) for CI.
- **Trivy** (images/fs/IaC/secrets) + **gitleaks** in CI. **Falco** later for
  runtime detection of compromised containers. No Harbor, no GitLab, no Jenkins.

### Provisioning model (CLI-first, GitOps-lite)
A CLI/templating tool takes `(customer, site-profile, domain)` and:
1. Creates the ZFS dataset(s).
2. Renders `docker-compose.yml` + `.env` from the profile template.
3. Creates the shared-DB database + user (for profiles needing DB).
4. Commits rendered config to the `deployments` repo (audit trail).
5. Runs `docker compose up -d` in the customer directory.
Deprovision/backup/restore are additional CLI subcommands. The customer-facing
panel is a later wrapper over this same tooling.

---

## Repository Structure

Small number of repos (not a monorepo, not one-per-thing) because secrets and
per-customer state have a different lifecycle than platform code:

- **`platform-infra`** — Ansible (Docker, ZFS, nftables, host bootstrap), base
  compose for Traefik / monitoring / Forgejo.
- **`site-templates`** — Dockerfiles for standard images (wordpress, php-fpm,
  static) + compose templates per site profile.
- **`deployments`** — one directory per customer; rendered compose + env refs.
  **Secrets via SOPS/age or a vault — never plaintext.**
- **`control-panel`** — the CLI tooling now; the customer panel later.

Solo/small-team may start as one monorepo with these as top-level dirs, but keep
the **deployments/secrets boundary clean from day one**.

---

## Documents to Create Before Code

1. **Architecture overview + threat model** — boundaries and what they do/don't
   protect (this doc is the seed).
2. **ADRs** — one short record per major call: Traefik edge, shared MariaDB,
   Forgejo-vs-GitLab, CLI-first-vs-panel. Captures the *why*.
3. **Naming & conventions** — dataset, network, container, database, domain
   naming.
4. **Site profile spec** — exact contents of wordpress / custom-php / static /
   redirect.
5. **Provisioning workflow** — signup → live, step by step.
6. **Backup & DR runbook** — two decoupled streams (web files via ZFS
   snapshot/send; DB via automysqlbackup + rsync), cadence and near-simultaneous
   scheduling, offsite targets, single-customer two-step restore drill.
7. **Repo layout / GitOps doc.**
8. **Roadmap / phased task plan** (section below).

---

## Phased Task Roadmap

### Phase 0 — Documents & decisions
- Write the 8 documents above (start from this plan).
- Finalize naming conventions and dataset layout.

### Phase 1 — Host baseline (Ansible)
- Provision the bare-metal host: ZFS pool + datasets, Docker, nftables (with
  Docker-aware rules), admin SSH hardening, egress filtering.
- `platform-infra` repo with idempotent Ansible.

### Phase 2 — Platform services
- Deploy Traefik (ACME/TLS, `proxy` network, dashboards secured).
- Deploy shared MariaDB (own datadir dataset, tuned buffer pool).
- Deploy Forgejo + registry.
- Smoke test: a hello-world container routed + TLS via Traefik.

### Phase 3 — Site templates & images
- Build standard images (php-fpm non-root, nginx, static) in `site-templates`.
- Trivy + gitleaks scanning in CI on those images.
- Author compose templates for the 4 site profiles.

### Phase 4 — Provisioning CLI
- `control-panel` CLI: provision / deprovision / list, rendering templates,
  creating ZFS datasets, creating DB+user, committing to `deployments`,
  `compose up`.
- SFTP access per customer (chrooted).

### Phase 5 — Backup & DR
- **Web files:** ZFS snapshot schedule + `zfs send` offsite (lean incrementals).
- **DB:** automysqlbackup (per-DB, rotated) → `db-backups` → rsync offsite,
  scheduled close to the snapshot.
- **Restore drill:** restore one test customer end-to-end from offsite — files
  from ZFS **and** DB from the dump repo (two coordinated steps).

### Phase 6 — Observability
- node_exporter + cAdvisor + Traefik metrics → Prometheus + Grafana; Loki +
  Promtail. Uptime-Kuma. Basic alerts.

### Phase 7 — Migration
- Migrate existing sites profile-by-profile (static/redirect first, then custom
  PHP, then WordPress). Cut DNS per site after validation.

### Phase 8 — Customer panel (later)
- Web panel wrapping the Phase 4 CLI: create/delete site, trigger backup,
  self-service restore, web file manager. Optionally Falco for runtime detection.

---

## Verification / Validation

Because this is infrastructure, "done" is proven by drills, not unit tests:

1. **Routing/TLS:** provision a test `static` site → reachable over HTTPS via
   Traefik with a valid Let's Encrypt cert.
2. **Isolation:** from inside customer A's container, confirm you **cannot**
   reach customer B's containers/network or read B's files; confirm outbound
   SMTP is blocked.
3. **DB least-privilege:** confirm customer A's DB user cannot see B's database.
4. **Provisioning round-trip:** CLI provisions a WordPress site from zero
   (dataset + DB + compose + route) and it serves; then deprovision cleans up.
5. **Backup/restore drill (the critical one):** ZFS snapshot + `zfs send` of web
   files, automysqlbackup + rsync of the DB, both offsite → restore that single
   customer onto a clean target (files from ZFS, DB from the dump repo) and
   verify site + data come back, neighbours untouched. Also verify an **unchanged
   site yields a near-empty ZFS incremental** (confirms the lean-dataset goal).
6. **CI security gate:** Trivy/gitleaks fail a build on a known-vulnerable image
   or planted secret.