Refine to nested customer -> site model

Adopt a two-level ownership model: a customer owns many sites, with
isolation/containers/DB/backup at the site level and grouping (one SFTP
login, recursive backup, billing, bulk delete) at the customer level.

- 03-naming-conventions: rewritten for customer/site ids and slug =
  <customer>-<site>; nested ZFS, per-customer SFTP chroot, deployments
  mirror the nesting.
- Propagated paths through docs 01, 04, 05, 06, 07 and deployments/README.
- ADR 0005 retitled/updated to per-site datasets nested under customer.
- architecture-plan.md: note pointing to doc 03 as authoritative on naming.

Clarifies subdomains: same-app subdomains are aliases on one site; a
separate-app subdomain is its own isolated site under the same customer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bart Van Geyt 2026-07-07 12:37:39 +02:00
parent 9f819df4d9
commit e5ff798dd4
11 changed files with 171 additions and 115 deletions

View file

@ -1,14 +1,15 @@
# deployments
GitOps state — one directory per site, rendered by the provisioning CLI
(Phase 4). Empty until the first site is provisioned.
GitOps state — nested `customer/site` directories, rendered by the provisioning
CLI (Phase 4). Empty until the first site is provisioned.
Per-site layout (see
Layout (see
[../docs/03-naming-conventions.md](../docs/03-naming-conventions.md) §7):
```
<slug>/
├── site.yaml # declarative source of truth (slug, profile, domains)
<customer>/
└── <site>/
├── site.yaml # declarative source of truth (customer, site, profile, domains)
├── docker-compose.yml # rendered from a site-templates profile
├── .env.example # non-secret references
└── secrets.enc.yaml # SOPS/age-encrypted secrets (committed encrypted only)

View file

@ -37,7 +37,7 @@ The full approved architecture plan is at
1. **The customer is the boundary** — isolation, backup, restore align on it.
2. **Containers isolate, they don't secure by themselves** — defense in depth
(non-root FPM, read-only rootfs, per-customer networks, egress filtering,
(non-root FPM, read-only rootfs, per-site networks, egress filtering,
least-privilege DB users).
3. **Each backup stream matches its data's change pattern** — don't fold the DB
dump into the web dataset (it would bloat every incremental).

View file

@ -19,10 +19,10 @@ WordPress, custom PHP, static HTML, and redirect-only sites.
└───┬─────────┬───┘
│ │ shared "proxy" network
┌───────────────┘ └───────────────┐
│ per-customer network A │ per-customer network B
│ per-site network (acme-shop_net) │ per-site network (bar-www_net)
┌────▼─────┐ ┌──────────┐ ┌─────▼────┐ ┌──────────┐
│ nginx │──▶│ php-fpm │ │ nginx │──▶│ php-fpm │
(site A) │ │ (site A) │ │ (site B) │ │ (site B)
acme/shop │ │acme/shop │ │ bar/www │ │ bar/www
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
│ web root vol │ │ web root vol │
│ └──────────┐ ┌──────────┘ │
@ -32,11 +32,15 @@ WordPress, custom PHP, static HTML, and redirect-only sites.
│ │ (platform net) │ least-priv user │
│ └─────────────────┘ │
▼ ▼
ZFS tank/customers/A/web ZFS tank/customers/B/web
tank/customers/acme/shop/web tank/customers/bar/www/web
```
Platform services (not shown per-customer): Forgejo + registry, Prometheus /
Grafana / Loki, Uptime-Kuma. All run as their own compose projects on the host.
The isolation unit is the **site** (`<customer>/<site>`). A customer may own
several sites, each fully isolated from the others but grouped under the customer
for SFTP/backup/billing (see [03](03-naming-conventions.md)).
Platform services (not shown per-site): Forgejo + registry, Prometheus / Grafana
/ Loki, Uptime-Kuma. All run as their own compose projects on the host.
### Request path
1. DNS points the customer domain at the host.
@ -58,8 +62,8 @@ This is deliberate, not redundancy.
| Boundary | Mechanism | Protects against |
|----------|-----------|------------------|
| Process/filesystem | Separate containers per site | One site reading another's files/processes |
| Network (east-west) | Per-customer Docker bridge; only Traefik bridges to `proxy` | Site A reaching Site B's containers |
| Data at rest | ZFS dataset per customer, bind-mounted web root | Cross-customer data access; enables clean restore |
| Network (east-west) | Per-site Docker bridge; only Traefik bridges to `proxy` | Site A reaching Site B's containers (even same customer) |
| Data at rest | ZFS dataset per site (nested under customer), bind-mounted web root | Cross-site data access; enables clean per-site restore |
| Database | Per-site DB + least-privilege user on shared instance | Site A reading Site B's tables |
| Privilege | php-fpm non-root; read-only rootfs where possible; writable web root only | Privilege escalation within a container |
| Egress | Firewall egress filtering (esp. SMTP) | Hacked site sending spam / exfiltration |

View file

@ -1,99 +1,134 @@
# 03 — Naming & Conventions
Consistent, predictable names are what make CLI-driven provisioning and
scripted backup/restore reliable. Every resource for a site is derivable from a
single **slug**.
Consistent, predictable names are what make CLI-driven provisioning and scripted
backup/restore reliable. The model has **two levels**: a **customer** (owner) may
own many **sites** (deployable units). Isolation, containers, and databases live
at the **site** level; grouping (SFTP login, billing, bulk backup/delete) lives
at the **customer** level.
## 1. The slug
## 1. Identifiers
Each site has one canonical **slug**: lowercase, ASCII, `[a-z0-9-]`, 332 chars,
starting with a letter. Derived from the primary domain (dots → hyphens) or set
explicitly.
| Id | Scope | Charset | Example |
|----|-------|---------|---------|
| `customer` | Owner / billing entity | `[a-z0-9-]`, 232, starts with a letter | `acme` |
| `site` | One deployable unit, unique within its customer | `[a-z0-9-]`, 240, starts with a letter | `shop` |
| `slug` | Globally-unique flattened id = `<customer>-<site>` | derived | `acme-shop` |
- `example.com``example-com`
- `blog.example.com``blog-example-com`
- **`customer`** is chosen once (short, stable — usually the company name).
- **`site`** identifies an app within that customer. Derive from the primary
domain or set explicitly (`acme.com``acme-com`; `shop.acme.com``shop`).
- **`slug`** = `<customer>-<site>`, the join key for Docker and database names.
Neither `customer` nor `site` changes for the life of the resource (renaming =
new id + migration).
The slug is the join key across ZFS, Docker, database, and deployment config. It
never changes for the life of the site (renaming = new slug + migration).
### Same app vs separate app (subdomains)
- A subdomain that serves the **same application** (`www.acme.com` + `acme.com`,
vanity domains) is an **alias** — extra entries in one site's `domains:` list,
one web root, one cert. Not a new site.
- A subdomain that is a **separate application** (`shop.acme.com` webshop vs
`acme.com` marketing) is a **separate site** under the same customer — its own
slug, web root, DB, containers, and network. Isolated from the customer's other
sites; grouped under the customer for SFTP/backup/billing.
> A customer may own several sites. Where a **customer** grouping is needed
> (billing, SFTP account), use a separate `customer` id with the same charset
> rules. The default is one slug per site.
## 2. ZFS datasets
## 2. ZFS datasets — nested customer → site
```
tank/
├── customers/
│ └── <slug>/
│ └── <customer>/ # groups all of a customer's sites
│ └── <site>/
│ └── web # web root ONLY (bind-mounted into the site)
└── platform/
├── docker # Docker data-root (images/layers/volumes)
├── mariadb # shared MariaDB datadir
├── db-backups/<slug> # automysqlbackup output per site
├── db-backups/<customer>/<site># automysqlbackup output per site
├── traefik # ACME store + dynamic config
├── forgejo # Git + registry data
└── monitoring # Prometheus/Loki data
```
Snapshots: `tank/customers/<slug>/web@auto-YYYYMMDD-HHMM`.
Example for customer `acme`:
```
tank/customers/acme/acme-com/web # WordPress marketing site
tank/customers/acme/shop/web # custom-PHP webshop (separate app + DB)
tank/customers/acme/blog/web # another WordPress
```
## 3. Docker resources
Snapshots:
- Per site: `tank/customers/<customer>/<site>/web@auto-YYYYMMDD-HHMM`.
- Whole customer (recursive): `zfs snapshot -r tank/customers/<customer>@...`.
## 3. Docker resources (keyed by the flattened slug)
| Resource | Pattern | Example |
|----------|---------|---------|
| Compose project | `<slug>` | `example-com` |
| Container | `<slug>_<role>` | `example-com_nginx`, `example-com_fpm` |
| Per-customer network | `<slug>_net` | `example-com_net` |
| Compose project | `<slug>` | `acme-shop` |
| Container | `<slug>_<role>` | `acme-shop_nginx`, `acme-shop_fpm` |
| Per-site network | `<slug>_net` | `acme-shop_net` |
| Shared edge network | `proxy` (fixed) | `proxy` |
| Platform network | `platform` (fixed) | `platform` (MariaDB, monitoring) |
| Named volume (if used) | `<slug>_<purpose>` | `example-com_fpmtmp` |
| Named volume (if used) | `<slug>_<purpose>` | `acme-shop_fpmtmp` |
Traefik router/service labels also key off the slug:
`traefik.http.routers.<slug>.rule=Host(...)`.
## 4. Database (shared MariaDB)
Hyphens in the slug become underscores for MySQL identifiers
(`acme-shop``acme_shop`).
| Resource | Pattern | Example |
|----------|---------|---------|
| Database | `db_<slug_underscored>` | `db_example_com` |
| DB user | `u_<slug_underscored>` | `u_example_com` |
| Grants | `ALL PRIVILEGES ON db_<...>.* ` to that user only | — |
| Database | `db_<customer>_<site>` | `db_acme_shop` |
| DB user | `u_<customer>_<site>` | `u_acme_shop` |
| Grants | `ALL PRIVILEGES ON db_<customer>_<site>.*` to that user only | — |
Hyphens in the slug become underscores for MySQL identifiers
(`example-com``example_com`). Passwords are generated, stored encrypted
(SOPS/age), never reused across sites.
Passwords are generated, stored encrypted (SOPS/age), never reused across sites.
## 5. Domains & TLS
- Primary domain drives the slug; additional aliases are listed in the site's
deployment config and added to the Traefik Host rule.
- Certificates are issued per domain automatically by Traefik/ACME; no manual
naming.
- The site's **primary domain** drives the default `site` id; additional aliases
live in the site's `domains:` list and are added to the Traefik Host rule.
- Certificates are issued per domain automatically by Traefik/ACME.
## 6. SFTP accounts
## 6. SFTP accounts — per customer
One login per customer, chrooted to the customer directory so they see all their
sites as sub-folders.
| Resource | Pattern | Example |
|----------|---------|---------|
| SFTP user | `sftp_<slug_underscored>` | `sftp_example_com` |
| Chroot path | `tank/customers/<slug>/web` | — |
| SFTP user | `sftp_<customer>` | `sftp_acme` |
| Chroot path | `tank/customers/<customer>/` | sees `acme-com/web`, `shop/web`, … |
## 7. Deployment config (in `deployments/`)
(A future option: per-site SFTP users for customers who want to delegate access
to a single site. Default is one per customer.)
## 7. Deployment config (in `deployments/`) — mirrors ZFS
```
deployments/
└── <slug>/
└── <customer>/
└── <site>/
├── site.yaml # declarative source of truth
├── docker-compose.yml # rendered from a site-templates profile
├── .env.example # non-secret defaults / references
├── secrets.enc.yaml # SOPS/age-encrypted secrets (committed)
└── site.yaml # slug, profile, domains, options (source of truth)
└── secrets.enc.yaml # SOPS/age-encrypted secrets (committed)
```
`site.yaml` is the declarative source of truth the CLI reads/writes; everything
else is rendered from it.
`site.yaml` is the source of truth the CLI reads/writes; everything else is
rendered from it. It carries both ids:
```yaml
customer: acme
site: shop
profile: custom-php
domains: [shop.acme.com]
```
## 8. Reserved words
Slugs may not be: `proxy`, `platform`, `traefik`, `mariadb`, `forgejo`,
`monitoring`, `docker`, `customers`, `platform-infra`, `site-templates`,
`deployments`, `control-panel` (avoids collisions with platform names).
`customer` and `site` ids may not be: `proxy`, `platform`, `traefik`, `mariadb`,
`forgejo`, `monitoring`, `docker`, `customers`, `platform-infra`,
`site-templates`, `deployments`, `control-panel` (avoids collisions with
platform names).

View file

@ -7,7 +7,7 @@ domains, and options. Four profiles cover all current workloads.
Common to every profile:
- Traefik labels for routing + TLS (Host rule from domains, ACME resolver).
- Joined to the site's own `<slug>_net` network; only Traefik bridges to `proxy`.
- Web root bind-mounted from `tank/customers/<slug>/web`.
- Web root bind-mounted from `tank/customers/<customer>/<site>/web`.
- Per-project CPU/memory limits.
- Containers run non-root; rootfs read-only where the profile allows.
@ -24,7 +24,7 @@ Common to every profile:
- **Containers:** one nginx serving the web root read-only. For very simple
cases, Traefik can serve files directly with no container.
- **Volumes:** `tank/customers/<slug>/web` → `/usr/share/nginx/html` (read-only).
- **Volumes:** `tank/customers/<customer>/<site>/web` → `/usr/share/nginx/html` (read-only).
- **DB:** none.
- **Notes:** cheapest profile; near-zero backup incrementals when unchanged.
@ -43,8 +43,8 @@ Common to every profile:
- **Containers:**
- `nginx` — serves static assets, proxies `.php` to fpm over FastCGI.
- `php-fpm` — non-root; only the web root (and a small tmp) writable.
- **Volumes:** `tank/customers/<slug>/web` shared by both (web root).
- **DB:** optional — if requested, a `db_<slug>` + `u_<slug>` on shared MariaDB;
- **Volumes:** `tank/customers/<customer>/<site>/web` shared by both (web root).
- **DB:** optional — if requested, a `db_<customer>_<site>` + `u_<customer>_<site>` on shared MariaDB;
credentials injected via env from `secrets.enc.yaml`.
- **Options:** PHP version (pinned image tag), extensions, `php.ini` overrides,
cron (via a scheduled fpm exec) if needed.
@ -53,9 +53,9 @@ Common to every profile:
- **Containers:** same nginx + php-fpm pair as `custom-php`, using a WordPress
base image (or WP installed into the web root on first provision).
- **Volumes:** `tank/customers/<slug>/web` (WordPress core, themes, plugins,
- **Volumes:** `tank/customers/<customer>/<site>/web` (WordPress core, themes, plugins,
uploads).
- **DB:** **required**`db_<slug>` + `u_<slug>` on shared MariaDB.
- **DB:** **required**`db_<customer>_<site>` + `u_<customer>_<site>` on shared MariaDB.
- **Hardening baked in:**
- php-fpm non-root; `wp-content/uploads` writable, PHP execution denied there
(nginx rule) to blunt upload-based RCE.
@ -73,9 +73,10 @@ Every profile is rendered from the same declarative fields; unused fields are
ignored per profile:
```yaml
slug: example-com
customer: acme # owner id
site: acme-com # site id, unique within the customer
profile: wordpress # static | redirect | custom-php | wordpress
domains: # first is primary → drives cert + slug
domains: # first is primary → drives cert + default site id
- example.com
- www.example.com
php_version: "8.3" # custom-php / wordpress

View file

@ -13,23 +13,25 @@ failed provision can be resumed.
## New site — `provision`
Input: `slug`, `profile`, `domains`, profile options (see
[04](04-site-profiles.md) `site.yaml`).
Input: `customer`, `site`, `profile`, `domains`, profile options (see
[04](04-site-profiles.md) `site.yaml`). The flattened `slug` = `<customer>-<site>`.
1. **Validate**slug charset/reserved-word check
1. **Validate**customer/site charset + reserved-word check
([03](03-naming-conventions.md)), domains resolvable/owned, profile known.
2. **ZFS** — create `tank/customers/<slug>/web` (and `db-backups/<slug>` if the
profile uses a DB).
3. **Database** (DB profiles) — create `db_<slug>` + `u_<slug>` with least
privilege on shared MariaDB; generate password; write to `secrets.enc.yaml`
(SOPS/age).
4. **Render** — produce `deployments/<slug>/docker-compose.yml` + `.env` from the
profile template and `site.yaml`.
5. **SFTP** — create chrooted `sftp_<slug>` account bound to the web root.
6. **Commit** — commit `deployments/<slug>/` to the `deployments` repo (audit
trail; secrets committed only in encrypted form).
7. **Deploy**`docker compose up -d` in `deployments/<slug>/`. Traefik
discovers the route from labels; ACME issues the certificate.
2. **ZFS** — ensure `tank/customers/<customer>` exists, then create
`tank/customers/<customer>/<site>/web` (and `db-backups/<customer>/<site>` if
the profile uses a DB).
3. **Database** (DB profiles) — create `db_<customer>_<site>` +
`u_<customer>_<site>` with least privilege on shared MariaDB; generate
password; write to `secrets.enc.yaml` (SOPS/age).
4. **Render** — produce `deployments/<customer>/<site>/docker-compose.yml` +
`.env` from the profile template and `site.yaml`.
5. **SFTP** — ensure the customer's chrooted `sftp_<customer>` account exists
(chroot `tank/customers/<customer>/`); the new site appears as a sub-folder.
6. **Commit** — commit `deployments/<customer>/<site>/` to the `deployments` repo
(audit trail; secrets committed only in encrypted form).
7. **Deploy**`docker compose up -d` in `deployments/<customer>/<site>/`.
Traefik discovers the route from labels; ACME issues the certificate.
8. **Verify** — HTTPS reachability + valid cert; for WordPress, run WP-CLI
install. Print access details.
@ -47,11 +49,13 @@ delta. TLS for new domains is automatic.
1. `docker compose down` (optionally `--remove-orphans`).
2. **Final backup** — take a last ZFS snapshot + final DB dump, retained per the
deprovision retention policy before deletion.
3. Drop `db_<slug>` + `u_<slug>` (after the final dump).
4. Remove the SFTP account.
5. Destroy `tank/customers/<slug>/web` (and `db-backups/<slug>`) **after** the
retention window — never immediately.
6. Remove `deployments/<slug>/` and commit.
3. Drop `db_<customer>_<site>` + `u_<customer>_<site>` (after the final dump).
4. Remove the site's SFTP sub-folder access; remove the `sftp_<customer>` account
only when the customer has no remaining sites.
5. Destroy `tank/customers/<customer>/<site>` (and `db-backups/<customer>/<site>`)
**after** the retention window — never immediately. Destroy the customer
dataset only when their last site is removed.
6. Remove `deployments/<customer>/<site>/` and commit.
> Destroys are gated: the CLI refuses to delete data younger than the retention
> window without an explicit `--force`, and always snapshots before destroying.
@ -60,7 +64,8 @@ delta. TLS for new domains is automatic.
Routine backups run on a schedule (not per-command); restore is on demand. Both
are specified in the [Backup & DR runbook](06-backup-and-dr.md). CLI surface:
`backup <slug>` (ad-hoc), `restore <slug> --snapshot <name> --db <dump>`.
`backup <customer> <site>` (ad-hoc),
`restore <customer> <site> --snapshot <name> --db <dump>`.
## CLI command summary

View file

@ -7,7 +7,7 @@ offsite transfer cheap while preserving per-customer restore.
| Stream | Data | Mechanism | Onsite | Offsite |
|--------|------|-----------|--------|---------|
| A | Web files | ZFS snapshot + `zfs send` | snapshots on `tank` | incremental `send` to offsite pool |
| B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<slug>` | rsync to offsite |
| B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<customer>/<site>` | rsync to offsite |
| C | Logs | Promtail → Loki | Loki store | (per Loki retention; not customer-restore data) |
Platform state (Traefik ACME store, Forgejo data, `deployments` repo, Prometheus
@ -32,26 +32,29 @@ config) is backed up with the same ZFS mechanism from `tank/platform/*`.
- **DB dumps:** rsync to the same or another offsite location (delta transfer).
- Both offsite copies are the recovery source if the primary host is lost.
## 3. Restore — single customer (the critical drill)
## 3. Restore — single site (the critical drill)
Two coordinated steps:
**A. Web files (ZFS)**
1. Identify the target snapshot: `tank/customers/<slug>/web@auto-...`.
1. Identify the target snapshot: `tank/customers/<customer>/<site>/web@auto-...`.
2. Restore by clone/rollback (or `zfs receive` from offsite if the host is
gone) into `tank/customers/<slug>/web`.
gone) into `tank/customers/<customer>/<site>/web`.
**B. Database (dump)**
3. Pick the matching dump from `db-backups/<slug>` (or offsite).
4. Recreate `db_<slug>` + `u_<slug>` if needed; import the dump into shared
MariaDB.
3. Pick the matching dump from `db-backups/<customer>/<site>` (or offsite).
4. Recreate `db_<customer>_<site>` + `u_<customer>_<site>` if needed; import the
dump into shared MariaDB.
**C. Bring up**
5. `docker compose up -d` in `deployments/<slug>/`.
5. `docker compose up -d` in `deployments/<customer>/<site>/`.
6. Verify site + data; for WordPress confirm site URL / run WP-CLI
`search-replace` if the domain changed.
CLI: `restore <slug> --snapshot <name> --db <dump-file>` wraps AC.
CLI: `restore <customer> <site> --snapshot <name> --db <dump-file>` wraps AC.
> To restore an entire customer at once, repeat AC per site, or use the
> recursive snapshot `tank/customers/<customer>@...` as the file source.
## 4. Restore — full host (DR rebuild)

View file

@ -56,7 +56,7 @@ calling the same operations.
## 3. GitOps flow
```
edit site.yaml ─▶ CLI renders ─▶ commit deployments/<slug> ─▶ compose up -d
edit site.yaml ─▶ CLI renders ─▶ commit deployments/<cust>/<site> ─▶ compose up -d
(intent) (from templates) (audit trail) (converge)
```

View file

@ -1,4 +1,4 @@
# ADR 0005 — ZFS dataset per customer
# ADR 0005 — ZFS dataset per site (nested under customer)
**Status:** Accepted
@ -8,10 +8,12 @@ touching neighbours. ZFS offers checksummed integrity, cheap copy-on-write
snapshots, and incremental `zfs send`.
## Decision
Create a **ZFS dataset per customer** (`tank/customers/<customer>/web`) holding
**only the web root**. Docker image/layer storage and the MariaDB datadir live
on separate platform datasets. Customer web data is bind-mounted into containers
from the customer dataset.
Create a **ZFS dataset per site**, nested under the owning customer
(`tank/customers/<customer>/<site>/web`), holding **only the web root**. The
customer level is a grouping parent enabling recursive snapshots and a single
SFTP chroot; the site level is the isolation/restore unit. Docker image/layer
storage and the MariaDB datadir live on separate platform datasets. Site web
data is bind-mounted into containers from its dataset.
## Consequences
- ✅ Snapshot and `zfs send` operate at the customer granularity — restore one

View file

@ -9,7 +9,7 @@ Decision → Consequences. Status is one of Proposed / Accepted / Superseded.
| [0002](0002-traefik-as-edge.md) | Traefik as the edge router | Accepted |
| [0003](0003-nginx-fpm-per-site.md) | Per-site nginx + php-fpm behind Traefik | Accepted |
| [0004](0004-shared-mariadb.md) | Shared MariaDB instance, per-site DB + user | Accepted |
| [0005](0005-zfs-per-customer.md) | ZFS dataset per customer | Accepted |
| [0005](0005-zfs-per-customer.md) | ZFS dataset per site (nested under customer) | Accepted |
| [0006](0006-decoupled-backup-streams.md) | Decoupled backup streams (files vs DB vs logs) | Accepted |
| [0007](0007-forgejo-over-gitlab.md) | Forgejo (lightweight) over GitLab | Accepted |
| [0008](0008-cli-first-panel-later.md) | CLI/templating first, customer panel later | Accepted |

View file

@ -1,5 +1,10 @@
# Rebuild: Multi-Tenant Web Hosting Platform (heleosv2)
> **Note:** This is the point-in-time *approved plan*. Where it refers to the
> per-customer boundary/naming, the design was later refined to a nested
> **customer → site** model — see [`03-naming-conventions.md`](03-naming-conventions.md),
> which is authoritative for naming and paths.
## Context
An existing small web-hosting business runs nginx + PHP-FPM + MySQL on a shared