bart/platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Refine to nested customer -> site model

Browse source 
Adopt a two-level ownership model: a customer owns many sites, with
isolation/containers/DB/backup at the site level and grouping (one SFTP
login, recursive backup, billing, bulk delete) at the customer level.

- 03-naming-conventions: rewritten for customer/site ids and slug =
  <customer>-<site>; nested ZFS, per-customer SFTP chroot, deployments
  mirror the nesting.
- Propagated paths through docs 01, 04, 05, 06, 07 and deployments/README.
- ADR 0005 retitled/updated to per-site datasets nested under customer.
- architecture-plan.md: note pointing to doc 03 as authoritative on naming.

Clarifies subdomains: same-app subdomains are aliases on one site; a
separate-app subdomain is its own isolated site under the same customer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bart Van Geyt 2026-07-07 12:37:39 +02:00
parent 9f819df4d9
commit e5ff798dd4
11 changed files with 171 additions and 115 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
deployments/README.md
View file

@ -1,17 +1,18 @@
# deployments # deployments
GitOps state — one directory per site, rendered by the provisioning CLI GitOps state — nested `customer/site` directories, rendered by the provisioning
(Phase 4). Empty until the first site is provisioned. CLI (Phase 4). Empty until the first site is provisioned.
Per-site layout (see Layout (see
[../docs/03-naming-conventions.md](../docs/03-naming-conventions.md) §7): [../docs/03-naming-conventions.md](../docs/03-naming-conventions.md) §7):
``` ```
<slug>/ <customer>/
├── site.yaml # declarative source of truth (slug, profile, domains) └── <site>/
├── docker-compose.yml # rendered from a site-templates profile ├── site.yaml # declarative source of truth (customer, site, profile, domains)
├── .env.example # non-secret references ├── docker-compose.yml # rendered from a site-templates profile
└── secrets.enc.yaml # SOPS/age-encrypted secrets (committed encrypted only) ├── .env.example # non-secret references
└── secrets.enc.yaml # SOPS/age-encrypted secrets (committed encrypted only)
``` ```
**Never commit plaintext secrets.** Only `*.enc.*` / `*.sops.*` are allowed. **Never commit plaintext secrets.** Only `*.enc.*` / `*.sops.*` are allowed.

2
docs/00-roadmap.md
View file

@ -37,7 +37,7 @@ The full approved architecture plan is at
1. **The customer is the boundary** — isolation, backup, restore align on it. 1. **The customer is the boundary** — isolation, backup, restore align on it.
2. **Containers isolate, they don't secure by themselves** — defense in depth 2. **Containers isolate, they don't secure by themselves** — defense in depth
(non-root FPM, read-only rootfs, per-customer networks, egress filtering, (non-root FPM, read-only rootfs, per-site networks, egress filtering,
least-privilege DB users). least-privilege DB users).
3. **Each backup stream matches its data's change pattern** — don't fold the DB 3. **Each backup stream matches its data's change pattern** — don't fold the DB
dump into the web dataset (it would bloat every incremental). dump into the web dataset (it would bloat every incremental).

18
docs/01-architecture-and-threat-model.md
View file

@ -19,10 +19,10 @@ WordPress, custom PHP, static HTML, and redirect-only sites.
└───┬─────────┬───┘ └───┬─────────┬───┘
│ │ shared "proxy" network │ │ shared "proxy" network
┌───────────────┘ └───────────────┐ ┌───────────────┘ └───────────────┐
│ per-customer network A │ per-customer network B │ per-site network (acme-shop_net) │ per-site network (bar-www_net)
┌────▼─────┐ ┌──────────┐ ┌─────▼────┐ ┌──────────┐ ┌────▼─────┐ ┌──────────┐ ┌─────▼────┐ ┌──────────┐
│ nginx │──▶│ php-fpm │ │ nginx │──▶│ php-fpm │ │ nginx │──▶│ php-fpm │ │ nginx │──▶│ php-fpm │
│ (site A) │ │ (site A) │ │ (site B) │ │ (site B) │ │acme/shop │ │acme/shop │ │ bar/www │ │ bar/www │
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
│ web root vol │ │ web root vol │ │ web root vol │ │ web root vol │
│ └──────────┐ ┌──────────┘ │ │ └──────────┐ ┌──────────┘ │
@ -32,11 +32,15 @@ WordPress, custom PHP, static HTML, and redirect-only sites.
│ │ (platform net) │ least-priv user │ │ │ (platform net) │ least-priv user │
│ └─────────────────┘ │ │ └─────────────────┘ │
▼ ▼ ▼ ▼
ZFS tank/customers/A/web ZFS tank/customers/B/web tank/customers/acme/shop/web tank/customers/bar/www/web
``` ```
Platform services (not shown per-customer): Forgejo + registry, Prometheus / The isolation unit is the **site** (`<customer>/<site>`). A customer may own
Grafana / Loki, Uptime-Kuma. All run as their own compose projects on the host. several sites, each fully isolated from the others but grouped under the customer
for SFTP/backup/billing (see [03](03-naming-conventions.md)).
Platform services (not shown per-site): Forgejo + registry, Prometheus / Grafana
/ Loki, Uptime-Kuma. All run as their own compose projects on the host.
### Request path ### Request path
1. DNS points the customer domain at the host. 1. DNS points the customer domain at the host.
@ -58,8 +62,8 @@ This is deliberate, not redundancy.
| Boundary | Mechanism | Protects against | | Boundary | Mechanism | Protects against |
|----------|-----------|------------------| |----------|-----------|------------------|
| Process/filesystem | Separate containers per site | One site reading another's files/processes | | Process/filesystem | Separate containers per site | One site reading another's files/processes |
| Network (east-west) | Per-customer Docker bridge; only Traefik bridges to `proxy` | Site A reaching Site B's containers | | Network (east-west) | Per-site Docker bridge; only Traefik bridges to `proxy` | Site A reaching Site B's containers (even same customer) |
| Data at rest | ZFS dataset per customer, bind-mounted web root | Cross-customer data access; enables clean restore | | Data at rest | ZFS dataset per site (nested under customer), bind-mounted web root | Cross-site data access; enables clean per-site restore |
| Database | Per-site DB + least-privilege user on shared instance | Site A reading Site B's tables | | Database | Per-site DB + least-privilege user on shared instance | Site A reading Site B's tables |
| Privilege | php-fpm non-root; read-only rootfs where possible; writable web root only | Privilege escalation within a container | | Privilege | php-fpm non-root; read-only rootfs where possible; writable web root only | Privilege escalation within a container |
| Egress | Firewall egress filtering (esp. SMTP) | Hacked site sending spam / exfiltration | | Egress | Firewall egress filtering (esp. SMTP) | Hacked site sending spam / exfiltration |

143
docs/03-naming-conventions.md
View file

@ -1,99 +1,134 @@
# 03 — Naming & Conventions # 03 — Naming & Conventions
Consistent, predictable names are what make CLI-driven provisioning and Consistent, predictable names are what make CLI-driven provisioning and scripted
scripted backup/restore reliable. Every resource for a site is derivable from a backup/restore reliable. The model has **two levels**: a **customer** (owner) may
single **slug**. own many **sites** (deployable units). Isolation, containers, and databases live
at the **site** level; grouping (SFTP login, billing, bulk backup/delete) lives
at the **customer** level.
## 1. The slug ## 1. Identifiers
Each site has one canonical **slug**: lowercase, ASCII, `[a-z0-9-]`, 3–32 chars, | Id | Scope | Charset | Example |
starting with a letter. Derived from the primary domain (dots → hyphens) or set |----|-------|---------|---------|
explicitly. | `customer` | Owner / billing entity | `[a-z0-9-]`, 2–32, starts with a letter | `acme` |
| `site` | One deployable unit, unique within its customer | `[a-z0-9-]`, 2–40, starts with a letter | `shop` |
| `slug` | Globally-unique flattened id = `<customer>-<site>` | derived | `acme-shop` |
- `example.com` → `example-com` - **`customer`** is chosen once (short, stable — usually the company name).
- `blog.example.com` → `blog-example-com` - **`site`** identifies an app within that customer. Derive from the primary
domain or set explicitly (`acme.com` → `acme-com`; `shop.acme.com` → `shop`).
- **`slug`** = `<customer>-<site>`, the join key for Docker and database names.
Neither `customer` nor `site` changes for the life of the resource (renaming =
new id + migration).
The slug is the join key across ZFS, Docker, database, and deployment config. It ### Same app vs separate app (subdomains)
never changes for the life of the site (renaming = new slug + migration). - A subdomain that serves the **same application** (`www.acme.com` + `acme.com`,
vanity domains) is an **alias** — extra entries in one site's `domains:` list,
one web root, one cert. Not a new site.
- A subdomain that is a **separate application** (`shop.acme.com` webshop vs
`acme.com` marketing) is a **separate site** under the same customer — its own
slug, web root, DB, containers, and network. Isolated from the customer's other
sites; grouped under the customer for SFTP/backup/billing.
> A customer may own several sites. Where a **customer** grouping is needed ## 2. ZFS datasets — nested customer → site
> (billing, SFTP account), use a separate `customer` id with the same charset
> rules. The default is one slug per site.
## 2. ZFS datasets
``` ```
tank/ tank/
├── customers/ ├── customers/
│ └── <slug>/ │ └── <customer>/ # groups all of a customer's sites
│ └── web # web root ONLY (bind-mounted into the site) │ └── <site>/
│ └── web # web root ONLY (bind-mounted into the site)
└── platform/ └── platform/
├── docker # Docker data-root (images/layers/volumes) ├── docker # Docker data-root (images/layers/volumes)
├── mariadb # shared MariaDB datadir ├── mariadb # shared MariaDB datadir
├── db-backups/<slug> # automysqlbackup output per site ├── db-backups/<customer>/<site># automysqlbackup output per site
├── traefik # ACME store + dynamic config ├── traefik # ACME store + dynamic config
├── forgejo # Git + registry data ├── forgejo # Git + registry data
└── monitoring # Prometheus/Loki data └── monitoring # Prometheus/Loki data
``` ```
Snapshots: `tank/customers/<slug>/web@auto-YYYYMMDD-HHMM`. Example for customer `acme`:
```
tank/customers/acme/acme-com/web # WordPress marketing site
tank/customers/acme/shop/web # custom-PHP webshop (separate app + DB)
tank/customers/acme/blog/web # another WordPress
```
## 3. Docker resources Snapshots:
- Per site: `tank/customers/<customer>/<site>/web@auto-YYYYMMDD-HHMM`.
- Whole customer (recursive): `zfs snapshot -r tank/customers/<customer>@...`.
## 3. Docker resources (keyed by the flattened slug)
| Resource | Pattern | Example | | Resource | Pattern | Example |
|----------|---------|---------| |----------|---------|---------|
| Compose project | `<slug>` | `example-com` | | Compose project | `<slug>` | `acme-shop` |
| Container | `<slug>_<role>` | `example-com_nginx`, `example-com_fpm` | | Container | `<slug>_<role>` | `acme-shop_nginx`, `acme-shop_fpm` |
| Per-customer network | `<slug>_net` | `example-com_net` | | Per-site network | `<slug>_net` | `acme-shop_net` |
| Shared edge network | `proxy` (fixed) | `proxy` | | Shared edge network | `proxy` (fixed) | `proxy` |
| Platform network | `platform` (fixed) | `platform` (MariaDB, monitoring) | | Platform network | `platform` (fixed) | `platform` (MariaDB, monitoring) |
| Named volume (if used) | `<slug>_<purpose>` | `example-com_fpmtmp` | | Named volume (if used) | `<slug>_<purpose>` | `acme-shop_fpmtmp` |
Traefik router/service labels also key off the slug: Traefik router/service labels also key off the slug:
`traefik.http.routers.<slug>.rule=Host(...)`. `traefik.http.routers.<slug>.rule=Host(...)`.
## 4. Database (shared MariaDB) ## 4. Database (shared MariaDB)
Hyphens in the slug become underscores for MySQL identifiers
(`acme-shop` → `acme_shop`).
| Resource | Pattern | Example | | Resource | Pattern | Example |
|----------|---------|---------| |----------|---------|---------|
| Database | `db_<slug_underscored>` | `db_example_com` | | Database | `db_<customer>_<site>` | `db_acme_shop` |
| DB user | `u_<slug_underscored>` | `u_example_com` | | DB user | `u_<customer>_<site>` | `u_acme_shop` |
| Grants | `ALL PRIVILEGES ON db_<...>.* ` to that user only | — | | Grants | `ALL PRIVILEGES ON db_<customer>_<site>.*` to that user only | — |
Hyphens in the slug become underscores for MySQL identifiers Passwords are generated, stored encrypted (SOPS/age), never reused across sites.
(`example-com` → `example_com`). Passwords are generated, stored encrypted
(SOPS/age), never reused across sites.
## 5. Domains & TLS ## 5. Domains & TLS
- Primary domain drives the slug; additional aliases are listed in the site's - The site's **primary domain** drives the default `site` id; additional aliases
deployment config and added to the Traefik Host rule. live in the site's `domains:` list and are added to the Traefik Host rule.
- Certificates are issued per domain automatically by Traefik/ACME; no manual - Certificates are issued per domain automatically by Traefik/ACME.
naming.
## 6. SFTP accounts ## 6. SFTP accounts — per customer
One login per customer, chrooted to the customer directory so they see all their
sites as sub-folders.
| Resource | Pattern | Example | | Resource | Pattern | Example |
|----------|---------|---------| |----------|---------|---------|
| SFTP user | `sftp_<slug_underscored>` | `sftp_example_com` | | SFTP user | `sftp_<customer>` | `sftp_acme` |
| Chroot path | `tank/customers/<slug>/web` | — | | Chroot path | `tank/customers/<customer>/` | sees `acme-com/web`, `shop/web`, … |
## 7. Deployment config (in `deployments/`) (A future option: per-site SFTP users for customers who want to delegate access
to a single site. Default is one per customer.)
## 7. Deployment config (in `deployments/`) — mirrors ZFS
``` ```
deployments/ deployments/
└── <slug>/ └── <customer>/
├── docker-compose.yml # rendered from a site-templates profile └── <site>/
├── .env.example # non-secret defaults / references ├── site.yaml # declarative source of truth
├── secrets.enc.yaml # SOPS/age-encrypted secrets (committed) ├── docker-compose.yml # rendered from a site-templates profile
└── site.yaml # slug, profile, domains, options (source of truth) ├── .env.example # non-secret defaults / references
└── secrets.enc.yaml # SOPS/age-encrypted secrets (committed)
``` ```
`site.yaml` is the declarative source of truth the CLI reads/writes; everything `site.yaml` is the source of truth the CLI reads/writes; everything else is
else is rendered from it. rendered from it. It carries both ids:
```yaml
customer: acme
site: shop
profile: custom-php
domains: [shop.acme.com]
```
## 8. Reserved words ## 8. Reserved words
Slugs may not be: `proxy`, `platform`, `traefik`, `mariadb`, `forgejo`, `customer` and `site` ids may not be: `proxy`, `platform`, `traefik`, `mariadb`,
`monitoring`, `docker`, `customers`, `platform-infra`, `site-templates`, `forgejo`, `monitoring`, `docker`, `customers`, `platform-infra`,
`deployments`, `control-panel` (avoids collisions with platform names). `site-templates`, `deployments`, `control-panel` (avoids collisions with
platform names).

17
docs/04-site-profiles.md
View file

@ -7,7 +7,7 @@ domains, and options. Four profiles cover all current workloads.
Common to every profile: Common to every profile:
- Traefik labels for routing + TLS (Host rule from domains, ACME resolver). - Traefik labels for routing + TLS (Host rule from domains, ACME resolver).
- Joined to the site's own `<slug>_net` network; only Traefik bridges to `proxy`. - Joined to the site's own `<slug>_net` network; only Traefik bridges to `proxy`.
- Web root bind-mounted from `tank/customers/<slug>/web`. - Web root bind-mounted from `tank/customers/<customer>/<site>/web`.
- Per-project CPU/memory limits. - Per-project CPU/memory limits.
- Containers run non-root; rootfs read-only where the profile allows. - Containers run non-root; rootfs read-only where the profile allows.
@ -24,7 +24,7 @@ Common to every profile:
- **Containers:** one nginx serving the web root read-only. For very simple - **Containers:** one nginx serving the web root read-only. For very simple
cases, Traefik can serve files directly with no container. cases, Traefik can serve files directly with no container.
- **Volumes:** `tank/customers/<slug>/web` → `/usr/share/nginx/html` (read-only). - **Volumes:** `tank/customers/<customer>/<site>/web` → `/usr/share/nginx/html` (read-only).
- **DB:** none. - **DB:** none.
- **Notes:** cheapest profile; near-zero backup incrementals when unchanged. - **Notes:** cheapest profile; near-zero backup incrementals when unchanged.
@ -43,8 +43,8 @@ Common to every profile:
- **Containers:** - **Containers:**
- `nginx` — serves static assets, proxies `.php` to fpm over FastCGI. - `nginx` — serves static assets, proxies `.php` to fpm over FastCGI.
- `php-fpm` — non-root; only the web root (and a small tmp) writable. - `php-fpm` — non-root; only the web root (and a small tmp) writable.
- **Volumes:** `tank/customers/<slug>/web` shared by both (web root). - **Volumes:** `tank/customers/<customer>/<site>/web` shared by both (web root).
- **DB:** optional — if requested, a `db_<slug>` + `u_<slug>` on shared MariaDB; - **DB:** optional — if requested, a `db_<customer>_<site>` + `u_<customer>_<site>` on shared MariaDB;
credentials injected via env from `secrets.enc.yaml`. credentials injected via env from `secrets.enc.yaml`.
- **Options:** PHP version (pinned image tag), extensions, `php.ini` overrides, - **Options:** PHP version (pinned image tag), extensions, `php.ini` overrides,
cron (via a scheduled fpm exec) if needed. cron (via a scheduled fpm exec) if needed.
@ -53,9 +53,9 @@ Common to every profile:
- **Containers:** same nginx + php-fpm pair as `custom-php`, using a WordPress - **Containers:** same nginx + php-fpm pair as `custom-php`, using a WordPress
base image (or WP installed into the web root on first provision). base image (or WP installed into the web root on first provision).
- **Volumes:** `tank/customers/<slug>/web` (WordPress core, themes, plugins, - **Volumes:** `tank/customers/<customer>/<site>/web` (WordPress core, themes, plugins,
uploads). uploads).
- **DB:** **required** — `db_<slug>` + `u_<slug>` on shared MariaDB. - **DB:** **required** — `db_<customer>_<site>` + `u_<customer>_<site>` on shared MariaDB.
- **Hardening baked in:** - **Hardening baked in:**
- php-fpm non-root; `wp-content/uploads` writable, PHP execution denied there - php-fpm non-root; `wp-content/uploads` writable, PHP execution denied there
(nginx rule) to blunt upload-based RCE. (nginx rule) to blunt upload-based RCE.
@ -73,9 +73,10 @@ Every profile is rendered from the same declarative fields; unused fields are
ignored per profile: ignored per profile:
```yaml ```yaml
slug: example-com customer: acme # owner id
site: acme-com # site id, unique within the customer
profile: wordpress # static | redirect | custom-php | wordpress profile: wordpress # static | redirect | custom-php | wordpress
domains: # first is primary → drives cert + slug domains: # first is primary → drives cert + default site id
- example.com - example.com
- www.example.com - www.example.com
php_version: "8.3" # custom-php / wordpress php_version: "8.3" # custom-php / wordpress

47
docs/05-provisioning-workflow.md
View file

@ -13,23 +13,25 @@ failed provision can be resumed.
## New site — `provision` ## New site — `provision`
Input: `slug`, `profile`, `domains`, profile options (see Input: `customer`, `site`, `profile`, `domains`, profile options (see
[04](04-site-profiles.md) `site.yaml`). [04](04-site-profiles.md) `site.yaml`). The flattened `slug` = `<customer>-<site>`.
1. **Validate** — slug charset/reserved-word check 1. **Validate** — customer/site charset + reserved-word check
([03](03-naming-conventions.md)), domains resolvable/owned, profile known. ([03](03-naming-conventions.md)), domains resolvable/owned, profile known.
2. **ZFS** — create `tank/customers/<slug>/web` (and `db-backups/<slug>` if the 2. **ZFS** — ensure `tank/customers/<customer>` exists, then create
profile uses a DB). `tank/customers/<customer>/<site>/web` (and `db-backups/<customer>/<site>` if
3. **Database** (DB profiles) — create `db_<slug>` + `u_<slug>` with least the profile uses a DB).
privilege on shared MariaDB; generate password; write to `secrets.enc.yaml` 3. **Database** (DB profiles) — create `db_<customer>_<site>` +
(SOPS/age). `u_<customer>_<site>` with least privilege on shared MariaDB; generate
4. **Render** — produce `deployments/<slug>/docker-compose.yml` + `.env` from the password; write to `secrets.enc.yaml` (SOPS/age).
profile template and `site.yaml`. 4. **Render** — produce `deployments/<customer>/<site>/docker-compose.yml` +
5. **SFTP** — create chrooted `sftp_<slug>` account bound to the web root. `.env` from the profile template and `site.yaml`.
6. **Commit** — commit `deployments/<slug>/` to the `deployments` repo (audit 5. **SFTP** — ensure the customer's chrooted `sftp_<customer>` account exists
trail; secrets committed only in encrypted form). (chroot `tank/customers/<customer>/`); the new site appears as a sub-folder.
7. **Deploy** — `docker compose up -d` in `deployments/<slug>/`. Traefik 6. **Commit** — commit `deployments/<customer>/<site>/` to the `deployments` repo
discovers the route from labels; ACME issues the certificate. (audit trail; secrets committed only in encrypted form).
7. **Deploy** — `docker compose up -d` in `deployments/<customer>/<site>/`.
Traefik discovers the route from labels; ACME issues the certificate.
8. **Verify** — HTTPS reachability + valid cert; for WordPress, run WP-CLI 8. **Verify** — HTTPS reachability + valid cert; for WordPress, run WP-CLI
install. Print access details. install. Print access details.
@ -47,11 +49,13 @@ delta. TLS for new domains is automatic.
1. `docker compose down` (optionally `--remove-orphans`). 1. `docker compose down` (optionally `--remove-orphans`).
2. **Final backup** — take a last ZFS snapshot + final DB dump, retained per the 2. **Final backup** — take a last ZFS snapshot + final DB dump, retained per the
deprovision retention policy before deletion. deprovision retention policy before deletion.
3. Drop `db_<slug>` + `u_<slug>` (after the final dump). 3. Drop `db_<customer>_<site>` + `u_<customer>_<site>` (after the final dump).
4. Remove the SFTP account. 4. Remove the site's SFTP sub-folder access; remove the `sftp_<customer>` account
5. Destroy `tank/customers/<slug>/web` (and `db-backups/<slug>`) **after** the only when the customer has no remaining sites.
retention window — never immediately. 5. Destroy `tank/customers/<customer>/<site>` (and `db-backups/<customer>/<site>`)
6. Remove `deployments/<slug>/` and commit. **after** the retention window — never immediately. Destroy the customer
dataset only when their last site is removed.
6. Remove `deployments/<customer>/<site>/` and commit.
> Destroys are gated: the CLI refuses to delete data younger than the retention > Destroys are gated: the CLI refuses to delete data younger than the retention
> window without an explicit `--force`, and always snapshots before destroying. > window without an explicit `--force`, and always snapshots before destroying.
@ -60,7 +64,8 @@ delta. TLS for new domains is automatic.
Routine backups run on a schedule (not per-command); restore is on demand. Both Routine backups run on a schedule (not per-command); restore is on demand. Both
are specified in the [Backup & DR runbook](06-backup-and-dr.md). CLI surface: are specified in the [Backup & DR runbook](06-backup-and-dr.md). CLI surface:
`backup <slug>` (ad-hoc), `restore <slug> --snapshot <name> --db <dump>`. `backup <customer> <site>` (ad-hoc),
`restore <customer> <site> --snapshot <name> --db <dump>`.
## CLI command summary ## CLI command summary

21
docs/06-backup-and-dr.md
View file

@ -7,7 +7,7 @@ offsite transfer cheap while preserving per-customer restore.
| Stream | Data | Mechanism | Onsite | Offsite | | Stream | Data | Mechanism | Onsite | Offsite |
|--------|------|-----------|--------|---------| |--------|------|-----------|--------|---------|
| A | Web files | ZFS snapshot + `zfs send` | snapshots on `tank` | incremental `send` to offsite pool | | A | Web files | ZFS snapshot + `zfs send` | snapshots on `tank` | incremental `send` to offsite pool |
| B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<slug>` | rsync to offsite | | B | Databases | automysqlbackup (per-DB, rotated) | `tank/platform/db-backups/<customer>/<site>` | rsync to offsite |
| C | Logs | Promtail → Loki | Loki store | (per Loki retention; not customer-restore data) | | C | Logs | Promtail → Loki | Loki store | (per Loki retention; not customer-restore data) |
Platform state (Traefik ACME store, Forgejo data, `deployments` repo, Prometheus Platform state (Traefik ACME store, Forgejo data, `deployments` repo, Prometheus
@ -32,26 +32,29 @@ config) is backed up with the same ZFS mechanism from `tank/platform/*`.
- **DB dumps:** rsync to the same or another offsite location (delta transfer). - **DB dumps:** rsync to the same or another offsite location (delta transfer).
- Both offsite copies are the recovery source if the primary host is lost. - Both offsite copies are the recovery source if the primary host is lost.
## 3. Restore — single customer (the critical drill) ## 3. Restore — single site (the critical drill)
Two coordinated steps: Two coordinated steps:
**A. Web files (ZFS)** **A. Web files (ZFS)**
1. Identify the target snapshot: `tank/customers/<slug>/web@auto-...`. 1. Identify the target snapshot: `tank/customers/<customer>/<site>/web@auto-...`.
2. Restore by clone/rollback (or `zfs receive` from offsite if the host is 2. Restore by clone/rollback (or `zfs receive` from offsite if the host is
gone) into `tank/customers/<slug>/web`. gone) into `tank/customers/<customer>/<site>/web`.
**B. Database (dump)** **B. Database (dump)**
3. Pick the matching dump from `db-backups/<slug>` (or offsite). 3. Pick the matching dump from `db-backups/<customer>/<site>` (or offsite).
4. Recreate `db_<slug>` + `u_<slug>` if needed; import the dump into shared 4. Recreate `db_<customer>_<site>` + `u_<customer>_<site>` if needed; import the
MariaDB. dump into shared MariaDB.
**C. Bring up** **C. Bring up**
5. `docker compose up -d` in `deployments/<slug>/`. 5. `docker compose up -d` in `deployments/<customer>/<site>/`.
6. Verify site + data; for WordPress confirm site URL / run WP-CLI 6. Verify site + data; for WordPress confirm site URL / run WP-CLI
`search-replace` if the domain changed. `search-replace` if the domain changed.
CLI: `restore <slug> --snapshot <name> --db <dump-file>` wraps A–C. CLI: `restore <customer> <site> --snapshot <name> --db <dump-file>` wraps A–C.
> To restore an entire customer at once, repeat A–C per site, or use the
> recursive snapshot `tank/customers/<customer>@...` as the file source.
## 4. Restore — full host (DR rebuild) ## 4. Restore — full host (DR rebuild)

2
docs/07-repo-layout-gitops.md
View file

@ -56,7 +56,7 @@ calling the same operations.
## 3. GitOps flow ## 3. GitOps flow
``` ```
edit site.yaml ─▶ CLI renders ─▶ commit deployments/<slug> ─▶ compose up -d edit site.yaml ─▶ CLI renders ─▶ commit deployments/<cust>/<site> ─▶ compose up -d
(intent) (from templates) (audit trail) (converge) (intent) (from templates) (audit trail) (converge)
``` ```

12
docs/adr/0005-zfs-per-customer.md
View file

@ -1,4 +1,4 @@
# ADR 0005 — ZFS dataset per customer # ADR 0005 — ZFS dataset per site (nested under customer)
**Status:** Accepted **Status:** Accepted
@ -8,10 +8,12 @@ touching neighbours. ZFS offers checksummed integrity, cheap copy-on-write
snapshots, and incremental `zfs send`. snapshots, and incremental `zfs send`.
## Decision ## Decision
Create a **ZFS dataset per customer** (`tank/customers/<customer>/web`) holding Create a **ZFS dataset per site**, nested under the owning customer
**only the web root**. Docker image/layer storage and the MariaDB datadir live (`tank/customers/<customer>/<site>/web`), holding **only the web root**. The
on separate platform datasets. Customer web data is bind-mounted into containers customer level is a grouping parent enabling recursive snapshots and a single
from the customer dataset. SFTP chroot; the site level is the isolation/restore unit. Docker image/layer
storage and the MariaDB datadir live on separate platform datasets. Site web
data is bind-mounted into containers from its dataset.
## Consequences ## Consequences
- ✅ Snapshot and `zfs send` operate at the customer granularity — restore one - ✅ Snapshot and `zfs send` operate at the customer granularity — restore one

2
docs/adr/README.md
View file

@ -9,7 +9,7 @@ Decision → Consequences. Status is one of Proposed / Accepted / Superseded.
| [0002](0002-traefik-as-edge.md) | Traefik as the edge router | Accepted | | [0002](0002-traefik-as-edge.md) | Traefik as the edge router | Accepted |
| [0003](0003-nginx-fpm-per-site.md) | Per-site nginx + php-fpm behind Traefik | Accepted | | [0003](0003-nginx-fpm-per-site.md) | Per-site nginx + php-fpm behind Traefik | Accepted |
| [0004](0004-shared-mariadb.md) | Shared MariaDB instance, per-site DB + user | Accepted | | [0004](0004-shared-mariadb.md) | Shared MariaDB instance, per-site DB + user | Accepted |
| [0005](0005-zfs-per-customer.md) | ZFS dataset per customer | Accepted | | [0005](0005-zfs-per-customer.md) | ZFS dataset per site (nested under customer) | Accepted |
| [0006](0006-decoupled-backup-streams.md) | Decoupled backup streams (files vs DB vs logs) | Accepted | | [0006](0006-decoupled-backup-streams.md) | Decoupled backup streams (files vs DB vs logs) | Accepted |
| [0007](0007-forgejo-over-gitlab.md) | Forgejo (lightweight) over GitLab | Accepted | | [0007](0007-forgejo-over-gitlab.md) | Forgejo (lightweight) over GitLab | Accepted |
| [0008](0008-cli-first-panel-later.md) | CLI/templating first, customer panel later | Accepted | | [0008](0008-cli-first-panel-later.md) | CLI/templating first, customer panel later | Accepted |

5
docs/architecture-plan.md
View file

@ -1,5 +1,10 @@
# Rebuild: Multi-Tenant Web Hosting Platform (heleosv2) # Rebuild: Multi-Tenant Web Hosting Platform (heleosv2)
> **Note:** This is the point-in-time *approved plan*. Where it refers to the
> per-customer boundary/naming, the design was later refined to a nested
> **customer → site** model — see [`03-naming-conventions.md`](03-naming-conventions.md),
> which is authoritative for naming and paths.
## Context ## Context
An existing small web-hosting business runs nginx + PHP-FPM + MySQL on a shared An existing small web-hosting business runs nginx + PHP-FPM + MySQL on a shared