Compare commits
No commits in common. "a90d07875d30dd6d32824985102ce756fe827e98" and "f2ab25c99fdd6f4aec7b19cdee8f09b2aa3cb9fa" have entirely different histories.
a90d07875d
...
f2ab25c99f
37 changed files with 60 additions and 1647 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -34,6 +34,3 @@ __pycache__/
|
|||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
*.egg-info/
|
||||
.pytest_cache/
|
||||
control-panel/config.yaml
|
||||
|
|
|
|||
17
CLAUDE.md
17
CLAUDE.md
|
|
@ -25,21 +25,8 @@ host, Docker Compose per site, no orchestrator.
|
|||
- ✅ Phase 1 — host baseline Ansible ([platform-infra/ansible/](platform-infra/ansible/README.md))
|
||||
- ✅ Phase 2 — platform services (Traefik, MariaDB, Forgejo) in [platform-infra/stacks/](platform-infra/stacks/README.md)
|
||||
- ✅ Phase 3 — site templates & base images in [site-templates/](site-templates/README.md)
|
||||
- ✅ Phase 4 — provisioning CLI `heleosctl` in [control-panel/](control-panel/README.md) (provision/deprovision/list/render/backup/restore; `--dry-run`; 22 tests)
|
||||
- ✅ Phase 5 — backup/DR automation in the `backup` Ansible role (sanoid snapshots, per-DB dump timer, optional syncoid/rsync offsite)
|
||||
- 🚧 Phase 6 — observability (Prometheus/Grafana + cAdvisor/node_exporter/Traefik metrics, Loki/Promtail, Uptime-Kuma)
|
||||
- ⬜ Phases 7–8 — migration, customer panel
|
||||
|
||||
### Current focus (Aug 2026)
|
||||
Bringing up Phases 1–5 on a **cost-optimized test VM** (Ubuntu). ZFS is
|
||||
**simulated via a file-backed pool** (`zfs_pool_mode: file`, default) since the
|
||||
VM has no spare disk; the `zfs` role also supports `single`/`mirror` for a real
|
||||
host later. Existing box `korat` (Hetzner, in prod) is NOT the target — a fresh
|
||||
VM is. Next: finish the VM playbook run, then Phase 2 stacks on the VM (or
|
||||
author Phase 6 observability).
|
||||
|
||||
### Phase 4 follow-ups (not yet done)
|
||||
- `reconfigure` + `rotate-secret` commands; git-commit of `deployments/` on provision; php-fpm umask for two-way SFTP editing (see control-panel/README.md).
|
||||
- 🚧 Phase 4 — provisioning CLI in `control-panel/` (renders templates → deployments/, creates ZFS+DB+SFTP, `compose up`)
|
||||
- ⬜ Phases 5–8 — backup/DR, observability, migration, panel
|
||||
|
||||
## Repo map
|
||||
| Path | Purpose |
|
||||
|
|
|
|||
|
|
@ -1,87 +1,18 @@
|
|||
# control-panel — heleosctl
|
||||
# control-panel
|
||||
|
||||
Provisioning CLI for the platform. Reads a site's `site.yaml`, renders the
|
||||
[site-templates](../site-templates/README.md) into
|
||||
[deployments/](../deployments/README.md), creates the ZFS dataset + database +
|
||||
SFTP account, and runs `docker compose up`. Implements the flow in
|
||||
[docs/05-provisioning-workflow.md](../docs/05-provisioning-workflow.md).
|
||||
The provisioning CLI (Phase 4), later the customer-facing web panel (Phase 8).
|
||||
Empty until Phase 4.
|
||||
|
||||
> Runs **on the host** (Linux), typically as root, since it drives `zfs`,
|
||||
> `docker`, `useradd`, and `systemctl`. Use `--dry-run` to preview every action
|
||||
> first — nothing touches the system until you drop the flag.
|
||||
**Planned CLI surface** (see
|
||||
[../docs/05-provisioning-workflow.md](../docs/05-provisioning-workflow.md)):
|
||||
|
||||
## Install
|
||||
```bash
|
||||
cd control-panel
|
||||
python -m venv .venv && . .venv/bin/activate
|
||||
pip install -e ".[dev]" # installs the `heleosctl` command + pytest
|
||||
cp config.example.yaml config.yaml && $EDITOR config.yaml
|
||||
```
|
||||
Config is found via `--config`, `$HELEOS_CONFIG`, `./config.yaml`, or
|
||||
`/etc/heleos/config.yaml`.
|
||||
| Command | Action |
|
||||
|---------|--------|
|
||||
| `provision` | Create a new site end-to-end (ZFS + DB + compose + route + SFTP). |
|
||||
| `reconfigure` | Apply changes from an edited `site.yaml`. |
|
||||
| `deprovision` | Tear down a site with gated, backed-up deletion. |
|
||||
| `list` | Show all sites, profiles, status. |
|
||||
| `backup` / `restore` | Ad-hoc backup; two-step (files + DB) restore. |
|
||||
| `rotate-secret` | Regenerate DB/SFTP credentials. |
|
||||
|
||||
## Usage
|
||||
```bash
|
||||
# Preview everything (no changes)
|
||||
heleosctl --dry-run provision -f examples/acme-shop.site.yaml
|
||||
|
||||
# Provision for real
|
||||
heleosctl provision -f examples/acme-shop.site.yaml
|
||||
|
||||
# Inspect / operate on an existing site (by customer+site or by file)
|
||||
heleosctl list
|
||||
heleosctl render -c acme -s shop # print rendered config
|
||||
heleosctl backup -c acme -s shop # snapshot + DB dump
|
||||
heleosctl restore -c acme -s shop --snapshot auto-20260101-0300 --db /path/dump.sql
|
||||
heleosctl deprovision -c acme -s shop # stop + drop DB, KEEP data
|
||||
heleosctl deprovision -c acme -s shop --purge # also destroy the dataset (after a final backup)
|
||||
```
|
||||
|
||||
## What each command does
|
||||
| Command | Actions |
|
||||
|---------|---------|
|
||||
| `provision` | ZFS web dataset + ownership → DB + user (if any) → `.env` + SOPS encrypt → render compose/nginx/site.yaml → per-customer SFTP account → `compose up -d`. |
|
||||
| `deprovision` | `compose down` → final backup → drop DB/user → (with `--purge`) destroy dataset + remove config. |
|
||||
| `render` | Print the rendered files without writing (debugging). |
|
||||
| `backup` | ZFS snapshot + `mariadb-dump` into `db-backups/<customer>/<site>`. |
|
||||
| `restore` | ZFS rollback + import a DB dump. |
|
||||
| `list` | Table of provisioned sites found under the deployments dir. |
|
||||
|
||||
## Design notes
|
||||
- **Idempotent & declarative:** `site.yaml` is the source of truth; steps use
|
||||
`zfs create -p`, `CREATE ... IF NOT EXISTS`, and existence checks so re-running
|
||||
converges. A customer's SFTP password is set only when the account is first
|
||||
created.
|
||||
- **Secrets:** the DB password is generated per site, written to a git-ignored
|
||||
`.env`, and encrypted to `secrets.enc.yaml` via SOPS/age (set
|
||||
`sops_age_recipient`). Passwords never appear in command logs (`--redacted`).
|
||||
- **Safety:** all host mutations go through one runner supporting `--dry-run`;
|
||||
data is destroyed only with `--purge`, always after a final backup.
|
||||
|
||||
## Layout
|
||||
```
|
||||
src/heleos/
|
||||
cli.py # click commands
|
||||
config.py # Config + Site loading/validation
|
||||
naming.py # slug / db / sftp identifier rules (docs/03)
|
||||
context.py # Site -> template variables
|
||||
render.py # Jinja2 render of a profile
|
||||
runner.py # command + file runner with --dry-run
|
||||
zfs.py database.py secrets.py sftp.py compose.py # host ops
|
||||
provision.py # provision / deprovision orchestration
|
||||
backup.py # snapshot/dump + restore
|
||||
tests/ # pure-logic unit tests (naming, config, render)
|
||||
```
|
||||
|
||||
## Test
|
||||
```bash
|
||||
pip install -e ".[dev]" && pytest # or: PYTHONPATH=src pytest
|
||||
```
|
||||
|
||||
## Known follow-ups
|
||||
- **Web-root umask for php-fpm:** SFTP writes with umask 0002 (group-writable);
|
||||
php-fpm-created files may need the same for two-way SFTP editing. See the note
|
||||
in [site-templates/README.md](../site-templates/README.md).
|
||||
- **git commit of `deployments/`** on provision (GitOps audit trail) — currently
|
||||
left to the operator; wire into the flow when the deployments repo is set up.
|
||||
- **`reconfigure` / `rotate-secret`** commands (documented in docs/05) — to add.
|
||||
Idempotent, declarative (`site.yaml` is source of truth), logs to Loki.
|
||||
|
|
|
|||
|
|
@ -1,34 +0,0 @@
|
|||
# heleos control-panel config. Copy to config.yaml (or /etc/heleos/config.yaml,
|
||||
# or point at it with HELEOS_CONFIG) and adjust for your host.
|
||||
|
||||
pool: tank
|
||||
customers_root: /tank/customers
|
||||
|
||||
# Where rendered per-site configs are written (the deployments repo/dir).
|
||||
deployments_dir: /opt/heleos/deployments
|
||||
# Where the profile templates live (this monorepo's site-templates/profiles).
|
||||
templates_dir: /opt/heleos/site-templates/profiles
|
||||
|
||||
# Container image registry + image reference templates.
|
||||
registry: git.example.com/heleos
|
||||
images:
|
||||
nginx: "{registry}/nginx:latest"
|
||||
php_fpm: "{registry}/php-fpm:{php_version}"
|
||||
wordpress: "wordpress:{php_version}-fpm-alpine"
|
||||
|
||||
# Shared MariaDB. Root password is read from this .env (or the
|
||||
# MARIADB_ROOT_PASSWORD environment variable).
|
||||
mariadb_container: mariadb
|
||||
mariadb_env_file: /opt/heleos/platform-infra/stacks/mariadb/.env
|
||||
|
||||
# SOPS/age recipient (public key) used to encrypt each site's .env into
|
||||
# secrets.enc.yaml. Leave empty to skip encryption (dev only).
|
||||
sops_age_recipient: ""
|
||||
|
||||
# fpm user/group the web root is owned by (matches the alpine images: www-data).
|
||||
fpm_uid: 82
|
||||
fpm_gid: 82
|
||||
|
||||
default_resources:
|
||||
cpu: "1.0"
|
||||
memory: "512m"
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
# Example site definition (custom-php with a database).
|
||||
customer: acme
|
||||
site: shop
|
||||
profile: custom-php
|
||||
domains:
|
||||
- shop.acme.com
|
||||
- www.shop.acme.com
|
||||
php_version: "8.3"
|
||||
database: true
|
||||
resources:
|
||||
cpu: "1.0"
|
||||
memory: "512m"
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
[project]
|
||||
name = "heleos-control-panel"
|
||||
version = "0.1.0"
|
||||
description = "Provisioning CLI for the heleos multi-tenant hosting platform"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"click>=8.1",
|
||||
"jinja2>=3.1",
|
||||
"pyyaml>=6.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=8.0"]
|
||||
|
||||
[project.scripts]
|
||||
heleosctl = "heleos.cli:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
"""heleos control-panel — provisioning CLI for the multi-tenant hosting platform."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
"""Backup/restore for a single site: ZFS snapshot for files, mariadb-dump for DB.
|
||||
|
||||
Two decoupled streams (see docs/06-backup-and-dr.md). Offsite `zfs send` / rsync
|
||||
are scheduled separately (Phase 5); this module covers the on-host operations the
|
||||
CLI drives.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from . import database, naming, zfs
|
||||
from .config import Config, Site
|
||||
from .runner import Runner
|
||||
|
||||
|
||||
def db_backup_dir(cfg: Config, site: Site) -> str:
|
||||
return f"/{cfg.pool}/platform/db-backups/{site.customer}/{site.site}"
|
||||
|
||||
|
||||
def dump_database(runner: Runner, cfg: Config, site: Site) -> str | None:
|
||||
if not site.database:
|
||||
return None
|
||||
db = naming.database_name(site.customer, site.site)
|
||||
ts = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
out_dir = db_backup_dir(cfg, site)
|
||||
dest = f"{out_dir}/{db}-{ts}.sql"
|
||||
runner.mkdir(out_dir)
|
||||
pw = "<redacted>" if runner.dry_run else database.root_password(cfg)
|
||||
sql = runner.run(
|
||||
["docker", "exec", cfg.mariadb_container, "mariadb-dump",
|
||||
"--single-transaction", "--databases", db, "-uroot", f"-p{pw}"],
|
||||
capture=True, secret=True,
|
||||
)
|
||||
if not runner.dry_run:
|
||||
runner.write_file(dest, sql)
|
||||
else:
|
||||
runner._echo(f"DRY write {dest}")
|
||||
return dest
|
||||
|
||||
|
||||
def backup_site(runner: Runner, cfg: Config, site: Site) -> dict:
|
||||
snap = zfs.snapshot(runner, zfs.web_dataset(cfg, site))
|
||||
dump = dump_database(runner, cfg, site)
|
||||
return {"snapshot": snap, "dump": dump}
|
||||
|
||||
|
||||
def restore_files(runner: Runner, cfg: Config, site: Site, snapshot: str) -> None:
|
||||
ds = zfs.web_dataset(cfg, site)
|
||||
name = snapshot if "@" in snapshot else f"{ds}@{snapshot}"
|
||||
# -r discards snapshots newer than the target; acceptable for a deliberate restore.
|
||||
runner.run(["zfs", "rollback", "-r", name])
|
||||
|
||||
|
||||
def restore_database(runner: Runner, cfg: Config, site: Site, dump_path: str) -> None:
|
||||
if not site.database:
|
||||
return
|
||||
pw = "<redacted>" if runner.dry_run else database.root_password(cfg)
|
||||
data = "" if runner.dry_run else Path(dump_path).read_text(encoding="utf-8")
|
||||
runner.run(
|
||||
["docker", "exec", "-i", cfg.mariadb_container, "mariadb", "-uroot", f"-p{pw}"],
|
||||
input=data, secret=True,
|
||||
)
|
||||
|
|
@ -1,149 +0,0 @@
|
|||
"""heleosctl — command-line entry point."""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import yaml
|
||||
|
||||
from . import provision as provision_mod
|
||||
from . import backup as backup_mod
|
||||
from . import render as render_mod
|
||||
from .config import Config, Site
|
||||
from .errors import HeleosError
|
||||
from .runner import Runner
|
||||
|
||||
|
||||
def _load_site(cfg: Config, file: str | None, customer: str | None, site: str | None) -> Site:
|
||||
if file:
|
||||
return Site.load(file)
|
||||
if customer and site:
|
||||
path = cfg.deployments_dir / customer / site / "site.yaml"
|
||||
if not path.is_file():
|
||||
raise HeleosError(f"no site.yaml at {path}")
|
||||
return Site.load(path)
|
||||
raise HeleosError("provide --file, or both --customer and --site")
|
||||
|
||||
|
||||
@click.group()
|
||||
@click.option("--config", "config_path", default=None, help="Path to config.yaml.")
|
||||
@click.option("--dry-run", is_flag=True, help="Print actions without executing them.")
|
||||
@click.option("--quiet", is_flag=True, help="Suppress the command log.")
|
||||
@click.pass_context
|
||||
def cli(ctx: click.Context, config_path, dry_run, quiet):
|
||||
"""Provisioning CLI for the heleos hosting platform."""
|
||||
cfg = Config.load(config_path)
|
||||
ctx.obj = {
|
||||
"cfg": cfg,
|
||||
"runner": Runner(dry_run=dry_run, verbose=not quiet),
|
||||
}
|
||||
|
||||
|
||||
# Shared site-selection options.
|
||||
def _site_opts(f):
|
||||
f = click.option("-f", "--file", default=None, help="Path to a site.yaml.")(f)
|
||||
f = click.option("-c", "--customer", default=None)(f)
|
||||
f = click.option("-s", "--site", default=None)(f)
|
||||
return f
|
||||
|
||||
|
||||
@cli.command()
|
||||
@_site_opts
|
||||
@click.pass_context
|
||||
def provision(ctx, file, customer, site):
|
||||
"""Provision a site from its site.yaml."""
|
||||
cfg, runner = ctx.obj["cfg"], ctx.obj["runner"]
|
||||
s = _load_site(cfg, file, customer, site)
|
||||
summary = provision_mod.provision(cfg, s, runner)
|
||||
click.echo(yaml.safe_dump({"provisioned": summary}, sort_keys=False))
|
||||
|
||||
|
||||
@cli.command()
|
||||
@_site_opts
|
||||
@click.option("--purge", is_flag=True, help="Destroy the site's data (after a final backup).")
|
||||
@click.option("--no-backup", is_flag=True, help="Skip the final backup on deprovision.")
|
||||
@click.pass_context
|
||||
def deprovision(ctx, file, customer, site, purge, no_backup):
|
||||
"""Tear down a site. Data is kept unless --purge is given."""
|
||||
cfg, runner = ctx.obj["cfg"], ctx.obj["runner"]
|
||||
s = _load_site(cfg, file, customer, site)
|
||||
summary = provision_mod.deprovision(cfg, s, runner, purge=purge, final_backup=not no_backup)
|
||||
click.echo(yaml.safe_dump({"deprovisioned": summary}, sort_keys=False))
|
||||
|
||||
|
||||
@cli.command()
|
||||
@_site_opts
|
||||
@click.pass_context
|
||||
def render(ctx, file, customer, site):
|
||||
"""Render a site's templates to stdout (no writes)."""
|
||||
cfg = ctx.obj["cfg"]
|
||||
s = _load_site(cfg, file, customer, site)
|
||||
for name, content in render_mod.render(cfg, s).items():
|
||||
click.echo(f"# ===== {name} =====")
|
||||
click.echo(content)
|
||||
|
||||
|
||||
@cli.command()
|
||||
@_site_opts
|
||||
@click.pass_context
|
||||
def backup(ctx, file, customer, site):
|
||||
"""Snapshot a site's files + dump its database."""
|
||||
cfg, runner = ctx.obj["cfg"], ctx.obj["runner"]
|
||||
s = _load_site(cfg, file, customer, site)
|
||||
result = backup_mod.backup_site(runner, cfg, s)
|
||||
click.echo(yaml.safe_dump({"backup": result}, sort_keys=False))
|
||||
|
||||
|
||||
@cli.command()
|
||||
@_site_opts
|
||||
@click.option("--snapshot", required=True, help="Snapshot name or full dataset@snap.")
|
||||
@click.option("--db", "dump", default=None, help="Path to a DB dump to restore.")
|
||||
@click.pass_context
|
||||
def restore(ctx, file, customer, site, snapshot, dump):
|
||||
"""Restore a site's files (from a snapshot) and DB (from a dump)."""
|
||||
cfg, runner = ctx.obj["cfg"], ctx.obj["runner"]
|
||||
s = _load_site(cfg, file, customer, site)
|
||||
backup_mod.restore_files(runner, cfg, s, snapshot)
|
||||
if dump:
|
||||
backup_mod.restore_database(runner, cfg, s, dump)
|
||||
click.echo("restore complete")
|
||||
|
||||
|
||||
@cli.command(name="list")
|
||||
@click.pass_context
|
||||
def list_sites(ctx):
|
||||
"""List provisioned sites found under the deployments directory."""
|
||||
cfg = ctx.obj["cfg"]
|
||||
rows = []
|
||||
for meta in sorted(cfg.deployments_dir.glob("*/*/site.yaml")):
|
||||
data = yaml.safe_load(meta.read_text(encoding="utf-8")) or {}
|
||||
rows.append((data.get("customer", "?"), data.get("site", "?"),
|
||||
data.get("profile", "?"), ",".join(data.get("domains", []))))
|
||||
if not rows:
|
||||
click.echo("no sites provisioned")
|
||||
return
|
||||
width = [max(len(r[i]) for r in rows + [("customer", "site", "profile", "domains")]) for i in range(4)]
|
||||
header = ("customer", "site", "profile", "domains")
|
||||
click.echo(" ".join(h.ljust(width[i]) for i, h in enumerate(header)))
|
||||
for r in rows:
|
||||
click.echo(" ".join(r[i].ljust(width[i]) for i in range(4)))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
cli(standalone_mode=False)
|
||||
return 0
|
||||
except HeleosError as e:
|
||||
click.echo(f"error: {e}", err=True)
|
||||
return 1
|
||||
except click.ClickException as e:
|
||||
e.show()
|
||||
return e.exit_code
|
||||
except click.exceptions.Abort:
|
||||
click.echo("aborted", err=True)
|
||||
return 130
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
"""Docker Compose bring-up / tear-down for a site's deployment directory."""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from .runner import Runner
|
||||
|
||||
|
||||
def _base(deploy_dir: Path) -> list[str]:
|
||||
return ["docker", "compose",
|
||||
"--project-directory", str(deploy_dir),
|
||||
"-f", str(deploy_dir / "docker-compose.yml")]
|
||||
|
||||
|
||||
def up(runner: Runner, deploy_dir: Path) -> None:
|
||||
runner.run(_base(deploy_dir) + ["up", "-d"])
|
||||
|
||||
|
||||
def down(runner: Runner, deploy_dir: Path, *, remove_orphans: bool = True) -> None:
|
||||
cmd = _base(deploy_dir) + ["down"]
|
||||
if remove_orphans:
|
||||
cmd.append("--remove-orphans")
|
||||
runner.run(cmd)
|
||||
|
|
@ -1,106 +0,0 @@
|
|||
"""Platform + site configuration loading and validation."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from . import naming
|
||||
from .errors import ValidationError
|
||||
|
||||
PROFILES = {"static", "redirect", "custom-php", "wordpress"}
|
||||
CONFIG_ENV = "HELEOS_CONFIG"
|
||||
DEFAULT_CONFIG_PATHS = ("./config.yaml", "/etc/heleos/config.yaml")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
pool: str = "tank"
|
||||
customers_root: str = "/tank/customers"
|
||||
deployments_dir: Path = Path("deployments")
|
||||
templates_dir: Path = Path("site-templates/profiles")
|
||||
registry: str = "git.example.com/heleos"
|
||||
images: dict = field(default_factory=lambda: {
|
||||
"nginx": "{registry}/nginx:latest",
|
||||
"php_fpm": "{registry}/php-fpm:{php_version}",
|
||||
"wordpress": "wordpress:{php_version}-fpm-alpine",
|
||||
})
|
||||
mariadb_container: str = "mariadb"
|
||||
mariadb_env_file: str = "platform-infra/stacks/mariadb/.env"
|
||||
sops_age_recipient: str = ""
|
||||
default_resources: dict = field(default_factory=lambda: {"cpu": "1.0", "memory": "512m"})
|
||||
fpm_uid: int = 82
|
||||
fpm_gid: int = 82
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: str | None = None) -> "Config":
|
||||
chosen = path or os.environ.get(CONFIG_ENV)
|
||||
if not chosen:
|
||||
chosen = next((p for p in DEFAULT_CONFIG_PATHS if Path(p).is_file()), None)
|
||||
data: dict = {}
|
||||
if chosen and Path(chosen).is_file():
|
||||
data = yaml.safe_load(Path(chosen).read_text(encoding="utf-8")) or {}
|
||||
cfg = cls(**{k: v for k, v in data.items() if k in cls.__dataclass_fields__})
|
||||
cfg.deployments_dir = Path(cfg.deployments_dir)
|
||||
cfg.templates_dir = Path(cfg.templates_dir)
|
||||
return cfg
|
||||
|
||||
def image(self, key: str, **fmt) -> str:
|
||||
return self.images[key].format(registry=self.registry, **fmt)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Site:
|
||||
customer: str
|
||||
site: str
|
||||
profile: str
|
||||
domains: list[str]
|
||||
php_version: str = "8.3"
|
||||
database: bool = False
|
||||
redirect_to: str | None = None
|
||||
redirect_code: int = 301
|
||||
resources: dict | None = None
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: str | Path) -> "Site":
|
||||
data = yaml.safe_load(Path(path).read_text(encoding="utf-8")) or {}
|
||||
return cls.from_dict(data)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "Site":
|
||||
known = {k: v for k, v in data.items() if k in cls.__dataclass_fields__}
|
||||
try:
|
||||
site = cls(**known)
|
||||
except TypeError as e:
|
||||
raise ValidationError(f"invalid site definition: {e}") from e
|
||||
site.validate()
|
||||
return site
|
||||
|
||||
def validate(self) -> None:
|
||||
naming.validate_customer(self.customer)
|
||||
naming.validate_site(self.site)
|
||||
if self.profile not in PROFILES:
|
||||
raise ValidationError(
|
||||
f"unknown profile {self.profile!r}; must be one of {sorted(PROFILES)}"
|
||||
)
|
||||
# WordPress always needs a database.
|
||||
if self.profile == "wordpress":
|
||||
self.database = True
|
||||
# Every profile needs at least one (source) domain for the Host rule.
|
||||
if not self.domains:
|
||||
raise ValidationError(f"{self.profile} site requires at least one domain")
|
||||
# Redirect additionally needs a target URL.
|
||||
if self.profile == "redirect" and not self.redirect_to:
|
||||
raise ValidationError("redirect profile requires redirect_to")
|
||||
if self.profile in {"static", "redirect"} and self.database:
|
||||
raise ValidationError(f"{self.profile} profile cannot have a database")
|
||||
|
||||
@property
|
||||
def slug(self) -> str:
|
||||
return naming.slug(self.customer, self.site)
|
||||
|
||||
@property
|
||||
def slug_underscored(self) -> str:
|
||||
return naming.slug_underscored(self.customer, self.site)
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
"""Build the render context (template variables) from Config + Site."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .config import Config, Site
|
||||
|
||||
|
||||
def webroot(cfg: Config, site: Site) -> str:
|
||||
return f"{cfg.customers_root}/{site.customer}/{site.site}/web"
|
||||
|
||||
|
||||
def build(cfg: Config, site: Site) -> dict:
|
||||
"""Return the variable dict used to render a profile's templates.
|
||||
|
||||
Matches the render-context table in site-templates/README.md.
|
||||
"""
|
||||
resources = site.resources or cfg.default_resources
|
||||
return {
|
||||
"customer": site.customer,
|
||||
"site": site.site,
|
||||
"slug": site.slug,
|
||||
"slug_underscored": site.slug_underscored,
|
||||
"profile": site.profile,
|
||||
"domains": list(site.domains),
|
||||
"webroot": webroot(cfg, site),
|
||||
"database": site.database,
|
||||
"redirect_to": site.redirect_to,
|
||||
"redirect_code": site.redirect_code,
|
||||
"php_version": site.php_version,
|
||||
"nginx_image": cfg.image("nginx"),
|
||||
"php_image": cfg.image("php_fpm", php_version=site.php_version),
|
||||
"wordpress_image": cfg.image("wordpress", php_version=site.php_version),
|
||||
"resources": resources,
|
||||
}
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
"""Shared-MariaDB operations: create/drop per-site database + least-priv user.
|
||||
|
||||
SQL is piped over stdin to ``docker exec -i mariadb mariadb`` so the root
|
||||
password never appears in the process list; the command itself is logged redacted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from . import naming
|
||||
from .config import Config, Site
|
||||
from .errors import CommandError
|
||||
from .runner import Runner
|
||||
|
||||
|
||||
def root_password(cfg: Config) -> str:
|
||||
val = os.environ.get("MARIADB_ROOT_PASSWORD")
|
||||
if val:
|
||||
return val
|
||||
p = Path(cfg.mariadb_env_file)
|
||||
if p.is_file():
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
if line.startswith("MARIADB_ROOT_PASSWORD="):
|
||||
return line.split("=", 1)[1].strip()
|
||||
raise CommandError(
|
||||
"MARIADB_ROOT_PASSWORD not set (env var or mariadb .env at "
|
||||
f"{cfg.mariadb_env_file})"
|
||||
)
|
||||
|
||||
|
||||
def _exec_sql(runner: Runner, cfg: Config, sql: str) -> None:
|
||||
pw = "<redacted>" if runner.dry_run else root_password(cfg)
|
||||
runner.run(
|
||||
["docker", "exec", "-i", cfg.mariadb_container, "mariadb", "-uroot", f"-p{pw}"],
|
||||
input=sql, secret=True,
|
||||
)
|
||||
|
||||
|
||||
def create_db_user(runner: Runner, cfg: Config, site: Site, password: str) -> None:
|
||||
db = naming.database_name(site.customer, site.site)
|
||||
user = naming.database_user(site.customer, site.site)
|
||||
sql = (
|
||||
f"CREATE DATABASE IF NOT EXISTS `{db}` "
|
||||
"CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;\n"
|
||||
f"CREATE USER IF NOT EXISTS '{user}'@'%' IDENTIFIED BY '{password}';\n"
|
||||
f"ALTER USER '{user}'@'%' IDENTIFIED BY '{password}';\n"
|
||||
f"GRANT ALL PRIVILEGES ON `{db}`.* TO '{user}'@'%';\n"
|
||||
"FLUSH PRIVILEGES;\n"
|
||||
)
|
||||
_exec_sql(runner, cfg, sql)
|
||||
|
||||
|
||||
def drop_db_user(runner: Runner, cfg: Config, site: Site) -> None:
|
||||
db = naming.database_name(site.customer, site.site)
|
||||
user = naming.database_user(site.customer, site.site)
|
||||
sql = (
|
||||
f"DROP USER IF EXISTS '{user}'@'%';\n"
|
||||
f"DROP DATABASE IF EXISTS `{db}`;\n"
|
||||
"FLUSH PRIVILEGES;\n"
|
||||
)
|
||||
_exec_sql(runner, cfg, sql)
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
"""Typed errors so the CLI can report clean messages instead of tracebacks."""
|
||||
|
||||
|
||||
class HeleosError(Exception):
|
||||
"""Base class for expected, user-facing errors."""
|
||||
|
||||
|
||||
class ValidationError(HeleosError):
|
||||
"""Invalid site.yaml, naming, or configuration."""
|
||||
|
||||
|
||||
class CommandError(HeleosError):
|
||||
"""A shelled-out command failed."""
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
"""Naming rules — the single source of the slug/identifier conventions.
|
||||
|
||||
Mirrors docs/03-naming-conventions.md. The slug (``<customer>-<site>``) is the
|
||||
join key across Docker, the database, and deployment paths.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from .errors import ValidationError
|
||||
|
||||
CUSTOMER_RE = re.compile(r"^[a-z][a-z0-9-]{1,31}$") # 2–32 chars, starts alpha
|
||||
SITE_RE = re.compile(r"^[a-z][a-z0-9-]{1,39}$") # 2–40 chars, starts alpha
|
||||
|
||||
# Names that would collide with platform resources (see docs/03 §8).
|
||||
RESERVED = {
|
||||
"proxy", "platform", "traefik", "mariadb", "forgejo", "monitoring", "docker",
|
||||
"customers", "platform-infra", "site-templates", "deployments", "control-panel",
|
||||
}
|
||||
|
||||
|
||||
def validate_customer(customer: str) -> str:
|
||||
if not CUSTOMER_RE.match(customer):
|
||||
raise ValidationError(
|
||||
f"invalid customer id {customer!r}: must match {CUSTOMER_RE.pattern}"
|
||||
)
|
||||
if customer in RESERVED:
|
||||
raise ValidationError(f"customer id {customer!r} is reserved")
|
||||
return customer
|
||||
|
||||
|
||||
def validate_site(site: str) -> str:
|
||||
if not SITE_RE.match(site):
|
||||
raise ValidationError(
|
||||
f"invalid site id {site!r}: must match {SITE_RE.pattern}"
|
||||
)
|
||||
if site in RESERVED:
|
||||
raise ValidationError(f"site id {site!r} is reserved")
|
||||
return site
|
||||
|
||||
|
||||
def slug(customer: str, site: str) -> str:
|
||||
"""Globally-unique flattened id used for Docker/router names."""
|
||||
return f"{customer}-{site}"
|
||||
|
||||
|
||||
def slug_underscored(customer: str, site: str) -> str:
|
||||
"""MySQL-safe identifier stem (hyphens → underscores)."""
|
||||
return slug(customer, site).replace("-", "_")
|
||||
|
||||
|
||||
def database_name(customer: str, site: str) -> str:
|
||||
return f"db_{slug_underscored(customer, site)}"
|
||||
|
||||
|
||||
def database_user(customer: str, site: str) -> str:
|
||||
return f"u_{slug_underscored(customer, site)}"
|
||||
|
||||
|
||||
def sftp_user(customer: str) -> str:
|
||||
return f"sftp_{customer}"
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
"""Provisioning orchestrator: ties ZFS, DB, secrets, render, SFTP and compose
|
||||
into the provision / deprovision flows described in docs/05-provisioning-workflow.md.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from . import backup, compose, database, render, secrets, sftp, zfs
|
||||
from .config import Config, Site
|
||||
from .runner import Runner
|
||||
|
||||
SITE_FIELDS = ("customer", "site", "profile", "domains", "php_version",
|
||||
"database", "redirect_to", "redirect_code", "resources")
|
||||
|
||||
|
||||
def deployment_dir(cfg: Config, site: Site) -> Path:
|
||||
return cfg.deployments_dir / site.customer / site.site
|
||||
|
||||
|
||||
def _site_to_dict(site: Site) -> dict:
|
||||
return {f: getattr(site, f) for f in SITE_FIELDS}
|
||||
|
||||
|
||||
def provision(cfg: Config, site: Site, runner: Runner) -> dict:
|
||||
"""Provision a site end-to-end. Idempotent where the underlying steps are."""
|
||||
deploy_dir = deployment_dir(cfg, site)
|
||||
summary: dict = {"slug": site.slug, "deploy_dir": str(deploy_dir)}
|
||||
|
||||
# 1. Storage
|
||||
zfs.create_web(runner, cfg, site)
|
||||
sftp.set_web_ownership(runner, cfg, site)
|
||||
|
||||
# 2. Database + secrets (only for DB profiles)
|
||||
if site.database:
|
||||
db_password = secrets.generate_password()
|
||||
database.create_db_user(runner, cfg, site, db_password)
|
||||
secrets.write_env(runner, deploy_dir, {secrets.db_env_key(site): db_password})
|
||||
secrets.encrypt_env(runner, cfg, deploy_dir)
|
||||
|
||||
# 3. Render compose + nginx config + persist site.yaml (source of truth)
|
||||
for name, content in render.render(cfg, site).items():
|
||||
runner.write_file(deploy_dir / name, content)
|
||||
runner.write_file(deploy_dir / "site.yaml",
|
||||
yaml.safe_dump(_site_to_dict(site), sort_keys=False))
|
||||
|
||||
# 4. SFTP (per customer; password only set when the account is first created)
|
||||
sftp_password = secrets.generate_password()
|
||||
created = sftp.ensure_customer_account(runner, cfg, site.customer, sftp_password)
|
||||
summary["sftp_user"] = f"sftp_{site.customer}"
|
||||
summary["sftp_password"] = sftp_password if created else None
|
||||
|
||||
# 5. Deploy
|
||||
compose.up(runner, deploy_dir)
|
||||
return summary
|
||||
|
||||
|
||||
def deprovision(cfg: Config, site: Site, runner: Runner, *,
|
||||
purge: bool = False, final_backup: bool = True) -> dict:
|
||||
"""Tear down a site. Data is destroyed only with ``purge=True`` and always
|
||||
after a final backup.
|
||||
"""
|
||||
deploy_dir = deployment_dir(cfg, site)
|
||||
summary: dict = {"slug": site.slug, "purged": purge}
|
||||
|
||||
compose.down(runner, deploy_dir)
|
||||
|
||||
if final_backup:
|
||||
summary["final_backup"] = backup.backup_site(runner, cfg, site)
|
||||
|
||||
if site.database:
|
||||
database.drop_db_user(runner, cfg, site)
|
||||
|
||||
if purge:
|
||||
# Destroy the site dataset (its /web child included) but keep the customer
|
||||
# dataset + SFTP account — other sites may still belong to this customer.
|
||||
zfs.destroy(runner, f"{cfg.pool}/customers/{site.customer}/{site.site}")
|
||||
runner.run(["rm", "-rf", str(deploy_dir)])
|
||||
summary["data_destroyed"] = True
|
||||
else:
|
||||
summary["data_destroyed"] = False
|
||||
summary["note"] = "config/containers removed; data retained. Re-run with purge to delete."
|
||||
return summary
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
"""Render a profile's Jinja2 templates into a site's deployment directory."""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Environment, FileSystemLoader, StrictUndefined
|
||||
|
||||
from . import context
|
||||
from .config import Config, Site
|
||||
from .errors import ValidationError
|
||||
|
||||
|
||||
def _env(profile_dir: Path) -> Environment:
|
||||
return Environment(
|
||||
loader=FileSystemLoader(str(profile_dir)),
|
||||
undefined=StrictUndefined,
|
||||
keep_trailing_newline=True,
|
||||
)
|
||||
|
||||
|
||||
def render(cfg: Config, site: Site) -> dict[str, str]:
|
||||
"""Render all ``*.j2`` files for the site's profile.
|
||||
|
||||
Returns a mapping of output filename (``.j2`` stripped) → rendered content.
|
||||
Pure: does not write to disk.
|
||||
"""
|
||||
profile_dir = cfg.templates_dir / site.profile
|
||||
if not profile_dir.is_dir():
|
||||
raise ValidationError(f"no templates for profile {site.profile!r} at {profile_dir}")
|
||||
ctx = context.build(cfg, site)
|
||||
env = _env(profile_dir)
|
||||
out: dict[str, str] = {}
|
||||
for tpl in sorted(profile_dir.glob("*.j2")):
|
||||
rendered = env.get_template(tpl.name).render(**ctx)
|
||||
out[tpl.name[:-len(".j2")]] = rendered
|
||||
return out
|
||||
|
||||
|
||||
def write(rendered: dict[str, str], out_dir: Path) -> list[Path]:
|
||||
"""Write rendered files into out_dir, returning the paths written."""
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
written = []
|
||||
for name, content in rendered.items():
|
||||
p = out_dir / name
|
||||
p.write_text(content, encoding="utf-8", newline="\n")
|
||||
written.append(p)
|
||||
return written
|
||||
|
|
@ -1,65 +0,0 @@
|
|||
"""Thin command runner. Every host mutation goes through here so we get uniform
|
||||
logging and a real ``--dry-run`` that prints commands instead of running them."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .errors import CommandError
|
||||
|
||||
|
||||
class Runner:
|
||||
def __init__(self, dry_run: bool = False, verbose: bool = True):
|
||||
self.dry_run = dry_run
|
||||
self.verbose = verbose
|
||||
|
||||
def _echo(self, msg: str) -> None:
|
||||
if self.verbose:
|
||||
print(msg, file=sys.stderr)
|
||||
|
||||
def run(self, cmd: list[str], *, input: str | None = None, check: bool = True,
|
||||
capture: bool = False, secret: bool = False) -> str:
|
||||
"""Run a command. In dry-run mode, print and return "".
|
||||
|
||||
``secret=True`` redacts the command in logs (e.g. contains a password).
|
||||
"""
|
||||
shown = "<redacted>" if secret else " ".join(shlex.quote(c) for c in cmd)
|
||||
prefix = "DRY " if self.dry_run else "RUN "
|
||||
self._echo(prefix + shown)
|
||||
if self.dry_run:
|
||||
return ""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd, input=input, text=True, check=check,
|
||||
stdout=subprocess.PIPE if capture else None,
|
||||
stderr=subprocess.PIPE if capture else None,
|
||||
)
|
||||
except FileNotFoundError as e:
|
||||
raise CommandError(f"command not found: {cmd[0]}") from e
|
||||
except subprocess.CalledProcessError as e:
|
||||
detail = (e.stderr or "").strip() if capture else ""
|
||||
raise CommandError(
|
||||
f"command failed ({e.returncode}): {shown}"
|
||||
+ (f"\n{detail}" if detail else "")
|
||||
) from e
|
||||
return (proc.stdout or "") if capture else ""
|
||||
|
||||
def mkdir(self, path: str | Path) -> None:
|
||||
self._echo(("DRY " if self.dry_run else "RUN ") + f"mkdir -p {path}")
|
||||
if not self.dry_run:
|
||||
Path(path).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def write_file(self, path: str | Path, content: str, *, mode: int | None = None,
|
||||
secret: bool = False) -> None:
|
||||
note = f"write {path}" + (" <secret>" if secret else "")
|
||||
self._echo(("DRY " if self.dry_run else "RUN ") + note)
|
||||
if self.dry_run:
|
||||
return
|
||||
p = Path(path)
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(content, encoding="utf-8", newline="\n")
|
||||
if mode is not None:
|
||||
os.chmod(p, mode)
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
"""Per-site secret generation + SOPS/age encryption of the .env file."""
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets as _secrets
|
||||
from pathlib import Path
|
||||
|
||||
from .config import Config, Site
|
||||
from .runner import Runner
|
||||
|
||||
|
||||
def generate_password(nbytes: int = 24) -> str:
|
||||
"""URL-safe token (~32 chars); no shell-special characters."""
|
||||
return _secrets.token_urlsafe(nbytes)
|
||||
|
||||
|
||||
def db_env_key(site: Site) -> str:
|
||||
return "WORDPRESS_DB_PASSWORD" if site.profile == "wordpress" else "DB_PASSWORD"
|
||||
|
||||
|
||||
def write_env(runner: Runner, out_dir: Path, mapping: dict[str, str]) -> None:
|
||||
body = "".join(f"{k}={v}\n" for k, v in mapping.items())
|
||||
runner.write_file(out_dir / ".env", body, mode=0o600, secret=True)
|
||||
|
||||
|
||||
def encrypt_env(runner: Runner, cfg: Config, out_dir: Path) -> None:
|
||||
"""Encrypt .env → secrets.enc.yaml with SOPS/age (the committed artifact)."""
|
||||
if not cfg.sops_age_recipient:
|
||||
runner._echo("WARN no sops_age_recipient configured; skipping encryption "
|
||||
"(.env is git-ignored but secrets.enc.yaml will not be created)")
|
||||
return
|
||||
if runner.dry_run:
|
||||
runner._echo(f"DRY sops --encrypt {out_dir/'.env'} > {out_dir/'secrets.enc.yaml'}")
|
||||
return
|
||||
out = runner.run(
|
||||
["sops", "--encrypt", "--age", cfg.sops_age_recipient,
|
||||
"--input-type", "dotenv", "--output-type", "yaml", str(out_dir / ".env")],
|
||||
capture=True,
|
||||
)
|
||||
runner.write_file(out_dir / "secrets.enc.yaml", out)
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
"""Per-customer SFTP account + web-root ownership.
|
||||
|
||||
One chrooted SFTP user per customer (``sftp_<customer>``), chrooted to the
|
||||
customer directory so they see each of their sites' web roots as sub-folders.
|
||||
|
||||
Ownership scheme (see site-templates/README.md "provisioning note"):
|
||||
- The customer dir (chroot target) stays root-owned 755 — an sshd requirement.
|
||||
- Each site's web dir is chowned to the fpm uid/gid and set 2775 (setgid) so
|
||||
files created there stay group-owned and group-writable.
|
||||
- The SFTP user's primary group is the fpm group, so php-fpm and SFTP share
|
||||
group access. SFTP writes with umask 0002 (set via ForceCommand) to keep new
|
||||
files group-writable. (php-fpm umask hardening is a documented follow-up.)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from . import context, naming
|
||||
from .config import Config, Site
|
||||
from .runner import Runner
|
||||
|
||||
SSHD_DROPIN = "/etc/ssh/sshd_config.d/sftp-{customer}.conf"
|
||||
|
||||
|
||||
def _user_exists(runner: Runner, user: str) -> bool:
|
||||
if runner.dry_run:
|
||||
return False
|
||||
out = runner.run(["id", "-u", user], check=False, capture=True)
|
||||
return out.strip().isdigit()
|
||||
|
||||
|
||||
def set_web_ownership(runner: Runner, cfg: Config, site: Site) -> None:
|
||||
web = context.webroot(cfg, site)
|
||||
runner.run(["chown", "-R", f"{cfg.fpm_uid}:{cfg.fpm_gid}", web])
|
||||
runner.run(["chmod", "2775", web])
|
||||
|
||||
|
||||
def ensure_customer_account(runner: Runner, cfg: Config, customer: str,
|
||||
password: str) -> bool:
|
||||
"""Ensure the customer's SFTP account + chroot config exist. Idempotent:
|
||||
the password is set only when the account is first created (so provisioning a
|
||||
second site for an existing customer doesn't reset their password).
|
||||
|
||||
Returns True if the account was newly created.
|
||||
"""
|
||||
user = naming.sftp_user(customer)
|
||||
home = f"{cfg.customers_root}/{customer}"
|
||||
created = not _user_exists(runner, user)
|
||||
if created:
|
||||
runner.run([
|
||||
"useradd", "-M", "-N",
|
||||
"-g", str(cfg.fpm_gid), # primary group = fpm group
|
||||
"-G", "sftponly", # supplementary: enables the Match block
|
||||
"-s", "/usr/sbin/nologin",
|
||||
"-d", home,
|
||||
user,
|
||||
])
|
||||
runner.run(["chpasswd"], input=f"{user}:{password}\n", secret=True)
|
||||
|
||||
dropin = SSHD_DROPIN.format(customer=customer)
|
||||
content = (
|
||||
f"Match User {user}\n"
|
||||
f" ChrootDirectory {home}\n"
|
||||
f" ForceCommand internal-sftp -u 0002\n"
|
||||
)
|
||||
runner.write_file(dropin, content, mode=0o644)
|
||||
runner.run(["systemctl", "reload", "ssh"])
|
||||
return created
|
||||
|
||||
|
||||
def remove_customer_account(runner: Runner, cfg: Config, customer: str) -> None:
|
||||
user = naming.sftp_user(customer)
|
||||
runner.run(["userdel", user], check=False)
|
||||
runner.run(["rm", "-f", SSHD_DROPIN.format(customer=customer)])
|
||||
runner.run(["systemctl", "reload", "ssh"])
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
"""ZFS operations for a site's web dataset."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from .config import Config, Site
|
||||
from .runner import Runner
|
||||
|
||||
|
||||
def web_dataset(cfg: Config, site: Site) -> str:
|
||||
return f"{cfg.pool}/customers/{site.customer}/{site.site}/web"
|
||||
|
||||
|
||||
def customer_dataset(cfg: Config, customer: str) -> str:
|
||||
return f"{cfg.pool}/customers/{customer}"
|
||||
|
||||
|
||||
def exists(runner: Runner, dataset: str) -> bool:
|
||||
if runner.dry_run:
|
||||
return False
|
||||
out = runner.run(["zfs", "list", "-H", "-o", "name", dataset],
|
||||
check=False, capture=True)
|
||||
return dataset in out.split()
|
||||
|
||||
|
||||
def create_web(runner: Runner, cfg: Config, site: Site) -> str:
|
||||
ds = web_dataset(cfg, site)
|
||||
if not exists(runner, ds):
|
||||
# -p creates the customer/site parent datasets too (root-owned, 755 —
|
||||
# suitable as the SFTP chroot). The web child is chowned by the sftp step.
|
||||
runner.run(["zfs", "create", "-p", ds])
|
||||
return ds
|
||||
|
||||
|
||||
def snapshot(runner: Runner, dataset: str, tag: str | None = None) -> str:
|
||||
tag = tag or "auto-" + datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
name = f"{dataset}@{tag}"
|
||||
runner.run(["zfs", "snapshot", name])
|
||||
return name
|
||||
|
||||
|
||||
def destroy(runner: Runner, dataset: str, *, recursive: bool = True) -> None:
|
||||
cmd = ["zfs", "destroy"]
|
||||
if recursive:
|
||||
cmd.append("-r")
|
||||
cmd.append(dataset)
|
||||
runner.run(cmd)
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
import pytest
|
||||
|
||||
from heleos.config import Site
|
||||
from heleos.errors import ValidationError
|
||||
|
||||
|
||||
def test_wordpress_forces_database():
|
||||
s = Site.from_dict({"customer": "acme", "site": "blog", "profile": "wordpress",
|
||||
"domains": ["blog.acme.com"], "database": False})
|
||||
assert s.database is True
|
||||
|
||||
|
||||
def test_redirect_requires_target():
|
||||
with pytest.raises(ValidationError):
|
||||
Site.from_dict({"customer": "acme", "site": "old", "profile": "redirect",
|
||||
"domains": ["old.acme.com"]})
|
||||
|
||||
|
||||
def test_redirect_needs_source_domain():
|
||||
with pytest.raises(ValidationError):
|
||||
Site.from_dict({"customer": "acme", "site": "old", "profile": "redirect",
|
||||
"domains": [], "redirect_to": "https://acme.com"})
|
||||
|
||||
|
||||
def test_static_cannot_have_database():
|
||||
with pytest.raises(ValidationError):
|
||||
Site.from_dict({"customer": "acme", "site": "site", "profile": "static",
|
||||
"domains": ["acme.com"], "database": True})
|
||||
|
||||
|
||||
def test_unknown_profile_rejected():
|
||||
with pytest.raises(ValidationError):
|
||||
Site.from_dict({"customer": "acme", "site": "x", "profile": "django",
|
||||
"domains": ["acme.com"]})
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
import pytest
|
||||
|
||||
from heleos import naming
|
||||
from heleos.errors import ValidationError
|
||||
|
||||
|
||||
def test_slug_and_db_identifiers():
|
||||
assert naming.slug("acme", "shop") == "acme-shop"
|
||||
assert naming.slug_underscored("acme", "my-shop") == "acme_my_shop"
|
||||
assert naming.database_name("acme", "shop") == "db_acme_shop"
|
||||
assert naming.database_user("acme", "shop") == "u_acme_shop"
|
||||
assert naming.sftp_user("acme") == "sftp_acme"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["", "A", "1abc", "ab_cd", "-abc", "x" * 40])
|
||||
def test_invalid_customer(bad):
|
||||
with pytest.raises(ValidationError):
|
||||
naming.validate_customer(bad)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", ["proxy", "platform", "mariadb", "deployments"])
|
||||
def test_reserved_names_rejected(name):
|
||||
with pytest.raises(ValidationError):
|
||||
naming.validate_customer(name)
|
||||
with pytest.raises(ValidationError):
|
||||
naming.validate_site(name)
|
||||
|
||||
|
||||
def test_valid_ids():
|
||||
assert naming.validate_customer("acme") == "acme"
|
||||
assert naming.validate_site("blog-2") == "blog-2"
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from heleos import context, render
|
||||
from heleos.config import Config, Site
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
TEMPLATES = REPO_ROOT / "site-templates" / "profiles"
|
||||
|
||||
|
||||
def _cfg():
|
||||
return Config(templates_dir=TEMPLATES, registry="reg.example.com/heleos")
|
||||
|
||||
|
||||
def test_context_resolves_images_and_paths():
|
||||
cfg = _cfg()
|
||||
s = Site.from_dict({"customer": "acme", "site": "shop", "profile": "custom-php",
|
||||
"domains": ["shop.acme.com"], "database": True})
|
||||
ctx = context.build(cfg, s)
|
||||
assert ctx["slug"] == "acme-shop"
|
||||
assert ctx["slug_underscored"] == "acme_shop"
|
||||
assert ctx["webroot"] == "/tank/customers/acme/shop/web"
|
||||
assert ctx["php_image"] == "reg.example.com/heleos/php-fpm:8.3"
|
||||
assert ctx["nginx_image"] == "reg.example.com/heleos/nginx:latest"
|
||||
|
||||
|
||||
def test_custom_php_renders_valid_compose_with_db():
|
||||
cfg = _cfg()
|
||||
s = Site.from_dict({"customer": "acme", "site": "shop", "profile": "custom-php",
|
||||
"domains": ["shop.acme.com", "www.shop.acme.com"], "database": True})
|
||||
out = render.render(cfg, s)
|
||||
doc = yaml.safe_load(out["docker-compose.yml"])
|
||||
assert doc["name"] == "acme-shop"
|
||||
assert set(doc["services"]) == {"nginx", "fpm"}
|
||||
# fpm reaches the DB, so it is on the platform network.
|
||||
assert "platform" in doc["services"]["fpm"]["networks"]
|
||||
labels = doc["services"]["nginx"]["labels"]
|
||||
rule = [l for l in labels if ".rule=" in l][0]
|
||||
assert "Host(`shop.acme.com`) || Host(`www.shop.acme.com`)" in rule
|
||||
|
||||
|
||||
def test_custom_php_without_db_has_no_platform_network():
|
||||
cfg = _cfg()
|
||||
s = Site.from_dict({"customer": "acme", "site": "api", "profile": "custom-php",
|
||||
"domains": ["api.acme.com"], "database": False})
|
||||
doc = yaml.safe_load(render.render(cfg, s)["docker-compose.yml"])
|
||||
assert doc["services"]["fpm"]["networks"] == ["site"]
|
||||
assert "platform" not in doc.get("networks", {})
|
||||
|
||||
|
||||
def test_static_profile_is_nginx_only():
|
||||
cfg = _cfg()
|
||||
s = Site.from_dict({"customer": "acme", "site": "www", "profile": "static",
|
||||
"domains": ["acme.com"]})
|
||||
doc = yaml.safe_load(render.render(cfg, s)["docker-compose.yml"])
|
||||
assert list(doc["services"]) == ["nginx"]
|
||||
|
||||
|
||||
def test_redirect_profile_renders_redirect_conf():
|
||||
cfg = _cfg()
|
||||
s = Site.from_dict({"customer": "acme", "site": "old", "profile": "redirect",
|
||||
"domains": ["old.acme.com"], "redirect_to": "https://acme.com",
|
||||
"redirect_code": 302})
|
||||
out = render.render(cfg, s)
|
||||
assert "return 302 https://acme.com$request_uri;" in out["nginx-redirect.conf"]
|
||||
|
|
@ -32,21 +32,6 @@ config) is backed up with the same ZFS mechanism from `tank/platform/*`.
|
|||
- **DB dumps:** rsync to the same or another offsite location (delta transfer).
|
||||
- Both offsite copies are the recovery source if the primary host is lost.
|
||||
|
||||
## 2b. Implementation (Phase 5)
|
||||
|
||||
Automated by the `backup` Ansible role (`platform-infra/ansible/roles/backup`):
|
||||
|
||||
- **Files:** [sanoid](https://github.com/jimsalterjrs/sanoid) takes and prunes
|
||||
snapshots per policy (`sanoid.conf`, driven by `sanoid_datasets`), on the
|
||||
packaged `sanoid.timer`. **syncoid** replicates offsite
|
||||
(`heleos-zfs-offsite`), enabled only when `zfs_offsite_target` is set.
|
||||
- **DB:** `heleos-db-backup` (systemd timer, nightly) walks the deployments dir,
|
||||
dumps each DB-backed site via `docker exec … mariadb-dump` into
|
||||
`db-backups/<customer>/<site>/{daily,weekly,monthly}` with rotation, then
|
||||
`heleos-db-offsite` rsyncs offsite (enabled only when `db_offsite_target` set).
|
||||
- Toggle streams with `backup_snapshots_enabled` / `db_backup_enabled` and the
|
||||
offsite target vars in `group_vars/all.yml`.
|
||||
|
||||
## 3. Restore — single site (the critical drill)
|
||||
|
||||
Two coordinated steps:
|
||||
|
|
|
|||
|
|
@ -2,22 +2,14 @@
|
|||
|
||||
Idempotent host configuration for the heleos platform, targeting **Ubuntu 24.04
|
||||
LTS**. Roles: base packages → ZFS pool/datasets → Docker (data-root on ZFS) →
|
||||
nftables firewall + container egress filter → SSH hardening → backup/DR
|
||||
(sanoid snapshots, per-DB dumps, offsite).
|
||||
nftables firewall + container egress filter → SSH hardening.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. An Ubuntu 24.04 VM you can SSH into as a **sudo-capable user**.
|
||||
2. Storage for the ZFS pool — pick one via `zfs_pool_mode`:
|
||||
- **`file`** (default): a loopback disk image, **no spare disk needed** —
|
||||
ideal for a cost-optimized VM. Real ZFS, backed by a file on the OS disk.
|
||||
- **`single` / `mirror`**: a dedicated spare disk (or two, mirrored) for a
|
||||
real host.
|
||||
3. Ansible on your workstation, installed **one way only** — prefer
|
||||
`pipx install --include-deps ansible` (isolated). Mixing apt's `ansible`
|
||||
with a pip `ansible` causes version-skew errors such as
|
||||
`No module named 'ansible.module_utils.six.moves'`; if you hit that, remove
|
||||
the duplicates and reinstall via pipx.
|
||||
2. A **dedicated second virtual disk** attached to the VM for the ZFS pool
|
||||
(e.g. `/dev/sdb` or `/dev/vdb`) — separate from the OS disk.
|
||||
3. Ansible installed on your workstation (`pipx install ansible` or apt).
|
||||
|
||||
## Configure
|
||||
|
||||
|
|
@ -26,16 +18,14 @@ cd platform-infra/ansible
|
|||
ansible-galaxy collection install -r requirements.yml
|
||||
|
||||
cp inventory/hosts.yml.example inventory/hosts.yml # edit host/user (git-ignored)
|
||||
$EDITOR group_vars/all.yml # set zfs_pool_mode, keys, etc.
|
||||
$EDITOR group_vars/all.yml # set zfs_pool_disk, keys, etc.
|
||||
```
|
||||
|
||||
Key variables in `group_vars/all.yml`:
|
||||
|
||||
| Variable | Meaning |
|
||||
|----------|---------|
|
||||
| `zfs_pool_mode` | `file` (loopback image, default), `single`, or `mirror`. |
|
||||
| `zfs_pool_file_path` / `zfs_pool_file_size` | File-mode image location + size (sparse). |
|
||||
| `zfs_pool_disks` | Disk(s) for `single`/`mirror` mode. **Contents destroyed.** |
|
||||
| `zfs_pool_disk` | The dedicated disk for the pool. **Its contents will be destroyed.** |
|
||||
| `zfs_pool_force` | Must be `true` to create a pool on a non-empty disk (safety gate). |
|
||||
| `admin_authorized_keys` | Public keys for the admin — required before disabling passwords. |
|
||||
| `ssh_disable_password_auth` | Leave `false` until key login is verified, then flip to `true`. |
|
||||
|
|
@ -44,49 +34,21 @@ Key variables in `group_vars/all.yml`:
|
|||
## Run
|
||||
|
||||
```bash
|
||||
ansible-playbook site.yml --syntax-check # no-host pre-flight
|
||||
ansible-playbook site.yml -K # apply (-K prompts for the sudo/become
|
||||
# password; omit only if the user has
|
||||
# passwordless sudo on the VM)
|
||||
ansible-playbook site.yml --check # dry run (note: first run can't fully
|
||||
# check tasks that depend on ZFS/Docker
|
||||
# not yet present)
|
||||
ansible-playbook site.yml # apply
|
||||
```
|
||||
|
||||
Everything runs via `become` (root), so `-K` is required unless the target user
|
||||
has passwordless sudo (`/etc/sudoers.d/… NOPASSWD:ALL`).
|
||||
|
||||
Run a single layer with tags: `--tags zfs`, `--tags docker`, `--tags firewall`,
|
||||
`--tags ssh`, `--tags base`.
|
||||
|
||||
## Running from Windows / WSL
|
||||
|
||||
Files on the `/mnt/c` drive mount are world-writable (mode 0777), so Ansible
|
||||
**ignores `ansible.cfg`** there (a security measure) — which then loses the
|
||||
inventory path and you get "no hosts matched". Two ways around it:
|
||||
|
||||
- **Preferred:** copy the repo into your WSL home and run from there, where
|
||||
permissions are normal (also much faster):
|
||||
```bash
|
||||
cp -r /mnt/c/claude/heleosv2 ~/heleosv2 && cd ~/heleosv2/platform-infra/ansible
|
||||
```
|
||||
- **Or** force the config explicitly (bypasses the world-writable check):
|
||||
```bash
|
||||
export ANSIBLE_CONFIG=$(pwd)/ansible.cfg
|
||||
```
|
||||
|
||||
Either way, create the inventory first (`cp inventory/hosts.yml.example
|
||||
inventory/hosts.yml` and edit it). WSL itself is not a valid target host (no
|
||||
spare disk for the ZFS pool); point the inventory at your Ubuntu VM.
|
||||
|
||||
Pre-flight without a host: `ansible-playbook --syntax-check site.yml`
|
||||
(`--check` is not meaningful on the first run — see Safety notes).
|
||||
|
||||
## Safety notes
|
||||
|
||||
- **ZFS disk modes are destructive:** in `single`/`mirror` mode the play refuses
|
||||
to create a pool on a disk that already has a filesystem/partition unless
|
||||
`zfs_pool_force: true`. Double-check `zfs_pool_disks` point at empty spare
|
||||
disks, not the OS disk; prefer stable `/dev/disk/by-id/...` paths in
|
||||
production. **`file` mode (default) wipes nothing** — it creates a loopback
|
||||
image at `zfs_pool_file_path`.
|
||||
- **ZFS is destructive:** the play refuses to create a pool on a disk that
|
||||
already has a filesystem/partition unless `zfs_pool_force: true`. Double-check
|
||||
`zfs_pool_disk` points at the empty spare disk, not the OS disk. For production
|
||||
prefer a stable `/dev/disk/by-id/...` path over `/dev/sdb`.
|
||||
- **SSH lock-out:** the play asserts that `admin_authorized_keys` is non-empty
|
||||
before it will disable password authentication. Verify you can log in with your
|
||||
key **before** setting `ssh_disable_password_auth: true`.
|
||||
|
|
@ -105,9 +67,6 @@ Pre-flight without a host: `ansible-playbook --syntax-check site.yml`
|
|||
ICMP); outbound SMTP blocked from containers.
|
||||
- Hardened SSH (key-first, root prohibit-password) and the `sftponly` group that
|
||||
per-customer SFTP accounts will join in Phase 4.
|
||||
- Backup/DR (Phase 5): sanoid snapshot policy + timer, nightly per-database dumps
|
||||
with rotation, and optional offsite `zfs send` (syncoid) / rsync — enable
|
||||
offsite by setting `zfs_offsite_target` / `db_offsite_target`.
|
||||
|
||||
## Verify after running
|
||||
|
||||
|
|
|
|||
|
|
@ -7,24 +7,14 @@
|
|||
host_timezone: "Europe/Brussels"
|
||||
|
||||
# --- ZFS --------------------------------------------------------------------
|
||||
# The pool can be backed three ways:
|
||||
# file - a loopback disk image (NO spare disk needed). Real ZFS features,
|
||||
# backed by a file on the OS disk. Ideal for a cheap test VM.
|
||||
# single - one whole spare disk/partition.
|
||||
# mirror - two disks/partitions (redundant); use for production.
|
||||
# Disk modes are DESTRUCTIVE to every listed device and refuse a non-empty
|
||||
# device unless zfs_pool_force: true. File mode wipes nothing.
|
||||
# The pool is created on a DEDICATED second virtual disk. Attach a disk to the
|
||||
# VM first (e.g. /dev/sdb or /dev/vdb) and set it here.
|
||||
#
|
||||
# ⚠️ zpool create is DESTRUCTIVE to the target disk. The playbook refuses to
|
||||
# touch a disk that already contains a filesystem/partition unless you set
|
||||
# zfs_pool_force: true. For production prefer a stable /dev/disk/by-id/... path.
|
||||
zfs_pool_name: tank
|
||||
zfs_pool_mode: file # file | single | mirror
|
||||
|
||||
# file mode:
|
||||
zfs_pool_file_path: /var/lib/heleos/tank.img
|
||||
zfs_pool_file_size: "30G" # sparse; grows as data is written
|
||||
|
||||
# single / mirror modes (prefer stable /dev/disk/by-id/... paths in production):
|
||||
zfs_pool_disks:
|
||||
- /dev/sdb
|
||||
|
||||
zfs_pool_disk: /dev/sdb
|
||||
zfs_pool_force: false
|
||||
zfs_compression: lz4 # lz4 (fast) or zstd (denser)
|
||||
|
||||
|
|
@ -42,10 +32,6 @@ zfs_child_datasets:
|
|||
- { path: "customers" }
|
||||
|
||||
# --- Docker -----------------------------------------------------------------
|
||||
# APT codename for Docker's repo. Defaults to the VM's release. Docker only
|
||||
# publishes repos for LTS + recent codenames — on a non-LTS Ubuntu (e.g.
|
||||
# oracular/plucky) set this to the nearest LTS, e.g. "noble".
|
||||
docker_apt_codename: "noble"
|
||||
# Native ZFS storage driver keeps image layers as ZFS datasets under the pool
|
||||
# (data-root sits on tank/platform/docker). Switch to overlay2 only if you have
|
||||
# a specific reason.
|
||||
|
|
@ -66,32 +52,5 @@ smtp_relay_host: "" # optional: allow SMTP only to this hos
|
|||
# ⚠️ If ssh_disable_password_auth is true you MUST provide admin_authorized_keys
|
||||
# for admin_user, or you will lock yourself out. The playbook asserts this.
|
||||
admin_user: "{{ ansible_user }}"
|
||||
admin_authorized_keys: # list of public key strings
|
||||
- "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHUeM0s5kNSRLQOjBjAVONtRJAnSwcFvvXWLwAXAec4 bart@kaluna2023"
|
||||
admin_authorized_keys: [] # list of public key strings
|
||||
ssh_disable_password_auth: false # flip to true once key login is verified
|
||||
|
||||
# --- Backups / DR (Phase 5) -------------------------------------------------
|
||||
mariadb_container: mariadb
|
||||
mariadb_env_file: /opt/heleos/platform-infra/stacks/mariadb/.env
|
||||
deployments_dir: /opt/heleos/deployments
|
||||
|
||||
# ZFS snapshots via sanoid (files stream). Retention is per policy below.
|
||||
backup_snapshots_enabled: true
|
||||
sanoid_datasets:
|
||||
- { name: "{{ zfs_pool_name }}/customers", recursive: true, hourly: 36, daily: 30, weekly: 8, monthly: 6 }
|
||||
- { name: "{{ zfs_pool_name }}/platform", recursive: true, hourly: 0, daily: 14, weekly: 4, monthly: 3 }
|
||||
|
||||
# Per-database dumps (DB stream), automysqlbackup-style rotation via docker exec.
|
||||
db_backup_enabled: true
|
||||
db_backup_dir: "/{{ zfs_pool_name }}/platform/db-backups"
|
||||
db_backup_oncalendar: "*-*-* 01:30:00"
|
||||
db_backup_keep_daily: 14
|
||||
db_backup_keep_weekly: 8
|
||||
db_backup_keep_monthly: 6
|
||||
|
||||
# Offsite. Leave the targets empty to disable that stream.
|
||||
zfs_offsite_target: "" # e.g. "user@backup-host:backup/heleos"
|
||||
zfs_offsite_datasets: ["{{ zfs_pool_name }}/customers", "{{ zfs_pool_name }}/platform"]
|
||||
zfs_offsite_oncalendar: "*-*-* 03:00:00"
|
||||
db_offsite_target: "" # e.g. "user@backup-host:/srv/heleos/db-backups"
|
||||
db_offsite_oncalendar: "*-*-* 03:30:00"
|
||||
|
|
|
|||
|
|
@ -1,4 +0,0 @@
|
|||
---
|
||||
- name: Reload systemd
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
|
|
@ -1,205 +0,0 @@
|
|||
---
|
||||
# ── ZFS snapshots (files stream) via sanoid ─────────────────────────────────
|
||||
- name: Install sanoid (snapshots + syncoid replication)
|
||||
ansible.builtin.apt:
|
||||
name: sanoid
|
||||
state: present
|
||||
when: backup_snapshots_enabled
|
||||
|
||||
- name: Ensure the sanoid config directory exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/sanoid
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: backup_snapshots_enabled
|
||||
|
||||
- name: Locate the packaged sanoid.defaults.conf
|
||||
ansible.builtin.shell: "dpkg -L sanoid | grep -m1 sanoid.defaults.conf"
|
||||
register: sanoid_defaults_src
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
when: backup_snapshots_enabled
|
||||
|
||||
- name: Install sanoid.defaults.conf (sanoid needs it next to sanoid.conf)
|
||||
ansible.builtin.copy:
|
||||
remote_src: true
|
||||
src: "{{ sanoid_defaults_src.stdout }}"
|
||||
dest: /etc/sanoid/sanoid.defaults.conf
|
||||
force: false
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when:
|
||||
- backup_snapshots_enabled
|
||||
- sanoid_defaults_src.stdout | default('') | length > 0
|
||||
- sanoid_defaults_src.stdout != '/etc/sanoid/sanoid.defaults.conf'
|
||||
|
||||
- name: Configure sanoid snapshot policy
|
||||
ansible.builtin.template:
|
||||
src: sanoid.conf.j2
|
||||
dest: /etc/sanoid/sanoid.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when: backup_snapshots_enabled
|
||||
|
||||
- name: Enable the sanoid timer
|
||||
ansible.builtin.systemd:
|
||||
name: sanoid.timer
|
||||
enabled: true
|
||||
state: started
|
||||
when: backup_snapshots_enabled
|
||||
|
||||
# ── Per-database dumps (DB stream) ──────────────────────────────────────────
|
||||
- name: Install the DB backup script
|
||||
ansible.builtin.template:
|
||||
src: heleos-db-backup.sh.j2
|
||||
dest: /usr/local/sbin/heleos-db-backup.sh
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: db_backup_enabled
|
||||
|
||||
- name: Install the DB backup service + timer
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/systemd/system/{{ item.name }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: "{{ item.content }}"
|
||||
loop:
|
||||
- name: heleos-db-backup.service
|
||||
content: |
|
||||
[Unit]
|
||||
Description=heleos per-database dumps (rotated)
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/heleos-db-backup.sh
|
||||
- name: heleos-db-backup.timer
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Run heleos DB backups on a schedule
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ db_backup_oncalendar }}
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when: db_backup_enabled
|
||||
notify: Reload systemd
|
||||
|
||||
- name: Enable the DB backup timer
|
||||
ansible.builtin.systemd:
|
||||
name: heleos-db-backup.timer
|
||||
enabled: true
|
||||
daemon_reload: true
|
||||
state: started
|
||||
when: db_backup_enabled
|
||||
|
||||
# ── Offsite: files (syncoid / zfs send) ─────────────────────────────────────
|
||||
- name: Install the ZFS offsite script
|
||||
ansible.builtin.template:
|
||||
src: heleos-zfs-offsite.sh.j2
|
||||
dest: /usr/local/sbin/heleos-zfs-offsite.sh
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: zfs_offsite_target | length > 0
|
||||
|
||||
- name: Install the ZFS offsite service + timer
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/systemd/system/{{ item.name }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: "{{ item.content }}"
|
||||
loop:
|
||||
- name: heleos-zfs-offsite.service
|
||||
content: |
|
||||
[Unit]
|
||||
Description=heleos offsite ZFS replication (syncoid)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/heleos-zfs-offsite.sh
|
||||
- name: heleos-zfs-offsite.timer
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Run heleos offsite ZFS replication on a schedule
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ zfs_offsite_oncalendar }}
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when: zfs_offsite_target | length > 0
|
||||
notify: Reload systemd
|
||||
|
||||
- name: Enable the ZFS offsite timer
|
||||
ansible.builtin.systemd:
|
||||
name: heleos-zfs-offsite.timer
|
||||
enabled: true
|
||||
daemon_reload: true
|
||||
state: started
|
||||
when: zfs_offsite_target | length > 0
|
||||
|
||||
# ── Offsite: DB dumps (rsync) ───────────────────────────────────────────────
|
||||
- name: Install the DB offsite script
|
||||
ansible.builtin.template:
|
||||
src: heleos-db-offsite.sh.j2
|
||||
dest: /usr/local/sbin/heleos-db-offsite.sh
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: db_offsite_target | length > 0
|
||||
|
||||
- name: Install the DB offsite service + timer
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/systemd/system/{{ item.name }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: "{{ item.content }}"
|
||||
loop:
|
||||
- name: heleos-db-offsite.service
|
||||
content: |
|
||||
[Unit]
|
||||
Description=heleos offsite DB dump sync (rsync)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/heleos-db-offsite.sh
|
||||
- name: heleos-db-offsite.timer
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Run heleos offsite DB sync on a schedule
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ db_offsite_oncalendar }}
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when: db_offsite_target | length > 0
|
||||
notify: Reload systemd
|
||||
|
||||
- name: Enable the DB offsite timer
|
||||
ansible.builtin.systemd:
|
||||
name: heleos-db-offsite.timer
|
||||
enabled: true
|
||||
daemon_reload: true
|
||||
state: started
|
||||
when: db_offsite_target | length > 0
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# heleos per-database dumps — automysqlbackup-style rotation, adapted for the
|
||||
# containerized shared MariaDB. Managed by Ansible. Iterates the deployments dir
|
||||
# (source of truth) and dumps each DB-backed site into
|
||||
# db-backups/<customer>/<site>/{daily,weekly,monthly}.
|
||||
set -euo pipefail
|
||||
|
||||
CONTAINER="{{ mariadb_container }}"
|
||||
BASE="{{ db_backup_dir }}"
|
||||
DEPLOYMENTS="{{ deployments_dir }}"
|
||||
KEEP_DAILY={{ db_backup_keep_daily }}
|
||||
KEEP_WEEKLY={{ db_backup_keep_weekly }}
|
||||
KEEP_MONTHLY={{ db_backup_keep_monthly }}
|
||||
|
||||
# Root password: env var wins, else read from the mariadb stack .env.
|
||||
ROOTPW="${MARIADB_ROOT_PASSWORD:-}"
|
||||
if [ -z "$ROOTPW" ] && [ -f "{{ mariadb_env_file }}" ]; then
|
||||
ROOTPW="$(grep -E '^MARIADB_ROOT_PASSWORD=' "{{ mariadb_env_file }}" | cut -d= -f2-)"
|
||||
fi
|
||||
if [ -z "$ROOTPW" ]; then
|
||||
echo "no MariaDB root password (env or {{ mariadb_env_file }})" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dow="$(date +%u)" # 7 = Sunday
|
||||
dom="$(date +%d)" # 01 = first of month
|
||||
ts="$(date +%Y%m%d-%H%M%S)"
|
||||
|
||||
shopt -s nullglob
|
||||
for meta in "$DEPLOYMENTS"/*/*/site.yaml; do
|
||||
d="$(dirname "$meta")"
|
||||
site="$(basename "$d")"
|
||||
customer="$(basename "$(dirname "$d")")"
|
||||
grep -qiE '^database:[[:space:]]*true' "$meta" || continue
|
||||
|
||||
db="db_${customer//-/_}_${site//-/_}"
|
||||
dest="$BASE/$customer/$site"
|
||||
mkdir -p "$dest/daily" "$dest/weekly" "$dest/monthly"
|
||||
|
||||
out="$dest/daily/${db}-${ts}.sql.gz"
|
||||
# MYSQL_PWD keeps the password out of the container's process list.
|
||||
docker exec -e MYSQL_PWD="$ROOTPW" "$CONTAINER" \
|
||||
mariadb-dump --single-transaction --databases "$db" -uroot | gzip > "$out"
|
||||
|
||||
[ "$dow" = "7" ] && cp -f "$out" "$dest/weekly/"
|
||||
[ "$dom" = "01" ] && cp -f "$out" "$dest/monthly/"
|
||||
|
||||
find "$dest/daily" -name '*.sql.gz' -type f -mtime +"$KEEP_DAILY" -delete
|
||||
find "$dest/weekly" -name '*.sql.gz' -type f -mtime +"$(( KEEP_WEEKLY * 7 ))" -delete
|
||||
find "$dest/monthly" -name '*.sql.gz' -type f -mtime +"$(( KEEP_MONTHLY * 31 ))" -delete
|
||||
done
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# heleos offsite DB dump sync via rsync. Managed by Ansible. Only installed/
|
||||
# enabled when db_offsite_target is set. Dumps are already rotated locally.
|
||||
set -euo pipefail
|
||||
|
||||
rsync -a --delete "{{ db_backup_dir }}/" "{{ db_offsite_target }}/"
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# heleos offsite ZFS replication via syncoid (incremental zfs send). Managed by
|
||||
# Ansible. Only installed/enabled when zfs_offsite_target is set.
|
||||
set -euo pipefail
|
||||
|
||||
TARGET="{{ zfs_offsite_target }}"
|
||||
|
||||
for ds in {{ zfs_offsite_datasets | join(' ') }}; do
|
||||
# DEST mirrors the last path component under the offsite base, e.g.
|
||||
# tank/customers -> <target>/customers
|
||||
dest="${TARGET}/${ds##*/}"
|
||||
echo "==> syncoid ${ds} -> ${dest}"
|
||||
syncoid --recursive --no-sync-snap "${ds}" "${dest}"
|
||||
done
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
# heleos sanoid snapshot policy — managed by Ansible.
|
||||
# Files stream only; DB dumps are handled separately (heleos-db-backup).
|
||||
{% for d in sanoid_datasets %}
|
||||
[{{ d.name }}]
|
||||
use_template = {{ d.name | replace('/', '_') }}
|
||||
recursive = {{ 'yes' if d.recursive else 'no' }}
|
||||
{% endfor %}
|
||||
|
||||
{% for d in sanoid_datasets %}
|
||||
[template_{{ d.name | replace('/', '_') }}]
|
||||
hourly = {{ d.hourly }}
|
||||
daily = {{ d.daily }}
|
||||
weekly = {{ d.weekly }}
|
||||
monthly = {{ d.monthly }}
|
||||
autosnap = yes
|
||||
autoprune = yes
|
||||
{% endfor %}
|
||||
|
|
@ -16,15 +16,9 @@
|
|||
repo: >-
|
||||
deb [arch={{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}
|
||||
signed-by=/etc/apt/keyrings/docker.asc]
|
||||
https://download.docker.com/linux/ubuntu {{ docker_apt_codename }} stable
|
||||
https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable
|
||||
filename: docker
|
||||
state: present
|
||||
register: docker_repo
|
||||
|
||||
- name: Refresh apt cache after adding the Docker repo
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
when: docker_repo is changed
|
||||
|
||||
- name: Ensure the ZFS docker dataset is mounted at the data-root
|
||||
ansible.builtin.command: "zfs get -H -o value mounted {{ zfs_pool_name }}/platform/docker"
|
||||
|
|
@ -53,8 +47,11 @@
|
|||
- name: Install Docker Engine + Compose plugin
|
||||
ansible.builtin.apt:
|
||||
name:
|
||||
- docker.io
|
||||
- docker-compose-v2
|
||||
- docker-ce
|
||||
- docker-ce-cli
|
||||
- containerd.io
|
||||
- docker-buildx-plugin
|
||||
- docker-compose-plugin
|
||||
state: present
|
||||
update_cache: true
|
||||
|
||||
|
|
|
|||
|
|
@ -20,60 +20,32 @@
|
|||
ansible.builtin.set_fact:
|
||||
zpool_exists: "{{ zpool_check.rc == 0 }}"
|
||||
|
||||
# --- File mode: loopback disk image (no spare disk needed) ------------------
|
||||
- name: Ensure the backing-file directory exists (file mode)
|
||||
ansible.builtin.file:
|
||||
path: "{{ zfs_pool_file_path | dirname }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
when: not zpool_exists and zfs_pool_mode == 'file'
|
||||
|
||||
- name: Create the sparse backing file (file mode)
|
||||
ansible.builtin.command: "truncate -s {{ zfs_pool_file_size }} {{ zfs_pool_file_path }}"
|
||||
args:
|
||||
creates: "{{ zfs_pool_file_path }}"
|
||||
when: not zpool_exists and zfs_pool_mode == 'file'
|
||||
|
||||
- name: Create the ZFS pool on the backing file (file mode)
|
||||
ansible.builtin.command: >-
|
||||
zpool create -o ashift=12
|
||||
-O compression={{ zfs_compression }}
|
||||
-O atime=off -O xattr=sa -O acltype=posixacl
|
||||
-O mountpoint=/{{ zfs_pool_name }}
|
||||
{{ zfs_pool_name }} {{ zfs_pool_file_path }}
|
||||
when: not zpool_exists and zfs_pool_mode == 'file'
|
||||
|
||||
# --- Disk modes: single / mirror (guarded & destructive) --------------------
|
||||
- name: Probe the target disks for existing data (disk modes)
|
||||
ansible.builtin.command: "lsblk -nro FSTYPE,MOUNTPOINT,PARTTYPE {{ item }}"
|
||||
register: disk_probes
|
||||
- name: Probe the target disk for existing data
|
||||
ansible.builtin.command: "lsblk -nro FSTYPE,MOUNTPOINT,PARTTYPE {{ zfs_pool_disk }}"
|
||||
register: disk_probe
|
||||
changed_when: false
|
||||
loop: "{{ zfs_pool_disks }}"
|
||||
when: not zpool_exists and zfs_pool_mode in ['single', 'mirror']
|
||||
when: not zpool_exists
|
||||
|
||||
- name: Refuse to create a pool on a non-empty disk unless forced
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (item.stdout | trim | length == 0) or zfs_pool_force
|
||||
- (disk_probe.stdout | trim | length == 0) or zfs_pool_force
|
||||
fail_msg: >-
|
||||
{{ item.item }} appears to already contain data ({{ item.stdout | trim }}).
|
||||
Refusing to create the pool. Verify the device, then set zfs_pool_force=true.
|
||||
loop: "{{ disk_probes.results | default([]) }}"
|
||||
loop_control:
|
||||
label: "{{ item.item | default('') }}"
|
||||
when: not zpool_exists and zfs_pool_mode in ['single', 'mirror']
|
||||
{{ zfs_pool_disk }} appears to already contain data
|
||||
({{ disk_probe.stdout | trim }}). Refusing to create the pool. Verify you
|
||||
picked the right disk, then set zfs_pool_force=true to override.
|
||||
when: not zpool_exists
|
||||
|
||||
- name: Create the ZFS pool on the dedicated disk(s) (disk modes)
|
||||
- name: Create the ZFS pool on the dedicated disk
|
||||
ansible.builtin.command: >-
|
||||
zpool create {{ '-f ' if zfs_pool_force else '' }}-o ashift=12
|
||||
-O compression={{ zfs_compression }}
|
||||
-O atime=off -O xattr=sa -O acltype=posixacl
|
||||
-O atime=off
|
||||
-O xattr=sa
|
||||
-O acltype=posixacl
|
||||
-O mountpoint=/{{ zfs_pool_name }}
|
||||
{{ zfs_pool_name }}
|
||||
{{ 'mirror ' if zfs_pool_mode == 'mirror' else '' }}{{ zfs_pool_disks | join(' ') }}
|
||||
when: not zpool_exists and zfs_pool_mode in ['single', 'mirror']
|
||||
{{ zfs_pool_name }} {{ zfs_pool_disk }}
|
||||
when: not zpool_exists
|
||||
|
||||
# --- Datasets ---------------------------------------------------------------
|
||||
- name: List existing datasets
|
||||
|
|
|
|||
|
|
@ -20,5 +20,3 @@
|
|||
tags: [firewall]
|
||||
- role: ssh_hardening
|
||||
tags: [ssh]
|
||||
- role: backup
|
||||
tags: [backup]
|
||||
|
|
|
|||
|
|
@ -31,11 +31,7 @@ services:
|
|||
- socketproxy
|
||||
|
||||
traefik:
|
||||
# v3.7+ required: Traefik v3.3's Docker provider fails to negotiate the API
|
||||
# version against Docker Engine 29 (API min 1.44) and falls back to 1.24,
|
||||
# which the daemon rejects ("client version 1.24 is too old"). v3.7.x
|
||||
# negotiates correctly through the socket-proxy.
|
||||
image: traefik:v3.7.10
|
||||
image: traefik:v3.3
|
||||
container_name: traefik
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue