From 293898758bed71286ada13b098e840176ef4dc53 Mon Sep 17 00:00:00 2001 From: Steven Welch Date: Tue, 27 Oct 2026 18:11:45 -0600 Subject: [PATCH 1/5] feat: attach retained tier disks and gate k3s on UUID mounts --- AGENTS.md | 19 ++++++++++++++ cloud-init/k3s/cloud_init.cfg | 47 +++++++++++++++++++++++++++++++++++ main.tf | 17 ++++++++++++- 3 files changed, 82 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index a2fa30e..1017201 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,3 +5,22 @@ OpenTofu root for the libvirt/KVM infrastructure that hosts the Make IT Work Clo Use GitHub MCP and PR CI plans as validation authority. Same-repository PRs can produce a credentialed plan; `main` is an environment-gated apply path. Do not run OpenTofu, Makefile, SSH, host-firewall, state, import, taint, repair, or apply operations from this server. `README.md` is terraform-docs generated in replace mode; keep operational context here or in retrievable skills. The shared workflow and canonical runner configuration are owned by `shared-workflows` and `images/tfroot-runner`. Keep SOPS and SSH material encrypted and never expose state, credentials, private keys, or sensitive plans. + +## Retained k3s storage + +`module.k3s.existing_disks` attaches externally owned sparse raw files; this root must not create, resize, import, format, or delete them. Keep `vda` and the managed `vdb` data disk unchanged. The additional `vdc`/`vdd` files live outside `/mnt/nvme/cluster`, so replacing VM-owned images does not remove their backing files. + +| Tier | Hero path | Maximum logical bytes | Guest mount | XFS UUID | +| --- | --- | ---: | --- | --- | +| SSD | `/mnt/ssd/persistent/k3s-pv-ssd.raw` | 1649267441664 (1.5 TiB) | `/srv/k3s/ssd` | `6bb8172e-7b83-4987-8eda-24b75c697860` | +| NVMe | `/mnt/nvme/persistent/k3s-pv-nvme.raw` | 429496729600 (400 GiB) | `/srv/k3s/nvme` | `028fc5ee-f25b-4b28-8619-2fbc78a0a3ae` | + +These are intended identities, not evidence that the images or filesystems exist. Creation, attachment and first-time initialization require separate approval of exact targets and commands. Before creation, verify the host arrays are mounted, budget existing images at maximum growth, preserve host headroom, and refuse an existing destination. Provision without preallocation and keep host access restricted to the established QEMU identity. Never initialize by guessed `/dev/vdX`: verify the configured serial, expected capacity, absence of filesystem/partition signatures and exclusive ownership first. Never force-format a replacement attachment or regenerate these UUIDs during a rebuild. + +Both UUID-based mount units are required by k3s. Missing mounts, wrong filesystem identities or mount removal intentionally prevent or stop the entire single-node k3s service. Cloud-init does not initialize these disks. Template updates prepare future bootstrap; they do not install mount units or service drop-ins in an existing VM. Existing `/dev/vdb` cloud-init formatting behavior is unchanged. + +Before any apply, inspect the CI plan for unexpected boot/data volume changes or VM replacement. A 2026-10-27 metadata observation found live boot capacity 5 GiB versus the declared 20 GiB; do not reconcile that drift incidentally. Obtain a verified recovery copy and an approved execution path outside the affected VM before disruption: the MCP/control workloads run on this same node. Root merge/apply, module publication, manual live setup and replacement-VM recovery remain distinct gates. + +After guest mounts are verified, `kustomize-cluster` owns StorageClass, static local PV and PVC declarations. Do not add them here, change existing claims, or infer data migration from new capacity. Retained local PVs need explicit node affinity and deliberate binding recovery after cluster recreation; `Retain` alone does not recover metadata or move data. Quiesced migration, isolated restore and application checks require separate approval and verification. + +Sparse allocation is not reserved capacity. Monitor host free space, preserve sparsity in copies, and verify guest discard/TRIM reaches host hole punching before claiming reclamation. Never mount a guest-owned filesystem simultaneously on Hero and a VM; fence the old VM before replacement attachment. RAID and retained files do not provide host-loss backup. diff --git a/cloud-init/k3s/cloud_init.cfg b/cloud-init/k3s/cloud_init.cfg index fd27efe..a0fdc32 100644 --- a/cloud-init/k3s/cloud_init.cfg +++ b/cloud-init/k3s/cloud_init.cfg @@ -19,6 +19,51 @@ write_files: - oidc-client-id=kubectl - oidc-username-claim=email - oidc-groups-claim=groups + - path: /etc/systemd/system/srv-k3s-ssd.mount + permissions: '0644' + content: | + [Unit] + Description=Retained k3s SSD data disk + Before=k3s.service + + [Mount] + What=/dev/disk/by-uuid/${retained_ssd_uuid} + Where=/srv/k3s/ssd + Type=xfs + Options=defaults + TimeoutSec=90 + DirectoryMode=0755 + + [Install] + WantedBy=multi-user.target + - path: /etc/systemd/system/srv-k3s-nvme.mount + permissions: '0644' + content: | + [Unit] + Description=Retained k3s NVMe data disk + Before=k3s.service + + [Mount] + What=/dev/disk/by-uuid/${retained_nvme_uuid} + Where=/srv/k3s/nvme + Type=xfs + Options=defaults + TimeoutSec=90 + DirectoryMode=0755 + + [Install] + WantedBy=multi-user.target + - path: /etc/systemd/system/k3s.service.d/retained-storage.conf + permissions: '0644' + content: | + [Unit] + Requires=srv-k3s-ssd.mount srv-k3s-nvme.mount + After=srv-k3s-ssd.mount srv-k3s-nvme.mount + BindsTo=srv-k3s-ssd.mount srv-k3s-nvme.mount + + [Service] + ExecStartPre=/bin/sh -ec 'test "$$(findmnt -n -o UUID --mountpoint /srv/k3s/ssd)" = "${retained_ssd_uuid}"' + ExecStartPre=/bin/sh -ec 'test "$$(findmnt -n -o UUID --mountpoint /srv/k3s/nvme)" = "${retained_nvme_uuid}"' groups: - default @@ -52,6 +97,8 @@ runcmd: - mkdir -p /var/lib/rancher - | set -e + systemctl daemon-reload + systemctl enable --now srv-k3s-ssd.mount srv-k3s-nvme.mount curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='${k3s_version}' \ sh -s - server --disable=traefik --disable=servicelb --write-kubeconfig-mode=0600 - | diff --git a/main.tf b/main.tf index 8f36d5f..8238c9b 100644 --- a/main.tf +++ b/main.tf @@ -18,6 +18,9 @@ locals { k3s_ip = "192.168.102.2" k3s_version = "v1.31.13+k3s1" + retained_ssd_uuid = "6bb8172e-7b83-4987-8eda-24b75c697860" + retained_nvme_uuid = "028fc5ee-f25b-4b28-8619-2fbc78a0a3ae" + # argocd-operator (community) — provides the argoproj.io/v1beta1 ArgoCD CRD # consumed by kustomize-cluster/bootstrap/argocd-config.yaml argocd_operator_version = "v0.14.1" @@ -72,7 +75,7 @@ module "runner" { } module "k3s" { - source = "git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git" + source = "git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git?ref=cbf1311b77b905fdd84051ae8aefe9cc2a26edcb" name = "k3s" description = "k3s single-node cluster" vcpu = 6 @@ -87,6 +90,16 @@ module "k3s" { size = 429496729600 # 400 GiB } ] + existing_disks = { + vdc = { + source_file = "/mnt/ssd/persistent/k3s-pv-ssd.raw" + serial = "k3s-retained-ssd" + } + vdd = { + source_file = "/mnt/nvme/persistent/k3s-pv-nvme.raw" + serial = "k3s-retained-nvme" + } + } cloudinit_meta_data_template = "${path.module}/cloud-init/meta_data.cfg" cloudinit_meta_data_vars = { hostname = "k3s" } cloudinit_user_data_template = "${path.module}/cloud-init/k3s/cloud_init.cfg" @@ -99,6 +112,8 @@ module "k3s" { cluster_repo_url = local.cluster_repo_url cluster_repo_branch = local.cluster_repo_branch cluster_repo_path = local.cluster_repo_path + retained_ssd_uuid = local.retained_ssd_uuid + retained_nvme_uuid = local.retained_nvme_uuid } cloudinit_network_config_template = "${path.module}/cloud-init/network_config.cfg" cloudinit_network_config_vars = { private_ip_addr = local.k3s_ip } From e833a234bf0b8d34ba2d1f395720be7e0e2c3502 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 28 Oct 2026 00:13:01 +0000 Subject: [PATCH 2/5] chore: apply pre-commit fixes --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 62a24d5..5ce9de4 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ | Name | Source | Version | | ---- | ------ | ------- | -| [k3s](#module\_k3s) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | n/a | +| [k3s](#module\_k3s) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | cbf1311b77b905fdd84051ae8aefe9cc2a26edcb | | [runner](#module\_runner) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | n/a | ## Resources From ed055d9b992713f11cda810907adbfe4173601d6 Mon Sep 17 00:00:00 2001 From: Steven Welch Date: Tue, 27 Oct 2026 19:47:25 -0600 Subject: [PATCH 3/5] docs: reduce retained SSD ceiling and clarify recovery budgets --- AGENTS.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 1017201..0fbc08c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,11 +12,13 @@ Use GitHub MCP and PR CI plans as validation authority. Same-repository PRs can | Tier | Hero path | Maximum logical bytes | Guest mount | XFS UUID | | --- | --- | ---: | --- | --- | -| SSD | `/mnt/ssd/persistent/k3s-pv-ssd.raw` | 1649267441664 (1.5 TiB) | `/srv/k3s/ssd` | `6bb8172e-7b83-4987-8eda-24b75c697860` | +| SSD | `/mnt/ssd/persistent/k3s-pv-ssd.raw` | 1374389534720 (1.25 TiB) | `/srv/k3s/ssd` | `6bb8172e-7b83-4987-8eda-24b75c697860` | | NVMe | `/mnt/nvme/persistent/k3s-pv-nvme.raw` | 429496729600 (400 GiB) | `/srv/k3s/nvme` | `028fc5ee-f25b-4b28-8619-2fbc78a0a3ae` | These are intended identities, not evidence that the images or filesystems exist. Creation, attachment and first-time initialization require separate approval of exact targets and commands. Before creation, verify the host arrays are mounted, budget existing images at maximum growth, preserve host headroom, and refuse an existing destination. Provision without preallocation and keep host access restricted to the established QEMU identity. Never initialize by guessed `/dev/vdX`: verify the configured serial, expected capacity, absence of filesystem/partition signatures and exclusive ownership first. Never force-format a replacement attachment or regenerate these UUIDs during a rebuild. +Provisionally budget 500 GiB for a private recovery archive and 500 GiB for a separate restore workspace before retained-storage growth. These are planning allowances, not allocated reservations or proven dependency-complete bounds. Do not assume the archive, restore workspace and maximum retained SSD growth can coexist. Preserve recovery copies and obtain separate approval before releasing workspace or deleting captures; no automatic cleanup is authorized. + Both UUID-based mount units are required by k3s. Missing mounts, wrong filesystem identities or mount removal intentionally prevent or stop the entire single-node k3s service. Cloud-init does not initialize these disks. Template updates prepare future bootstrap; they do not install mount units or service drop-ins in an existing VM. Existing `/dev/vdb` cloud-init formatting behavior is unchanged. Before any apply, inspect the CI plan for unexpected boot/data volume changes or VM replacement. A 2026-10-27 metadata observation found live boot capacity 5 GiB versus the declared 20 GiB; do not reconcile that drift incidentally. Obtain a verified recovery copy and an approved execution path outside the affected VM before disruption: the MCP/control workloads run on this same node. Root merge/apply, module publication, manual live setup and replacement-VM recovery remain distinct gates. From 403834371a88a745cf3c3af021b0c72da5ec1996 Mon Sep 17 00:00:00 2001 From: Steven Welch Date: Tue, 27 Oct 2026 20:17:09 -0600 Subject: [PATCH 4/5] chore: pin retained storage module to validated test correction --- main.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/main.tf b/main.tf index 8238c9b..f023319 100644 --- a/main.tf +++ b/main.tf @@ -75,7 +75,7 @@ module "runner" { } module "k3s" { - source = "git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git?ref=cbf1311b77b905fdd84051ae8aefe9cc2a26edcb" + source = "git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git?ref=5f89c3206c5ae48096cb38bd3714317d9f2b9a20" name = "k3s" description = "k3s single-node cluster" vcpu = 6 From 65788f9f943663ffc9ca56b22f2692cb2c0ebf5c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 28 Oct 2026 02:17:45 +0000 Subject: [PATCH 5/5] chore: apply pre-commit fixes --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5ce9de4..315dcb3 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ | Name | Source | Version | | ---- | ------ | ------- | -| [k3s](#module\_k3s) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | cbf1311b77b905fdd84051ae8aefe9cc2a26edcb | +| [k3s](#module\_k3s) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | 5f89c3206c5ae48096cb38bd3714317d9f2b9a20 | | [runner](#module\_runner) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | n/a | ## Resources