feat: prepare retained tier storage for k3s rebuilds

AGENTS.md

@@ -5,3 +5,24 @@ OpenTofu root for the libvirt/KVM infrastructure that hosts the Make IT Work Clo
 Use GitHub MCP and PR CI plans as validation authority. Same-repository PRs can produce a credentialed plan; `main` is an environment-gated apply path. Do not run OpenTofu, Makefile, SSH, host-firewall, state, import, taint, repair, or apply operations from this server.
 
 `README.md` is terraform-docs generated in replace mode; keep operational context here or in retrievable skills. The shared workflow and canonical runner configuration are owned by `shared-workflows` and `images/tfroot-runner`. Keep SOPS and SSH material encrypted and never expose state, credentials, private keys, or sensitive plans.
+
+## Retained k3s storage
+
+`module.k3s.existing_disks` attaches externally owned sparse raw files; this root must not create, resize, import, format, or delete them. Keep `vda` and the managed `vdb` data disk unchanged. The additional `vdc`/`vdd` files live outside `/mnt/nvme/cluster`, so replacing VM-owned images does not remove their backing files.
+
+| Tier | Hero path | Maximum logical bytes | Guest mount | XFS UUID |
+| --- | --- | ---: | --- | --- |
+| SSD | `/mnt/ssd/persistent/k3s-pv-ssd.raw` | 1374389534720 (1.25 TiB) | `/srv/k3s/ssd` | `6bb8172e-7b83-4987-8eda-24b75c697860` |
+| NVMe | `/mnt/nvme/persistent/k3s-pv-nvme.raw` | 429496729600 (400 GiB) | `/srv/k3s/nvme` | `028fc5ee-f25b-4b28-8619-2fbc78a0a3ae` |
+
+These are intended identities, not evidence that the images or filesystems exist. Creation, attachment and first-time initialization require separate approval of exact targets and commands. Before creation, verify the host arrays are mounted, budget existing images at maximum growth, preserve host headroom, and refuse an existing destination. Provision without preallocation and keep host access restricted to the established QEMU identity. Never initialize by guessed `/dev/vdX`: verify the configured serial, expected capacity, absence of filesystem/partition signatures and exclusive ownership first. Never force-format a replacement attachment or regenerate these UUIDs during a rebuild.
+
+Provisionally budget 500 GiB for a private recovery archive and 500 GiB for a separate restore workspace before retained-storage growth. These are planning allowances, not allocated reservations or proven dependency-complete bounds. Do not assume the archive, restore workspace and maximum retained SSD growth can coexist. Preserve recovery copies and obtain separate approval before releasing workspace or deleting captures; no automatic cleanup is authorized.
+
+Both UUID-based mount units are required by k3s. Missing mounts, wrong filesystem identities or mount removal intentionally prevent or stop the entire single-node k3s service. Cloud-init does not initialize these disks. Template updates prepare future bootstrap; they do not install mount units or service drop-ins in an existing VM. Existing `/dev/vdb` cloud-init formatting behavior is unchanged.
+
+Before any apply, inspect the CI plan for unexpected boot/data volume changes or VM replacement. A 2026-10-27 metadata observation found live boot capacity 5 GiB versus the declared 20 GiB; do not reconcile that drift incidentally. Obtain a verified recovery copy and an approved execution path outside the affected VM before disruption: the MCP/control workloads run on this same node. Root merge/apply, module publication, manual live setup and replacement-VM recovery remain distinct gates.
+
+After guest mounts are verified, `kustomize-cluster` owns StorageClass, static local PV and PVC declarations. Do not add them here, change existing claims, or infer data migration from new capacity. Retained local PVs need explicit node affinity and deliberate binding recovery after cluster recreation; `Retain` alone does not recover metadata or move data. Quiesced migration, isolated restore and application checks require separate approval and verification.
+
+Sparse allocation is not reserved capacity. Monitor host free space, preserve sparsity in copies, and verify guest discard/TRIM reaches host hole punching before claiming reclamation. Never mount a guest-owned filesystem simultaneously on Hero and a VM; fence the old VM before replacement attachment. RAID and retained files do not provide host-loss backup.

README.md

@@ -18,7 +18,7 @@
 
 | Name | Source | Version |
 | ---- | ------ | ------- |
-| <a name="module_k3s"></a> [k3s](#module\_k3s) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | n/a |
+| <a name="module_k3s"></a> [k3s](#module\_k3s) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | 5f89c3206c5ae48096cb38bd3714317d9f2b9a20 |
 | <a name="module_runner"></a> [runner](#module\_runner) | git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git | n/a |
 
 ## Resources

cloud-init/k3s/cloud_init.cfg

@@ -19,6 +19,51 @@ write_files:
         - oidc-client-id=kubectl
         - oidc-username-claim=email
         - oidc-groups-claim=groups
+  - path: /etc/systemd/system/srv-k3s-ssd.mount
+    permissions: '0644'
+    content: |
+      [Unit]
+      Description=Retained k3s SSD data disk
+      Before=k3s.service
+
+      [Mount]
+      What=/dev/disk/by-uuid/${retained_ssd_uuid}
+      Where=/srv/k3s/ssd
+      Type=xfs
+      Options=defaults
+      TimeoutSec=90
+      DirectoryMode=0755
+
+      [Install]
+      WantedBy=multi-user.target
+  - path: /etc/systemd/system/srv-k3s-nvme.mount
+    permissions: '0644'
+    content: |
+      [Unit]
+      Description=Retained k3s NVMe data disk
+      Before=k3s.service
+
+      [Mount]
+      What=/dev/disk/by-uuid/${retained_nvme_uuid}
+      Where=/srv/k3s/nvme
+      Type=xfs
+      Options=defaults
+      TimeoutSec=90
+      DirectoryMode=0755
+
+      [Install]
+      WantedBy=multi-user.target
+  - path: /etc/systemd/system/k3s.service.d/retained-storage.conf
+    permissions: '0644'
+    content: |
+      [Unit]
+      Requires=srv-k3s-ssd.mount srv-k3s-nvme.mount
+      After=srv-k3s-ssd.mount srv-k3s-nvme.mount
+      BindsTo=srv-k3s-ssd.mount srv-k3s-nvme.mount
+
+      [Service]
+      ExecStartPre=/bin/sh -ec 'test "$$(findmnt -n -o UUID --mountpoint /srv/k3s/ssd)" = "${retained_ssd_uuid}"'
+      ExecStartPre=/bin/sh -ec 'test "$$(findmnt -n -o UUID --mountpoint /srv/k3s/nvme)" = "${retained_nvme_uuid}"'
 
 groups:
   - default
@@ -52,6 +97,8 @@ runcmd:
   - mkdir -p /var/lib/rancher
   - |
     set -e
+    systemctl daemon-reload
+    systemctl enable --now srv-k3s-ssd.mount srv-k3s-nvme.mount
     curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='${k3s_version}' \
       sh -s - server --disable=traefik --disable=servicelb --write-kubeconfig-mode=0600
   - |

main.tf

@@ -18,6 +18,9 @@ locals {
   k3s_ip      = "192.168.102.2"
   k3s_version = "v1.31.13+k3s1"
 
+  retained_ssd_uuid  = "6bb8172e-7b83-4987-8eda-24b75c697860"
+  retained_nvme_uuid = "028fc5ee-f25b-4b28-8619-2fbc78a0a3ae"
+
   # argocd-operator (community) — provides the argoproj.io/v1beta1 ArgoCD CRD
   # consumed by kustomize-cluster/bootstrap/argocd-config.yaml
   argocd_operator_version = "v0.14.1"
@@ -72,7 +75,7 @@ module "runner" {
 }
 
 module "k3s" {
-  source         = "git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git"
+  source         = "git::https://github.com/makeitworkcloud/terraform-libvirt-domain.git?ref=5f89c3206c5ae48096cb38bd3714317d9f2b9a20"
   name           = "k3s"
   description    = "k3s single-node cluster"
   vcpu           = 6
@@ -87,6 +90,16 @@ module "k3s" {
       size = 429496729600 # 400 GiB
     }
   ]
+  existing_disks = {
+    vdc = {
+      source_file = "/mnt/ssd/persistent/k3s-pv-ssd.raw"
+      serial      = "k3s-retained-ssd"
+    }
+    vdd = {
+      source_file = "/mnt/nvme/persistent/k3s-pv-nvme.raw"
+      serial      = "k3s-retained-nvme"
+    }
+  }
   cloudinit_meta_data_template = "${path.module}/cloud-init/meta_data.cfg"
   cloudinit_meta_data_vars     = { hostname = "k3s" }
   cloudinit_user_data_template = "${path.module}/cloud-init/k3s/cloud_init.cfg"
@@ -99,6 +112,8 @@ module "k3s" {
     cluster_repo_url        = local.cluster_repo_url
     cluster_repo_branch     = local.cluster_repo_branch
     cluster_repo_path       = local.cluster_repo_path
+    retained_ssd_uuid       = local.retained_ssd_uuid
+    retained_nvme_uuid      = local.retained_nvme_uuid
   }
   cloudinit_network_config_template = "${path.module}/cloud-init/network_config.cfg"
   cloudinit_network_config_vars     = { private_ip_addr = local.k3s_ip }