Skip to content

feat: prepare retained tier storage for k3s rebuilds - #31

Open
xnoto wants to merge 5 commits into
mainfrom
feat/retained-k3s-storage
Open

xnoto wants to merge 5 commits into
mainfrom
feat/retained-k3s-storage

Conversation

@xnoto

@xnoto xnoto commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

HOLD — verified disruptive apply: the earlier CI resource-action summary does not replace the k3s VM or boot/data volumes; it updates the domain in place and replaces the cloud-init ISO. However, libvirt provider 0.9.9 Update stops, redefines and restarts a running VM for device changes. Its default shutdown timeout is 30 seconds with forced power-off on timeout. Therefore this is not a non-disruptive attachment. Do not merge/apply before the owner-approved data-protection/migration and external-control gates. Important state must be inventoried beyond PVC declarations, and a later full rebuild must not precede verified migration to retained storage. No live setup or migration has occurred.

Prepare two externally owned sparse raw disks for k3s: SSD 1.25 TiB and NVMe 400 GiB. Attach as vdc/vdd, preserving vda and managed vdb. Add UUID-based XFS mount units and fail-closed k3s dependencies/checks.

Depends on terraform-libvirt-domain #28; k3s pins reviewed module commit 5f89c3206c5ae48096cb38bd3714317d9f2b9a20. Owner-requested preparation, no issue reference. Not ready for apply or live setup.

Type of change

  • Feature / enhancement
  • Documentation
  • Infrastructure (OpenTofu root or module)

Validation

  • Required pull-request checks pass
  • Generated or centrally distributed files were regenerated by their owning automation, not hand-edited

Latest root CI passed test and plan on 65788f9f943663ffc9ca56b22f2692cb2c0ebf5c; apply was skipped. Selected producer CI passed all 27 mock tests on 5f89c3206c5ae48096cb38bd3714317d9f2b9a20. This is CI validation, not capture, restore or runtime verification.

Use unchanged PR CI/pre-commit and credentialed plan. Root README is generated in replace mode; existing automation may commit updates to this branch. No workflow, runner image or centralized hook changes.

Adversarial, infrastructure-security, delivery-integration and release reviews cleared the completed candidate. QA identified an explicit gap: there are no dedicated root cloud-init render/guest behavior tests. The owner declined the proposed new CI job; this is an accepted CI coverage gap, not runtime safety verification or a waiver. Module #28 adds 22 mock tests through existing module CI. No local OpenTofu commands ran.

Before any apply: inspect resource actions for VM replacement or unexpected boot/data volume changes, obtain verified recovery protection, and establish control outside the affected node. Existing observed boot drift (5 GiB live versus 20 GiB declared) is not intentionally corrected here.

Impact and rollout

  • Changed paths: main.tf, cloud-init/k3s/cloud_init.cfg, operational contract in AGENTS.md; generated README may follow automatically. Existing runner source and managed volume declarations unchanged.
  • Module PR CI/publication and root PR/selection/apply are distinct. Root source selects the exact feature commit, not a published version tag. Merge/release/apply are not authorized. Existing main workflow can initiate production-environment-gated apply after merge.
  • Backing files are attachment-only inputs, not libvirt_volume resources. No file creation, resizing, import, new-disk formatting or deletion. Paths, capacities and intended UUIDs are documented in AGENTS.md.
  • Cloud-init changes prepare future bootstraps; they do not configure mounts in the current VM. Missing either mount or a wrong UUID intentionally prevents k3s startup; mount removal stops the whole single-node service.
  • Live creation/attachment/initialization require separate exact-operation confirmation. Guest SSH execution was unavailable and no guest agent was configured in the prior inspection; no credentials or privileged bypass were introduced.
  • After attachment and verified guest mounts, kustomize-cluster owns StorageClass/static PV/PVC work. No Kubernetes objects, claim replacement or migration are included. Rebinding, recovery and discard/TRIM remain unverified.
  • Rollback: revert only before dependence on these mounts, or plan a deliberate gated detach after workloads are quiesced. Never format existing retained files or attach them to simultaneous writers. RAID is not host-loss backup.

Safety and secrets

  • Contains no plaintext secrets, decrypted SOPS values, state files, kubeconfigs, tokens, or private endpoints
  • No local OpenTofu init/plan/apply/destroy/import/state operations were run or claimed — plans come from pull-request checks
  • Breaking or irreversible effects are described above with rollback notes

AI-assisted implementation and independent specialist reviews. No host/guest mutations, merge, publication, deployment or workflow dispatch requested.

@xnoto
xnoto requested a review from a team as a code owner October 28, 2026 00:12
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

OpenTofu Test

OpenTofu test passed.

View the workflow run.

View run output
Terraform validate.......................................................Passed
Terraform validate with tflint...........................................Passed
Checkov..................................................................Passed
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Detect hardcoded secrets.................................................Passed
check for case conflicts.................................................Passed
check for merge conflicts................................................Passed
check for broken symlinks............................(no files to check)Skipped
check vcs permalinks.....................................................Passed
detect destroyed symlinks................................................Passed
detect private key.......................................................Passed
fix end of files.........................................................Passed
mixed line ending........................................................Passed
trim trailing whitespace.................................................Passed
don't commit to branch..................................................Skipped
check for added large files..............................................Passed

@github-actions

Copy link
Copy Markdown

OpenTofu Plan

OpenTofu plan passed.

View the workflow run.

View run output
OpenTofu will perform the following actions:

  # module.k3s.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: k3s
            local-hostname: k3s
        EOT
      + name           = "k3s_commoninit"
      + network_config = <<-EOT
            version: 2
            ethernets:
              enp1s0:
                dhcp4: true
              enp2s0:
                dhcp4: false
                addresses:
                  - 192.168.102.2/24
        EOT
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

  # module.k3s.libvirt_domain.vm will be updated in-place
  ~ resource "libvirt_domain" "vm" {
      ~ devices     = {
          ~ disks      = [
              ~ {
                  - device = "cdrom" -> null
                  + driver = {
                      + discard = "unmap"
                      + name    = "qemu"
                      + type    = "raw"
                    }
                  + serial = "k3s-retained-ssd"
                  ~ source = {
                      + file   = {
                          + file = "/mnt/ssd/persistent/k3s-pv-ssd.raw"
                        }
                      - volume = {
                          - pool   = "cluster" -> null
                          - volume = "k3s_cloudinit.iso" -> null
                        } -> null
                    }
                  ~ target = {
                      ~ bus = "sata" -> "virtio"
                      ~ dev = "sda" -> "vdc"
                    }
                },
              + {
                  + driver = {
                      + discard = "unmap"
                      + name    = "qemu"
                      + type    = "raw"
                    }
                  + serial = "k3s-retained-nvme"
                  + source = {
                      + file = {
                          + file = "/mnt/nvme/persistent/k3s-pv-nvme.raw"
                        }
                    }
                  + target = {
                      + bus = "virtio"
                      + dev = "vdd"
                    }
                },
              + {
                  + device = "cdrom"
                  + source = {
                      + volume = {
                          + pool   = "cluster"
                          + volume = "k3s_cloudinit.iso"
                        }
                    }
                  + target = {
                      + bus = "sata"
                      + dev = "sda"
                    }
                },
                # (2 unchanged elements hidden)
            ]
            # (2 unchanged attributes hidden)
        }
        id          = 1
        name        = "k3s"
        # (10 unchanged attributes hidden)
    }

  # module.k3s.libvirt_volume.cloudinit will be replaced due to changes in replace_triggered_by
-/+ resource "libvirt_volume" "cloudinit" {
      ~ allocation = 49152 -> (known after apply)
      ~ capacity   = 49152 -> (known after apply)
      ~ create     = {
          ~ content = {
              ~ url = "/tmp/terraform-provider-libvirt-cloudinit/cloudinit-4c952d6c62b57a7c.iso" -> (known after apply)
            }
        }
      ~ id         = "/mnt/nvme/cluster/k3s_cloudinit.iso" -> (known after apply)
      ~ key        = "/mnt/nvme/cluster/k3s_cloudinit.iso" -> (known after apply)
        name       = "k3s_cloudinit.iso"
      ~ path       = "/mnt/nvme/cluster/k3s_cloudinit.iso" -> (known after apply)
      ~ physical   = 49152 -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # module.k3s.terraform_data.cloudinit_content will be updated in-place
  ~ resource "terraform_data" "cloudinit_content" {
        id     = "7de40bc4-9054-f847-1b2b-d84f3f3af2d1"
      ~ input  = (sensitive value)
      ~ output = "b363b3b1637577baa97428e2069ee49eaec7cbbaf0429b889d527db4e17ba7cf" -> (known after apply)
    }

  # module.runner.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: runner
            local-hostname: runner
        EOT
      + name           = "runner_commoninit"
      + network_config = (sensitive value)
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

Plan: 3 to add, 2 to change, 1 to destroy.
OpenTofu will perform the following actions:

  # module.k3s.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: k3s
            local-hostname: k3s
        EOT
      + name           = "k3s_commoninit"
      + network_config = <<-EOT
            version: 2
            ethernets:
              enp1s0:
                dhcp4: true
              enp2s0:
                dhcp4: false
                addresses:
                  - 192.168.102.2/24
        EOT
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

  # module.k3s.libvirt_domain.vm will be updated in-place
  ~ resource "libvirt_domain" "vm" {
      ~ devices     = {
          ~ disks      = [
              ~ {
                  - device = "cdrom" -> null
                  + driver = {
                      + discard = "unmap"
                      + name    = "qemu"
                      + type    = "raw"
                    }
                  + serial = "k3s-retained-ssd"
                  ~ source = {
                      + file   = {
                          + file = "/mnt/ssd/persistent/k3s-pv-ssd.raw"
                        }
                      - volume = {
                          - pool   = "cluster" -> null
                          - volume = "k3s_cloudinit.iso" -> null
                        } -> null
                    }
                  ~ target = {
                      ~ bus = "sata" -> "virtio"
                      ~ dev = "sda" -> "vdc"
                    }
                },
              + {
                  + driver = {
                      + discard = "unmap"
                      + name    = "qemu"
                      + type    = "raw"
                    }
                  + serial = "k3s-retained-nvme"
                  + source = {
                      + file = {
                          + file = "/mnt/nvme/persistent/k3s-pv-nvme.raw"
                        }
                    }
                  + target = {
                      + bus = "virtio"
                      + dev = "vdd"
                    }
                },
              + {
                  + device = "cdrom"
                  + source = {
                      + volume = {
                          + pool   = "cluster"
                          + volume = "k3s_cloudinit.iso"
                        }
                    }
                  + target = {
                      + bus = "sata"
                      + dev = "sda"
                    }
                },
                # (2 unchanged elements hidden)
            ]
            # (2 unchanged attributes hidden)
        }
        id          = 1
        name        = "k3s"
        # (10 unchanged attributes hidden)
    }

  # module.k3s.libvirt_volume.cloudinit will be replaced due to changes in replace_triggered_by
-/+ resource "libvirt_volume" "cloudinit" {
      ~ allocation = 49152 -> (known after apply)
      ~ capacity   = 49152 -> (known after apply)
      ~ create     = {
          ~ content = {
              ~ url = "/tmp/terraform-provider-libvirt-cloudinit/cloudinit-4c952d6c62b57a7c.iso" -> (known after apply)
            }
        }
      ~ id         = "/mnt/nvme/cluster/k3s_cloudinit.iso" -> (known after apply)
      ~ key        = "/mnt/nvme/cluster/k3s_cloudinit.iso" -> (known after apply)
        name       = "k3s_cloudinit.iso"
      ~ path       = "/mnt/nvme/cluster/k3s_cloudinit.iso" -> (known after apply)
      ~ physical   = 49152 -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # module.k3s.terraform_data.cloudinit_content will be updated in-place
  ~ resource "terraform_data" "cloudinit_content" {
        id     = "7de40bc4-9054-f847-1b2b-d84f3f3af2d1"
      ~ input  = (sensitive value)
      ~ output = "b363b3b1637577baa97428e2069ee49eaec7cbbaf0429b889d527db4e17ba7cf" -> (known after apply)
    }

  # module.runner.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: runner
            local-hostname: runner
        EOT
      + name           = "runner_commoninit"
      + network_config = (sensitive value)
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

Plan: 3 to add, 2 to change, 1 to destroy.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant