Repository navigation
Scaffold hardened Axum application Docker recipes - #401
ChristianPavilonis wants to merge 10 commits into
Conversation
Retain full vulnerability evidence in framework packaging checks without treating the fixture as an approved release. Keep strict defaults and reject report-only GHCR publication. Update inherited Bookworm packages and cover the policy boundary with regression tests.
Reject empty and non-string CVE/package identities in both scan policies. Record the successful hosted native builds and artifact/index handoff.
Replace Python helpers and tests with dependency-free Node.js tooling, update workflows and documentation, and preserve OCI artifact verification contracts. Accept valid repeated separators in image repository names and share Compose startup polling with regression coverage.
prk-Jr
left a comment
There was a problem hiding this comment.
🔧 Please resolve the integration conflicts against current main and validate the resulting revision before merging.
The reviewed PR head is 493644242f5839afbe5d37a032e08c13a381dc04; current main was checked at bc4aeb850c74ba982bdb4ad45309520b2ebb07b5. The merge check identifies conflicts in crates/edgezero-cli/src/generator.rs, .github/workflows/deploy-action.yml, and the location of docs/specs/aws-reference-deployment.md.
When resolving the generator conflict, keep the generated Rust version and the Docker builder digest aligned with main's toolchain upgrade. This PR introduces GENERATED_RUST_VERSION = "1.95.0" and a pinned 1.95 builder, while current main uses 1.98.1. Update the toolchain declaration and builder digest together; changing the image tag alone leaves the old digest selected. Rerun the affected checks on the integrated revision, including container preparation and native image execution.
No additional confirmed code defects were found in this review pass across the 29 changed files and relevant consumers.
Validation at the reviewed PR head: workspace tests passed (1,424 passed, one ignored), formatting and strict all-feature Clippy passed, native feature compilation and Spin WASM compilation passed, container helper tests passed including the prepared-fixture checks, and docs lint/format/build passed. Workflow and shell static checks passed, with a different local actionlint version from CI. These results do not validate a resolved merge with current main. Real Docker execution could not be repeated because the local daemon was unavailable; no current-head hosted checks were listed.
😃 The publisher independently recomputes committed source identities before registry login, verifies both architecture bundles, and prevents report-only scans from authorizing GHCR publication.
aram356
left a comment
There was a problem hiding this comment.
PR Review
Summary
Adds a hardened Axum Dockerfile scaffold, native amd64/arm64 container CI with an OCI artifact handoff, an app-owned GHCR publishing recipe, and Docker/Kubernetes/AWS deployment docs. The verification tooling is careful, but the generated image ignores SIGTERM, its runtime base means the shipped strict release gate rejects every build, and several integration hazards with current main merge without a textual conflict. This review adds to the existing conflict request rather than repeating it.
I built the fixture image from this head locally (native arm64) and measured its stop behavior and scan results; the numbers are in the inline comments.
Findings
Blocking
- 🔧 The container ignores SIGTERM, so every stop ends in SIGKILL (
crates/edgezero-cli/src/templates/root/Dockerfile.hbs:64, inline) - 🔧 The runtime base makes the shipped strict release gate unpassable (
crates/edgezero-cli/src/templates/root/Dockerfile.hbs:52, inline) - 🔧 New
stackpoplinks after the repository transfer; the generated README's docs link returns 404 (crates/edgezero-cli/src/templates/root/README.md.hbs:57, inline) - 🔧 The toolchain test breaks on the required Rust 1.98.1 bump (
crates/edgezero-cli/src/generator.rs:1141, inline) - 🔧 Merging would close #398, #399 and #400: GitHub lists all three as closing references for this PR, but the description says #399 and #400 must stay open and that #398's production acceptance is still outstanding. The description also calls the PR a draft, but it is not marked as one. Remove the #399/#400 links from the Development sidebar, decide explicitly whether this merge should close #398, and convert the PR to draft if it is not ready to merge.
- 🔧 The spec relocation breaks guide links (in addition to the existing conflict request): on main,
docs/specsmoved underdocs/superpowers/, whichdocs/.vitepress/config.mtsexcludes withsrcExclude: ['superpowers/**']. Onceaws-reference-deployment.mdmoves there, the relative links atdocs/guide/deploying-axum-containers.md:150anddocs/guide/deploying-axum-kubernetes.md:160,240point at pages the site no longer builds, and VitePress fails the build on dead links by default. Link to the file on GitHub instead, or drop those links. Also note that no hosted CI has run on49364424: GitHub does not runpull_requestworkflows while a PR has merge conflicts, so the integrated revision will be the first to exercise the new workflows on this head.
Non-blocking
- ♻️ Split this PR: 29 files and +4,325 lines combine about 340 lines of scaffold change (#398), about 3,000 lines of CI, verification and publishing tooling with tests (#399), and about 1,000 lines of deployment and AWS design docs (#400/#410). Each part has different reviewers, risk and acceptance criteria. Suggested split: (1) the Dockerfile,
.dockerignoreand README templates, generator tests and the Axum guide section; (2) native CI, the handoff helpers and the publishing recipe; (3) the design docs underdocs/superpowers/specs/. - ♻️ Keep point-in-time status out of committed docs: several pages record review-time state that will be wrong soon after merge. Examples: dated scan counts and a specific CI run link (
docs/guide/publishing-axum-containers.md:208-226); this PR's own heads and CI status (docs/specs/aws-reference-deployment.md:24-36); a developer's local, uncommitted worktree (issue-410-native-store-providers,docs/specs/aws-reference-deployment.md:32and:215); and issue-ownership notes in the Axum guide's "Packaging verification and limits" section. That belongs in the PR description and the issues. The docs should state the contract and its current limits. - ♻️ Keep the Kubernetes draft out of the published navigation (
docs/.vitepress/config.mts:61, inline) - ♻️ Replace the hand-written TOML parser (
scripts/axum_container_toml.mjs:3, inline) - ♻️ One implementation of the context identity, not three (
examples/container-publishing/release.yml:46, inline) - ♻️ Every source edit reinstalls cmake and rebuilds all dependencies (
crates/edgezero-cli/src/templates/root/Dockerfile.hbs:40, inline) - 🤔 The vulnerability gate can't see the Rust dependencies (
crates/edgezero-cli/src/templates/root/Dockerfile.hbs:46, inline) - 🤔 The container CI resolves a fresh dependency graph on every run (
scripts/prepare_generated_axum_container.sh:22, inline) - 🤔 Downstream-specific build steps in the framework guide (
docs/guide/adapters/axum.md:305, inline) - ⛏ Non-hermetic unit test fails on hosts without
readelf(scripts/tests/test_axum_container_artifacts.test.mjs:325, inline) - ⛏ Stale Python references after the Node migration (
scripts/axum_container_helpers.mjs:87, inline) - ⛏ Prerequisite lists disagree (
scripts/test_generated_axum_container.sh:7, inline) - ⛏ Absolute symlink targets never resolve (
scripts/axum_container_helpers.mjs:171, inline) - ⛏ Duplicate BuildKit pin for an unused builder (
.github/workflows/axum-container.yml:104, inline) - ⛏ Docs-only edits run both native container builds (
.github/workflows/axum-container.yml:18, inline) - ⛏ Drop bzip2/xz decompression of artifact input (
examples/container-publishing/verify_artifacts.mjs:189, inline)
CI Status
Local runs at 49364424:
- fmt: PASS
- clippy (
--workspace --all-targets --all-features -- -D warnings): PASS - tests (
cargo test --workspace --all-targets): PASS - Node helper tests: 50 pass, 1 fail on macOS (non-hermetic; see inline), 3 skipped by default. The 3 prepared-fixture tests pass with
AXUM_PREPARED_CONTEXTset. - shellcheck, actionlint and zizmor on the new workflows and scripts: clean
- Fixture
docker build(native arm64): PASS - Hosted CI on this head: none has run (blocked by merge conflicts)
| strip --strip-debug /out/app | ||
| SH | ||
|
|
||
| FROM debian:bookworm-slim@sha256:3783cc01769c7b2b1b83a5c5ad96c815348e28ed7da68e2e3687004faa906251 AS runtime |
There was a problem hiding this comment.
🔧 The runtime base makes the shipped strict release gate unpassable
The publishing recipe rejects any High or Critical finding, but this base puts 91 Debian packages around a 6.6 MB binary. Scanning the fixture image with the pinned TRIVY_IMAGE from tools.env (arm64, today): 56 High, 1 Critical, none fixable. Every finding is in a package the binary never loads: util-linux and its libraries (libmount1, libblkid1, libuuid1, libsmartcols1, bsdutils, mount), ncurses, perl-base, libsystemd0/libudev1, gzip, and zlib1g (CVE-2023-45853). ldd /usr/local/bin/app needs only libc, libm and libgcc_s. The image also ships sh, bash, apt, dpkg, perl and openssl.
The same binary on other bases, same scanner:
| Runtime base | High | Critical | Size |
|---|---|---|---|
debian:bookworm-slim plus upgrade (this PR) |
56 | 1 | 177 MB |
debian:trixie-slim plus upgrade |
43 | 0 | 171 MB |
gcr.io/distroless/cc-debian12:nonroot |
1 (fixable) | 0 | 57 MB |
The distroless variant served /, completed an outbound TLS proxy request using its CA bundle, ran read-only as 10001:10001, and stopped cleanly with STOPSIGNAL SIGINT.
Fix: use a digest-pinned gcr.io/distroless/cc-debian12:nonroot@sha256:… runtime and drop the apt layer; that image already contains CA certificates and libgcc_s. The smokes need adjusting, because they run sh, ldd and openssl s_server from inside the app image (scripts/test_generated_axum_container.sh:134-152 and :169-173, scripts/test_axum_container_deployment.sh:36). Check user, capabilities and read-only root through docker inspect, and run the TLS upstream from a separate pinned image.
Separately, Bookworm has been oldstable since Trixie's release in August 2025, so a new scaffold should not start there.
| .args(["-c", check]) | ||
| .current_dir(temp.path()) | ||
| .env("PATH", &path) | ||
| .env("RUST_VERSION", GENERATED_RUST_VERSION) |
There was a problem hiding this comment.
🔧 This test breaks on the required Rust 1.98.1 bump
The fixtures hard-code rust 1.95.0, channel = "1.95.0" and an installed 1.95.0, but RUST_VERSION comes from GENERATED_RUST_VERSION. Integrating main requires that constant to become 1.98.1 (main's build_tool_versions_pins_track_repo_tool_versions enforces it). With the constant set to 1.98.1 locally, this test fails on its first case:
container toolchain: update the Docker Rust pin to match .tool-versions
The test checks the shell script's logic, not the constant, so pin its input:
Fix:
.env("RUST_VERSION", "1.95.0")Alternatively, build every fixture string from GENERATED_RUST_VERSION with format!.
| with: | ||
| version: v0.37.0 | ||
| use: false | ||
| driver-opts: image=moby/buildkit:v0.25.1@sha256:79cc6476ab1a3371c9afd8b44e7c55610057c43e18d9b39b68e2b0c2475cc1b6 |
There was a problem hiding this comment.
⛏ Duplicate BuildKit pin for an unused builder
With use: false, this step only installs buildx; scripts/test_axum_container_handoff.sh creates its own builder from BUILDKIT_IMAGE in examples/container-publishing/tools.env. The driver-opts image here, and the same line in examples/container-publishing/release.yml:116, is a second copy of that pin that configures a builder nothing uses, so the two can drift unnoticed. Drop driver-opts and keep tools.env as the single source.
dhruv8sh
left a comment
There was a problem hiding this comment.
PR Review
Summary
Solid, security-conscious Axum container packaging, but the branch is stale against main (missing #366 and #419), conflicts, and reintroduces stackpop references. Rebase and the items below are needed before merge.
😃 Praise
- Strong hardening: non-root UID 10001,
--read-only,cap-drop=ALL,no-new-privileges; digest-pinned base images and actions withpersist-credentials: false; toolchain verified beforerustup; strict scan policy and post-push digest verification for GHCR; Dockerfile/.dockerignore generated only for the Axum adapter, with good unit tests (mock-rustuptoolchain check, digit-leading app names).
Findings
Blocking
- 🔧 Stale base / merge conflict: branch lacks #366 (Rust 1.98.1, Node 24.20.0) and #419 (IABTechLab transfer);
git merge-treereports a conflict.GENERATED_RUST_VERSION = "1.95.0"ingenerator.rswhile main emitsrust 1.98.1. Rebase, bump to 1.98.1, re-resolve therust:1.98.1-bookworm@sha256:…digest inDockerfile.hbs, and update the hard-coded"1.95.0"in the new tests. - 🔧
stackpopreferences reintroduced (inline on the README template). Also indocs/guide/publishing-axum-containers.md(git dependency examples), issue/PR links in the Kubernetes guide anddocs/specs/aws-reference-deployment.md, and theorg.opencontainers.image.sourcefallbackstackpop/edgezeroinexamples/container-publishing/release.yml. Switch toIABTechLab/edgezero/iabtechlab.github.io/edgezero. - 🔧 Hard-coded Node 24.12.0 in workflows (inline).
- ❓ Draft docs in main nav (inline).
Non-blocking
- 🤔 Bespoke tooling scope:
scripts/axum_container_toml.mjsis a 182-line hand-rolled TOML parser andexamples/container-publishing/verify_artifacts.mjsis 524 lines of OCI/tar verification; withrelease.ymlandpublish.shthat is a lot of security-sensitive code for the framework to own as an "example". Consider a small TOML dep (e.g.smol-toml) for tests, or splitting the publishing recipe into its own PR. - 🤔 Missing
# syntax=directive for heredocRUN(inline). - ⛏ No dependency layer caching in the Dockerfile (inline).
- ⛏
rmSyncwithoutforce(inline). - 📝 Gate toolchain: clippy on 1.98.1 fails in untouched
edgezero-macros(inline_trait_bounds) because the branch still pins 1.95.0; resolved by the rebase.
CI Status
- fmt: PASS
- clippy: PASS on branch pin 1.95.0; FAIL on 1.98.1 (pre-existing lint in untouched crate, fixed by rebase)
- tests: PASS
- Docker/OCI workflows and Node helper tests: not run locally (require Docker/buildx)
|
|
||
| # Support exact release pins and the conventional [toolchain] channel declaration. | ||
| # Validate before invoking rustup, which could otherwise download a directory override. | ||
| RUN <<'SH' |
There was a problem hiding this comment.
🤔 Heredoc RUN <<'SH' without a # syntax= directive: heredocs need Dockerfile frontend 1.4+; legacy builders/older engines fail with an opaque parse error.
Fix: add # syntax=docker/dockerfile:1.x@sha256:… as line 1, digest-pinned like the other inputs.
| ARG RUST_VERSION | ||
| ENV RUSTUP_TOOLCHAIN=${RUST_VERSION} | ||
| WORKDIR /build | ||
| COPY . . |
There was a problem hiding this comment.
⛏ No dependency layer caching: COPY . . before cargo build rebuilds all dependencies on every source change. Optional: cargo-chef stage or RUN --mount=type=cache,target=/usr/local/cargo/registry ....
Resolve merge conflicts, preserve combined CI coverage, align container scaffolding with Rust 1.98.1, and repair relocated documentation links.
…ps. Changes remain uncommitted; generated checkout artifacts were removed. Validation remains incomplete for native arm64 and hosted release flows.
prk-Jr
left a comment
There was a problem hiding this comment.
🔧 Review: request changes
Reviewed all 33 changed files at 1f244a7c1b78dafd5bc9291ad86c385360636f72, against 87456c708ffe9edf86b3a0f8b06dfd15d60b07b5. One existing merge-metadata blocker remains; one new, nonblocking Compose smoke finding is attached inline. No additional blocking defect was confirmed in scaffolding, toolchain validation, container hardening, TLS checks, publishing permissions, or deployment contracts.
🔧 P2 — Merging would prematurely close unfinished issues (existing finding, still unresolved)
GitHub currently lists #398, #399, and #400 as closing references. The PR explicitly says #399/#400 must remain open, while #398 still has outstanding production acceptance. Merging would therefore close work the PR acknowledges is incomplete.
Remove #399/#400 from the closing associations and retain #398 until its remaining acceptance is complete, or explicitly separate the delivered packaging slice. The PR also remains marked ready for review despite its description calling it draft.
😃 The artifact handoff is carefully designed. The publisher independently recomputes committed source identities before registry login, verifies both architecture bundles and their metadata, and preserves tested child digests. Report-only scans cannot authorize GHCR publication.
Validation: Rust 1.98.1 workspace tests passed (1,553 passed, one ignored), together with formatting, strict all-feature Clippy, native feature compilation, and Spin WASM compilation. The helper suite passed 77 tests, followed by all three prepared-context checks passing. Local Node was 24.12.0; hosted checks used the declared 24.20.0. ShellCheck, actionlint, and diff checks passed. Hosted native amd64/arm64 container and handoff checks passed; their merge-result file tree exactly matches the reviewed head.
Local Docker execution was unavailable because the daemon was stopped. Protected GHCR publication, attestations, Kubernetes, and AWS acceptance remain unexecuted; the docs explicitly defer those claims. Green packaging checks do not establish release security approval: current hosted logs retain one High OS finding and five Rust advisories.
| cleanup() { | ||
| status=$? | ||
| compose logs > "$WORK/compose.log" 2>&1 || true | ||
| compose down >/dev/null 2>&1 || true |
There was a problem hiding this comment.
🤔 P3 — Compose smoke reports successful teardown when cleanup fails (nonblocking)
The cleanup trap discards compose down failures, deletes diagnostics, and returns the earlier successful status. The script nevertheless prints that teardown passed.
Executed reproduction at 1f244a7c: changing only the existing mock's down operation to return failure produced exit 0, the teardown PASS message, and empty stderr. A real removal failure could leave the fixture container, network, and published port behind without reporting it.
Run checked teardown before printing PASS, retaining best-effort trap cleanup for failure paths. Add a regression requiring failed teardown to return nonzero and preserve diagnostics. This does not invalidate the successful startup or image-integrity checks.
Summary
.dockerignore, using the application's lockfile and declared Rust toolchain, pinned GNU/Bookworm inputs, CA trust, numeric non-root identity, exec startup, and explicit container binding.Scope and issue relationships
Part of #391.
Closes #398.
The existing #398 closing association remains, but this PR is still draft. Packaging evidence does not satisfy its remaining production-integration acceptance or justify merging before those requirements are resolved.
Related to #399 and #400. This PR implements their preparation slices, not their complete acceptance criteria; neither issue should close on this merge.
Related to #410 and draft PR #414 for AWS provider integration. This PR adds reference design and deployment guidance only, not provider implementation, DynamoDB/identity certification, real AWS acceptance or a closing association for #410.
Applications own their release artifacts, registry approval, credentials, configuration, secrets, storage policy, and deployment. Trusted Server's actual image, asset/config integration, installer, and releases remain downstream.
Changes
scripts/prepare_generated_axum_container.shcontainer-probe, deliberately stage framework sources inside its context, preserve dependency features, and prepare one application-owned lockfile for both architecturesscripts/test_generated_axum_container.sh.github/workflows/axum-container.ymlscripts/test_axum_container_handoff.shandexamples/container-publishing/examples/axum-deployment/compose.yamland deployment smokedocs/guide/deploying-axum-kubernetes.mdand documentation navigationdocs/specs/aws-reference-deployment.mdThe CI image is a generated disposable
container-probeapplication, not app-demo, Trusted Server, or a public EdgeZero application image. Test-only source staging does not change scaffold dependency selection or replace the original standalone-consumer proof using portable Git dependencies.CI and recorded security status
The documentation-only update is pushed at
7f846ae02cf2ff334673c59ccd2f6eef076139ba. Current-head hosted CI still requires verification. Container/runtime evidence below was recorded at implementation head68894a05f6f4a8c49a8a2b7bd7fee2bed24efcfe; it is not a claim that those jobs have rerun at the documentation head.custom-bwithmissing partial committed response. The previous run passed; intermittence is suspected, not confirmed. Investigate and verify current-head results rather than treat it as an image-scan failure.main; branch integration and affected validation remain outstanding.Framework packaging/handoff uses
SCAN_POLICY=report-only. Scan execution, malformed reports and artifact-integrity failures still block; vulnerability findings are retained and reported, not waived. The publisher and release-record verification remain strict by default and reject High/Critical findings, including unfixed findings, unless an independently trusted owner-reviewed scoped exception is supplied. Report-only policy cannot authorize GHCR publication.Bookworm package upgrades cleared the eight findings with available fixes in the recorded scan. The latest recorded image scans report 50 High and 1 Critical finding, with no available fixes reported. Packaging passes under report-only; strict release verification rejects the same image. No vulnerability exceptions have been authorized. The image scanner found no language-specific files, so the SBOM is not independent certification of every statically linked Rust dependency.
Documentation update validation
For the documentation-only change in
7f846ae0:npm run lintandnpm run buildpassed at the unchanged documentation inputs. The Kubernetes page rendered and local Markdown links resolved.cargo test -p edgezero-clipassed before commit: 207 unit tests and one external-consumer test passed; the generated-workspace build test remained ignored.git diff --checkand staged-diff checks passed. Only four documentation/navigation files changed.Recorded local validation
These are executed results from their recorded implementation checkpoints, not a claim that every local command was rerun at the current head or that all CI is green.
98930917, its own committedCargo.lock, and no out-of-context dependencies. Its unchanged generated Dockerfile passed the native amd64 smoke.Remaining acceptance
run_appentrypoint do not prove these properties.Packaging preparation is not production certification. Preserve accepted HTTP protocol limits and BestEffort/Unsupported distinctions. No externally published image, GHCR/ECR acceptance, Kubernetes/EKS acceptance, Fargate acceptance, real AWS provider acceptance, identity/CAS certification, end-client-delivery guarantee, or multi-writer embedded-store support is claimed.