Repository navigation
ci: keep the freeze cache alive and slim cold-build worker clones - #431
Merged
Merged
Conversation
Root causes found in run 41446738921 (PR #428, ~3 h wall time): - The repo-wide 10 GB cache pool was overflowing (~10.8 GB), so the _freeze cache got evicted and 668 QMD files re-executed (~85 min). Largest polluters: 2.3 GB setup-r-dependencies caches saved by pr-review.yml and ~1 GB renv caches saved per pull_request run (merge-ref caches are visible only to that same PR). - Each pre-render worker tar-copied the full tree (~283 MB) per file; mcp/ alone is ~109 MB of node assets no render touches. Fixes: - pr-review.yml: disable the 2.3 GB setup-r-dependencies cache. - setup-render-env: pull_request runs restore the renv cache but never save it (bypass-cache: true), with the same key format setup-renv uses so PRs still hit entries saved by main-branch runs. - render-site + ci-benchmark: exclude mcp/ and local cache dirs from the per-file worker clone (~70% less clone I/O on cold runs).
Contributor
✅ QMD Format CheckNo new QMD tutorial files to check. |
…ze entries The per-file worker clones discarded the project-level _freeze/site_libs/ directory, so a fresh render's html dependency copies (e.g. simplelightbox-2.1.0 newly emitted by the repinned pixture fork) never reached the shared freeze cache. Frozen renders then reference assets that exist nowhere, producing dangling site_libs links on the deployed site and aborting the zh dedup step (prune-zh-duplicates LEFTOVER, first seen on PRs #426/#428). - workers now rsync _freeze/site_libs/ back like the per-page dirs - a self-heal step drops freeze pages whose execute-results/html.json references site_libs assets missing from the shared cache, so they re-render and repopulate it
Member
Author
|
Added a follow-up fix for a related bug this PR's CI run exposed (also failing PRs #426/#428 independently): worker clones discarded the project-level |
Contributor
✅ QMD Format CheckNo new QMD tutorial files to check. |
Review follow-ups: heal now runs before the Julia pre-render in the benchmark (a healed Julia page could otherwise never be re-frozen), Julia pages are skipped by the heal itself (the pre-render work list never repopulates them), the rmtree target is guarded against a stray _freeze/execute-results layout, refs must be quote/paren-anchored and cannot contain .. segments, the worker site_libs rsync fails loudly, and a large mass-drop emits a workflow warning.
Contributor
✅ QMD Format CheckNo new QMD tutorial files to check. |
- handle ./ and ../ page-relative site_libs prefixes (nested pages emit ../site_libs/...) and exclude JSON escape backslashes from the captured path - a stale Julia freeze page is now reported instead of silently skipped by the loop guard
Contributor
✅ QMD Format CheckNo new QMD tutorial files to check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Run 41446738921 (PR #428) took ~3 h because it fell back to a full cold build. Phase timings from the log:
The trigger:
Restore freeze cachefound nothing (⚠️ No freeze cache – full execution required), so every QMD re-executed.Root cause: the freeze cache keeps getting evicted
GitHub's repo-wide Actions cache limit is 10 GB; the repo is at ~10.8 GB. Caches saved on a PR merge ref are visible only to that same PR, so the biggest savers add dead weight:
pr-review.yml→setup-r-dependencieswith defaultcache: truesaves ~2.3 GB per run that no other workflow can reuse.setup-render-env→setup-renvsaves ~1 GB per PR run, again visible only to that PR.These evict the ~200 MB
_freezecache that every other build depends on. This recurs on every R version or renv.lock change (R 4.6.2 was released 2026-10-29 — yesterday — and the pool blew up today).Upstream research (why no Quarto fork)
freeze+ render-once-per-file into temp copies. Forking Quarto would duplicate the epic; the remaining wins here are in cache discipline and clone cost, not in Quarto itself.Changes
pr-review.yml—setup-r-dependenciesgetscache: false(stops 2.3 GB dead-weight entries).setup-render-env— onpull_request, the renv cache is restored viaactions/cache/restorewith the exact key formatsetup-renvuses (verified against upstream action.yaml) andsetup-renvruns withbypass-cache: trueso it never saves on PRs. Push/schedule runs keep the default save path, so main still replenishes the shared pool.render-site+ci-benchmark— the per-file worker clone now excludes./mcp(~109 MB of node assets no render touches) and local cache dirs, cutting clone I/O ~70% (≈150 GB less tar I/O across a 668-file cold run).Tradeoffs
renv.lockchange pay a full ~26 min install until the lockfile lands on main; a 2.3 GB dead-weight entry is worth more than the 200 MB freeze cache it evicts.ci-benchmark.yml(manual, non-PR), which stays representative of the push/schedule pipeline.Review
Independently reviewed (YAML validity, Actions semantics,
setup-renvkey parity, bash failure propagation, regression risks). A Julia pre-render parallelization was prototyped and rejected in review: concurrentquarto renderin the shared workspace re-introduces the documentedsite_libsrace that silently drops pages.Closes parts of #306.