From e5f4d1783b4b2c92d05b287b19b2a70470515195 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 19:31:08 +0000 Subject: [PATCH] feat(hygiene): detect folders no README documents, the inverse of a dead ref MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The refs tier scanned one direction only — prose naming a target that no longer exists. The inverse never had a sense: a folder that exists, works, and has every reference resolving, but which its own parent README never names. A package added without touching that README is invisible to anyone browsing the folder list. `autolens_workspace`'s `interferometer/features/datacube` sat unlisted for three months while the user who requested it waited, and the audit that found it turned up thirteen more in the same repo (autolens_workspace#482). Added as a second finding kind in the existing refs tier rather than as a hard gate in PyAutoHands' check_navigator, whose docstring already records the decision that folder-list rules belong here, "where a false positive costs nothing", instead of being duplicated into two organs. That call holds on the evidence: six of the eleven workspace/HowTo repos carry pre-existing instances, so a gate would have turned them red on the next unrelated PR. Precision comes from the direction of travel. Candidates are real directories read off the filesystem, so the "is this token a path or a parameter name?" ambiguity that forces the quorum heuristic on structure-list bullets cannot arise, and none is needed. The mention test is deliberately permissive on the other axis — a word-boundary search of the whole README — so bold bullets, trailing slashes and plain prose all count; only a folder named nowhere at all is reported. Directories without example content are skipped. `Finding` gains a defaulted `kind`; hygiene.sh consumes only --summary and passes --json-row through, so the row schema change is additive. Dead-reference output is unchanged (9 findings before and after across the local checkouts). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015zxc8vHNQ6FPqsRJaT8tGk --- agents/conductors/hygiene/AGENTS.md | 4 +- agents/conductors/hygiene/_hygiene_refs.py | 80 +++++++++++++++++++++- agents/conductors/hygiene/hygiene.sh | 15 ++-- 3 files changed, 89 insertions(+), 10 deletions(-) diff --git a/agents/conductors/hygiene/AGENTS.md b/agents/conductors/hygiene/AGENTS.md index a43b0f6..a260ac4 100644 --- a/agents/conductors/hygiene/AGENTS.md +++ b/agents/conductors/hygiene/AGENTS.md @@ -78,7 +78,7 @@ names — and they keep reporting even when the repo-array modes are `unscanned` | `docs` | `docs/api/*.rst` + `currentmodule` counts across every managed repo shipping a `docs/api/` tree (**surface**) | `/audit_docs` (Heart, imports) | | `crlf` | executable scripts (`.sh` + shebang-`755` `.py`) with CRLF — the shebang breaks on Linux/HPC (**debris**, the ranked count); plain `.py` CRLF is reported separately as *cosmetic* (Python reads it fine — don't mass-normalise) | `/refactor` + `.gitattributes eol=lf` | | `docstrings` | consecutive module-level triple-quoted expressions separated only by whitespace in user-facing `*_workspace` and `HowTo*` root `*.py` entry scripts and `scripts/**/*.py` files (**finding**) | `/refactor` (mechanically merge each confirmed boundary) | -| `refs` | file/folder references in user-facing `*_workspace` and `HowTo*` prose (`scripts/**/*.py` docstrings + comments, every `scripts/**/README.md` and `config/**/README.md`, and the top-level README) whose target no longer exists — restructure debt no health sweep can see, since the scripts still run (**finding**). Covers the README idioms a `scripts/`-anchored matcher cannot see: structure-list bullets (``- `slam_pipeline`: ``), slash-less relative folder paths (`data_preparation/imaging`), and config YAML names | `/refactor` (re-point each reference; judge the intended target) | +| `refs` | file/folder references in user-facing `*_workspace` and `HowTo*` prose (`scripts/**/*.py` docstrings + comments, every `scripts/**/README.md` and `config/**/README.md`, and the top-level README) whose target no longer exists — restructure debt no health sweep can see, since the scripts still run (**finding**). Covers the README idioms a `scripts/`-anchored matcher cannot see: structure-list bullets (``- `slam_pipeline`: ``), slash-less relative folder paths (`data_preparation/imaging`), and config YAML names. Scans the **inverse direction** too: a folder that exists but whose own parent README never names it — a package can ship fully working, with every reference resolving, and still be invisible to a reader browsing the folder list (`interferometer/features/datacube` sat unlisted for three months) | `/refactor` (re-point each dead reference; add an entry for each undocumented folder, sourced from that folder's own README or a script docstring) | | `optdeps` | smoke-listed workspace scripts that construct an optional-dependency-gated API (`TransformerNUFFT` → `nufftax`) without the house `find_spec` skip guard, so they hard-fail the CI matrices that omit the extras (**finding**). AST-confirmed — prose mentions don't count; scripts outside `smoke_tests.txt` are never flagged | `/refactor` (add the skip guard) | | `extras` | the complement of `optdeps`: an optional dependency a library **declares** (in the `[optional]` extra `mode=release` installs) that the `workspace-validation.yml` **`mode=smoke`** leg never installs (**finding**). The extras chain only reaches each library's own `[jax]`, never a sibling's `[optional]`, so those need hand-adding and silently drift — the symptom is a script red in smoke and **green in release** | `/bug` (add the install; fix the install set, **never** the script) | | `config` | library `config/*.yaml` keys missing from the matching workspace config — recursive diff (**surface**) | `/refactor` (mirror keys) | @@ -97,7 +97,7 @@ pyauto-brain hygiene deps # dependency-cap surface → /dep_audit pyauto-brain hygiene docs # API-docs surface → /audit_docs pyauto-brain hygiene crlf # CRLF .py files → /refactor pyauto-brain hygiene docstrings # adjacent top-level documentation → /refactor -pyauto-brain hygiene refs # dead internal references in workspace prose → /refactor +pyauto-brain hygiene refs # folder-list drift in workspace prose → /refactor pyauto-brain hygiene optdeps # smoke-listed scripts missing an optional-dep skip guard → /refactor pyauto-brain hygiene extras # optional deps the smoke CI leg never installs → /bug pyauto-brain hygiene config # library→workspace config drift → /refactor diff --git a/agents/conductors/hygiene/_hygiene_refs.py b/agents/conductors/hygiene/_hygiene_refs.py index acba3b2..945e977 100644 --- a/agents/conductors/hygiene/_hygiene_refs.py +++ b/agents/conductors/hygiene/_hygiene_refs.py @@ -1,5 +1,14 @@ #!/usr/bin/env python3 -"""Read-only scanner for DEAD INTERNAL REFERENCES in workspace prose. +"""Read-only scanner for FOLDER-LIST DRIFT in workspace prose. + +Two directions of the same defect, scanned together: + +* **dead references** — prose names a file or folder that no longer exists; +* **undocumented folders** — a folder exists that its own parent README never + names (see ``documented_directories``). + +The rest of this docstring describes the dead-reference direction, which is the +older and more intricate of the two. Workspace/HowTo scripts document themselves by pointing at their siblings — "see the ``modeling/start_here.ipynb`` example", "checkout @@ -173,6 +182,7 @@ class Finding: file: str line: int reference: str + kind: str = "dead" def canonical(path: str) -> str: @@ -527,6 +537,65 @@ def findings_in_file( return sorted(findings, key=lambda finding: (finding.line, finding.reference)) +def documented_directories(repository: Path) -> list[Finding]: + """Report example folders that their own parent README never mentions. + + The inverse of the dead-reference scan above: that one starts from prose and + asks whether the target exists, this one starts from what exists and asks + whether any prose names it. Both directions are the same defect seen from + two ends — a folder list that has drifted from the tree — but they fail + apart. A package added without touching its parent README leaves every + reference resolving perfectly while the folder is invisible to a reader + browsing the list. ``interferometer/features/datacube`` sat unlisted that + way for three months (autolens_workspace#482), and the audit that found it + turned up thirteen more in the same repo. + + Precision comes from the direction of travel: the candidates are real + directories read off the filesystem, so the "is this token a path or a + parameter name?" ambiguity that constrains the structure-list rule above + cannot arise here, and no quorum heuristic is needed. The mention test is + deliberately permissive in the other axis — a bare word-boundary search of + the whole README — so bold bullets (``- **`name`**``), trailing slashes + (``- `name/```) and plain prose all count as documenting the folder. Only a + folder named *nowhere at all* is reported. + + A directory is a candidate only if it carries example content (a script, + notebook, or its own README); asset and output directories are skipped. + """ + findings: list[Finding] = [] + scripts = repository / "scripts" + if not scripts.is_dir(): + return findings + for readme in sorted(scripts.rglob("README.md")): + try: + text = readme.read_text(encoding="utf-8", errors="replace") + except OSError: + continue + for child in sorted(readme.parent.iterdir()): + if not child.is_dir() or child.name in PRUNED_DIRECTORIES: + continue + if child.name.startswith((".", "__")): + continue + has_content = any( + any(child.rglob(pattern)) + for pattern in ("*.py", "*.ipynb", "README.md") + ) + if not has_content: + continue + if re.search(rf"\b{re.escape(child.name)}\b", text): + continue + findings.append( + Finding( + repo=repository.name, + file=readme.relative_to(repository).as_posix(), + line=1, + reference=f"{child.name}/ (folder exists, README never names it)", + kind="orphan", + ) + ) + return findings + + def scan(root: Path) -> tuple[list[Finding], int, int]: resolver = Resolver(root) repositories = repository_paths(root) @@ -535,6 +604,7 @@ def scan(root: Path) -> tuple[list[Finding], int, int]: resolver.indexes.setdefault(repository.name, RepositoryIndex(repository)) for path in scanned_files(repository): findings.extend(findings_in_file(resolver, repository, path)) + findings.extend(documented_directories(repository)) return findings, len(repositories), resolver.suppressed @@ -542,8 +612,11 @@ def summary_for(findings: list[Finding], repository_count: int, skipped: int) -> file_count = len({(finding.repo, finding.file) for finding in findings}) affected = len({finding.repo for finding in findings}) file_label = "file" if file_count == 1 else "files" + dead = sum(1 for finding in findings if finding.kind == "dead") + orphans = sum(1 for finding in findings if finding.kind == "orphan") return ( - f"{len(findings)} dead internal references in {file_count} {file_label} " + f"{len(findings)} folder-list defects ({dead} dead references, " + f"{orphans} undocumented folders) in {file_count} {file_label} " f"across {affected}/{repository_count} repos; " f"{skipped} unresolvable refs skipped" ) @@ -569,7 +642,8 @@ def render_human(row: dict) -> None: if finding["repo"] != repo: repo = finding["repo"] print(f" {repo}:") - print(f" {finding['file']}:{finding['line']} -> {finding['reference']}") + arrow = "!!" if finding.get("kind") == "orphan" else "->" + print(f" {finding['file']}:{finding['line']} {arrow} {finding['reference']}") def main() -> int: diff --git a/agents/conductors/hygiene/hygiene.sh b/agents/conductors/hygiene/hygiene.sh index f91113d..3cd8939 100755 --- a/agents/conductors/hygiene/hygiene.sh +++ b/agents/conductors/hygiene/hygiene.sh @@ -16,7 +16,7 @@ # noise -> /cli_noise_clean (Heart) deps -> /dep_audit (Heart) # docs -> /audit_docs (Heart) packaging -> clean_slate.sh (Brain) # docstrings -> /refactor (exact findings; Hygiene remains read-only) -# refs -> /refactor (dead internal references in workspace prose) +# refs -> /refactor (folder-list drift in workspace prose: dead refs + undocumented folders) # optdeps -> /refactor (smoke-listed scripts missing an optional-dep skip guard) # extras -> /bug (optional deps a library declares that the smoke CI leg never installs) # The three Heart skills are read-only observation skills — measurement lives in @@ -34,7 +34,7 @@ # hygiene.sh docs # API-docs pre-scan -> /audit_docs # hygiene.sh crlf # executable scripts w/ CRLF break on HPC (+ cosmetic .py) -> /refactor # hygiene.sh docstrings # adjacent top-level script documentation -> /refactor -# hygiene.sh refs # dead internal references in workspace prose -> /refactor +# hygiene.sh refs # folder-list drift in workspace prose -> /refactor # hygiene.sh optdeps # smoke-listed scripts w/ a gated API but no skip guard -> /refactor # hygiene.sh extras # optional deps declared by a library but missing from the smoke CI install -> /bug # hygiene.sh config # library config keys missing downstream + orphan config files -> /refactor @@ -294,7 +294,9 @@ prescan_extras() { # scripts still run, so no health sweep can see it. The stdlib helper resolves # each reference against the checked-out repos (scripts/ and notebooks/ are one # namespace) and holds precision with documented suppressions; this compact form -# feeds the default ranked worklist. +# feeds the default ranked worklist. The same helper scans the inverse direction +# — a folder that exists but whose own parent README never names it, which is how +# a package can ship invisible to every reader browsing the folder list. prescan_refs() { python3 "$HERE/_hygiene_refs.py" --root "$ROOT" --summary } @@ -713,12 +715,15 @@ if [[ "$mode" == "docstrings" ]]; then echo echo "→ route the mechanical merges to /refactor; Hygiene never edits source." elif [[ "$mode" == "refs" ]]; then - echo "Dead internal references in workspace prose (read-only scan):" + echo "Folder-list drift in workspace prose (read-only scan):" python3 "$HERE/_hygiene_refs.py" --root "$ROOT" echo - echo "→ route the re-points to /refactor; Hygiene never edits source. Each finding is" + echo "→ route the re-points to /refactor; Hygiene never edits source. A '->' finding is" echo " the reference AS WRITTEN — judge the intended target before repointing (a moved" echo " file, a file that became a directory, or a reference meant for a sibling repo)." + echo " A '!!' finding is the inverse: the folder exists and the README never names it," + echo " so the fix is a new entry describing it, sourced from that folder's own README" + echo " or a script docstring — never inferred from the folder name." elif [[ "$mode" == "optdeps" ]]; then echo "Smoke-listed scripts missing an optional-dependency skip guard (read-only scan):" python3 "$HERE/_hygiene_optdeps.py" --root "$ROOT"