diff --git a/agents/conductors/hygiene/AGENTS.md b/agents/conductors/hygiene/AGENTS.md index a43b0f6..a260ac4 100644 --- a/agents/conductors/hygiene/AGENTS.md +++ b/agents/conductors/hygiene/AGENTS.md @@ -78,7 +78,7 @@ names — and they keep reporting even when the repo-array modes are `unscanned` | `docs` | `docs/api/*.rst` + `currentmodule` counts across every managed repo shipping a `docs/api/` tree (**surface**) | `/audit_docs` (Heart, imports) | | `crlf` | executable scripts (`.sh` + shebang-`755` `.py`) with CRLF — the shebang breaks on Linux/HPC (**debris**, the ranked count); plain `.py` CRLF is reported separately as *cosmetic* (Python reads it fine — don't mass-normalise) | `/refactor` + `.gitattributes eol=lf` | | `docstrings` | consecutive module-level triple-quoted expressions separated only by whitespace in user-facing `*_workspace` and `HowTo*` root `*.py` entry scripts and `scripts/**/*.py` files (**finding**) | `/refactor` (mechanically merge each confirmed boundary) | -| `refs` | file/folder references in user-facing `*_workspace` and `HowTo*` prose (`scripts/**/*.py` docstrings + comments, every `scripts/**/README.md` and `config/**/README.md`, and the top-level README) whose target no longer exists — restructure debt no health sweep can see, since the scripts still run (**finding**). Covers the README idioms a `scripts/`-anchored matcher cannot see: structure-list bullets (``- `slam_pipeline`: ``), slash-less relative folder paths (`data_preparation/imaging`), and config YAML names | `/refactor` (re-point each reference; judge the intended target) | +| `refs` | file/folder references in user-facing `*_workspace` and `HowTo*` prose (`scripts/**/*.py` docstrings + comments, every `scripts/**/README.md` and `config/**/README.md`, and the top-level README) whose target no longer exists — restructure debt no health sweep can see, since the scripts still run (**finding**). Covers the README idioms a `scripts/`-anchored matcher cannot see: structure-list bullets (``- `slam_pipeline`: ``), slash-less relative folder paths (`data_preparation/imaging`), and config YAML names. Scans the **inverse direction** too: a folder that exists but whose own parent README never names it — a package can ship fully working, with every reference resolving, and still be invisible to a reader browsing the folder list (`interferometer/features/datacube` sat unlisted for three months) | `/refactor` (re-point each dead reference; add an entry for each undocumented folder, sourced from that folder's own README or a script docstring) | | `optdeps` | smoke-listed workspace scripts that construct an optional-dependency-gated API (`TransformerNUFFT` → `nufftax`) without the house `find_spec` skip guard, so they hard-fail the CI matrices that omit the extras (**finding**). AST-confirmed — prose mentions don't count; scripts outside `smoke_tests.txt` are never flagged | `/refactor` (add the skip guard) | | `extras` | the complement of `optdeps`: an optional dependency a library **declares** (in the `[optional]` extra `mode=release` installs) that the `workspace-validation.yml` **`mode=smoke`** leg never installs (**finding**). The extras chain only reaches each library's own `[jax]`, never a sibling's `[optional]`, so those need hand-adding and silently drift — the symptom is a script red in smoke and **green in release** | `/bug` (add the install; fix the install set, **never** the script) | | `config` | library `config/*.yaml` keys missing from the matching workspace config — recursive diff (**surface**) | `/refactor` (mirror keys) | @@ -97,7 +97,7 @@ pyauto-brain hygiene deps # dependency-cap surface → /dep_audit pyauto-brain hygiene docs # API-docs surface → /audit_docs pyauto-brain hygiene crlf # CRLF .py files → /refactor pyauto-brain hygiene docstrings # adjacent top-level documentation → /refactor -pyauto-brain hygiene refs # dead internal references in workspace prose → /refactor +pyauto-brain hygiene refs # folder-list drift in workspace prose → /refactor pyauto-brain hygiene optdeps # smoke-listed scripts missing an optional-dep skip guard → /refactor pyauto-brain hygiene extras # optional deps the smoke CI leg never installs → /bug pyauto-brain hygiene config # library→workspace config drift → /refactor diff --git a/agents/conductors/hygiene/_hygiene_refs.py b/agents/conductors/hygiene/_hygiene_refs.py index acba3b2..945e977 100644 --- a/agents/conductors/hygiene/_hygiene_refs.py +++ b/agents/conductors/hygiene/_hygiene_refs.py @@ -1,5 +1,14 @@ #!/usr/bin/env python3 -"""Read-only scanner for DEAD INTERNAL REFERENCES in workspace prose. +"""Read-only scanner for FOLDER-LIST DRIFT in workspace prose. + +Two directions of the same defect, scanned together: + +* **dead references** — prose names a file or folder that no longer exists; +* **undocumented folders** — a folder exists that its own parent README never + names (see ``documented_directories``). + +The rest of this docstring describes the dead-reference direction, which is the +older and more intricate of the two. Workspace/HowTo scripts document themselves by pointing at their siblings — "see the ``modeling/start_here.ipynb`` example", "checkout @@ -173,6 +182,7 @@ class Finding: file: str line: int reference: str + kind: str = "dead" def canonical(path: str) -> str: @@ -527,6 +537,65 @@ def findings_in_file( return sorted(findings, key=lambda finding: (finding.line, finding.reference)) +def documented_directories(repository: Path) -> list[Finding]: + """Report example folders that their own parent README never mentions. + + The inverse of the dead-reference scan above: that one starts from prose and + asks whether the target exists, this one starts from what exists and asks + whether any prose names it. Both directions are the same defect seen from + two ends — a folder list that has drifted from the tree — but they fail + apart. A package added without touching its parent README leaves every + reference resolving perfectly while the folder is invisible to a reader + browsing the list. ``interferometer/features/datacube`` sat unlisted that + way for three months (autolens_workspace#482), and the audit that found it + turned up thirteen more in the same repo. + + Precision comes from the direction of travel: the candidates are real + directories read off the filesystem, so the "is this token a path or a + parameter name?" ambiguity that constrains the structure-list rule above + cannot arise here, and no quorum heuristic is needed. The mention test is + deliberately permissive in the other axis — a bare word-boundary search of + the whole README — so bold bullets (``- **`name`**``), trailing slashes + (``- `name/```) and plain prose all count as documenting the folder. Only a + folder named *nowhere at all* is reported. + + A directory is a candidate only if it carries example content (a script, + notebook, or its own README); asset and output directories are skipped. + """ + findings: list[Finding] = [] + scripts = repository / "scripts" + if not scripts.is_dir(): + return findings + for readme in sorted(scripts.rglob("README.md")): + try: + text = readme.read_text(encoding="utf-8", errors="replace") + except OSError: + continue + for child in sorted(readme.parent.iterdir()): + if not child.is_dir() or child.name in PRUNED_DIRECTORIES: + continue + if child.name.startswith((".", "__")): + continue + has_content = any( + any(child.rglob(pattern)) + for pattern in ("*.py", "*.ipynb", "README.md") + ) + if not has_content: + continue + if re.search(rf"\b{re.escape(child.name)}\b", text): + continue + findings.append( + Finding( + repo=repository.name, + file=readme.relative_to(repository).as_posix(), + line=1, + reference=f"{child.name}/ (folder exists, README never names it)", + kind="orphan", + ) + ) + return findings + + def scan(root: Path) -> tuple[list[Finding], int, int]: resolver = Resolver(root) repositories = repository_paths(root) @@ -535,6 +604,7 @@ def scan(root: Path) -> tuple[list[Finding], int, int]: resolver.indexes.setdefault(repository.name, RepositoryIndex(repository)) for path in scanned_files(repository): findings.extend(findings_in_file(resolver, repository, path)) + findings.extend(documented_directories(repository)) return findings, len(repositories), resolver.suppressed @@ -542,8 +612,11 @@ def summary_for(findings: list[Finding], repository_count: int, skipped: int) -> file_count = len({(finding.repo, finding.file) for finding in findings}) affected = len({finding.repo for finding in findings}) file_label = "file" if file_count == 1 else "files" + dead = sum(1 for finding in findings if finding.kind == "dead") + orphans = sum(1 for finding in findings if finding.kind == "orphan") return ( - f"{len(findings)} dead internal references in {file_count} {file_label} " + f"{len(findings)} folder-list defects ({dead} dead references, " + f"{orphans} undocumented folders) in {file_count} {file_label} " f"across {affected}/{repository_count} repos; " f"{skipped} unresolvable refs skipped" ) @@ -569,7 +642,8 @@ def render_human(row: dict) -> None: if finding["repo"] != repo: repo = finding["repo"] print(f" {repo}:") - print(f" {finding['file']}:{finding['line']} -> {finding['reference']}") + arrow = "!!" if finding.get("kind") == "orphan" else "->" + print(f" {finding['file']}:{finding['line']} {arrow} {finding['reference']}") def main() -> int: diff --git a/agents/conductors/hygiene/hygiene.sh b/agents/conductors/hygiene/hygiene.sh index f91113d..3cd8939 100755 --- a/agents/conductors/hygiene/hygiene.sh +++ b/agents/conductors/hygiene/hygiene.sh @@ -16,7 +16,7 @@ # noise -> /cli_noise_clean (Heart) deps -> /dep_audit (Heart) # docs -> /audit_docs (Heart) packaging -> clean_slate.sh (Brain) # docstrings -> /refactor (exact findings; Hygiene remains read-only) -# refs -> /refactor (dead internal references in workspace prose) +# refs -> /refactor (folder-list drift in workspace prose: dead refs + undocumented folders) # optdeps -> /refactor (smoke-listed scripts missing an optional-dep skip guard) # extras -> /bug (optional deps a library declares that the smoke CI leg never installs) # The three Heart skills are read-only observation skills — measurement lives in @@ -34,7 +34,7 @@ # hygiene.sh docs # API-docs pre-scan -> /audit_docs # hygiene.sh crlf # executable scripts w/ CRLF break on HPC (+ cosmetic .py) -> /refactor # hygiene.sh docstrings # adjacent top-level script documentation -> /refactor -# hygiene.sh refs # dead internal references in workspace prose -> /refactor +# hygiene.sh refs # folder-list drift in workspace prose -> /refactor # hygiene.sh optdeps # smoke-listed scripts w/ a gated API but no skip guard -> /refactor # hygiene.sh extras # optional deps declared by a library but missing from the smoke CI install -> /bug # hygiene.sh config # library config keys missing downstream + orphan config files -> /refactor @@ -294,7 +294,9 @@ prescan_extras() { # scripts still run, so no health sweep can see it. The stdlib helper resolves # each reference against the checked-out repos (scripts/ and notebooks/ are one # namespace) and holds precision with documented suppressions; this compact form -# feeds the default ranked worklist. +# feeds the default ranked worklist. The same helper scans the inverse direction +# — a folder that exists but whose own parent README never names it, which is how +# a package can ship invisible to every reader browsing the folder list. prescan_refs() { python3 "$HERE/_hygiene_refs.py" --root "$ROOT" --summary } @@ -713,12 +715,15 @@ if [[ "$mode" == "docstrings" ]]; then echo echo "→ route the mechanical merges to /refactor; Hygiene never edits source." elif [[ "$mode" == "refs" ]]; then - echo "Dead internal references in workspace prose (read-only scan):" + echo "Folder-list drift in workspace prose (read-only scan):" python3 "$HERE/_hygiene_refs.py" --root "$ROOT" echo - echo "→ route the re-points to /refactor; Hygiene never edits source. Each finding is" + echo "→ route the re-points to /refactor; Hygiene never edits source. A '->' finding is" echo " the reference AS WRITTEN — judge the intended target before repointing (a moved" echo " file, a file that became a directory, or a reference meant for a sibling repo)." + echo " A '!!' finding is the inverse: the folder exists and the README never names it," + echo " so the fix is a new entry describing it, sourced from that folder's own README" + echo " or a script docstring — never inferred from the folder name." elif [[ "$mode" == "optdeps" ]]; then echo "Smoke-listed scripts missing an optional-dependency skip guard (read-only scan):" python3 "$HERE/_hygiene_optdeps.py" --root "$ROOT"