Skip to content

Widen site-path gate and scrub internal paths (refs #2) - #16

Open
n-getty wants to merge 1 commit into
mainfrom
scrub/widen-site-path-gate
Open

n-getty wants to merge 1 commit into
mainfrom
scrub/widen-site-path-gate

Conversation

@n-getty

@n-getty n-getty commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Description

Refs #2 (open-source readiness). A review comment found /lus/flare/projects/ModCon/<user>/BaseMM_PRISM in docs/skills/prism-multi-agent-workflow/SKILL.md. Earlier scrubs missed it because the hygiene gate could not see it, so this PR fixes the leak and widens the gate. It does not close #2 and leaves a baselined allowlist (below).

Gate (tools/ci/check_site_paths.py)

  • Now scans experiments/, scaling-study/, requirements/, .github/ and root-level tracked files. Previously these were outside SCANNED_PREFIXES; experiments/ alone held ~22 paths.
  • Also matches the grand, gpfs and lustre roots. /soft/... is deliberately not matched (public system software).
  • New identity scan: ngetty, abalaji, smadireddy, BaseMM_PRISM count on the same ratchet (not double-counted inside a matched path). Exempt: CITATION.cff, pyproject.toml, .mailmap, SECURITY.md, CODE_OF_CONDUCT.md (deliberate author/contact emails), and the gate and its test.
  • 4 new tests in tests/test_site_path_gate.py.

Scrubbed

  • The leak in the review comment (prism-multi-agent-workflow/SKILL.md) and the stale BaseMM_PRISM.git clone hint in src/_repo_paths.py:40.
  • bioreason/GRPO scripts, smoke_test_grpo.sh, setup_daos_models.sh, run_agpt2b_interactive.sh, run_composite_interactive.sh, convert_parquet_to_webdataset.py, launch_polaris.py help, KEGG docs, tests/test_isoflop_launcher_flag.py.
  • experiments/prism_designs.yaml: removed the /raid/<user> resume path, the CALVIN calvin_root and the pinned AuroraGPT backbone_id overrides.
  • New shell variables PRISM_CONDA_ENV and PRISM_REPO_ROOT (documented in .env.template and docs/platforms/site_paths.md; scripts fail with set <NAME> if unset).

Removed

experiments/stage_a_round1_olmo3_1b.yaml, ablation_catalog.yaml, qwen3_siglip_smokes.yaml: internal study records with no code dependents.

Behavior changes to review

  • The AuroraGPT-2B design no longer pins global_step140300; it uses PRISM_AURORAGPT_2B_CHECKPOINT.
  • tools/universal_evaluator.py SciTS validation raises ValueError if neither PRISM_VAL_SHARDS_DIR nor PRISM_DATA_ROOT is set (was a personal path).
  • scripts/setup_daos_models.sh needs MODEL_ROOT or PRISM_HF_HUB; MODEL_ROOT_ALT is an optional override.
  • Interactive launch scripts need PRISM_REPO_ROOT; the AGPT2B one needs PRISM_AURORAGPT_2B_CHECKPOINT.

State of the allowlist

tools/ci/site_path_allowlist.json: 68 files, 197 paths (main had 73 files / 236 paths; the wider scan and identity check add baselined entries, and the scrubs above remove more than they add, for a net drop). The ratchet only goes down.

By area: docs 56, tools 52, applications 37, scripts 34, tests 15, .github 1, root 1, src 1. Largest: scripts/setup_daos_container.sh (26), applications/vision_language/hydrate_pixmo_pointing.py (10), docs/platforms/daos_setup.md (10).

Deliberately still present:

  • NOTICE and .github/dependabot.yml reference the internal repo as a tracking link; several docs link github.com/AI-ModCon/BaseMM_PRISM/....
  • src/data/kegg_curation/entrez_client.py:28 sends BaseMM_PRISM-kegg_curation as the NCBI tool name; renaming changes what NCBI sees, so left for a decision.
  • Stage A tooling (stage_a_round1_batch.sh, hold_4node_debug_scaling.sh, build_stage_a_plan.py) and the kept study files LLM_timeseries_scaling.yaml, qwen3_siglip_scaling.yaml, qwen3_siglip_2node_search.yaml carry TODO(release): keep and scrub, or remove.

Type of Change

  • Documentation update
  • Infrastructure, CI, or tooling

Compatibility

  • launcher flags (no flags changed; some scripts now require PRISM_* env vars, see above)
  • Hydra config keys, recipe IDs and src.* paths are unchanged. Three experiment catalogs were removed; nothing in code or tests loads them.

How Has This Been Tested?

  • python3.11 tools/ci/check_site_paths.py → OK: 68 allowlisted files, 197 hardcoded site paths, none new
  • bash -n on every edited shell script; py_compile on edited Python; yaml.safe_load on every experiments/*.yaml.
  • Not run: pytest (make test, tests/test_site_path_gate.py, test_docs_consistency.py, test_makefile_ci_parity.py) — no usable pytest on the HPC login node. Please rely on CI. Also not run: the edited scripts themselves, or make docs.
  • Platform tested on: Aurora login node (static checks only)

New Dependencies, Data, or Assets

None.

Checklist

  • I have performed a self-review of my own changes
  • No secrets, credentials, or absolute institutional paths are included (remaining ones are on the ratchet allowlist)

AI/LLM Assistance

  • AI/LLM tools were primarily used to generate code or artifacts in this PR
  • I have reviewed all such output line-by-line and understand it

Drafted with Claude Code; the human reviewer and author checks above are left for the humans.

Additional Context

Remaining items on #2 are outside this PR: baselined allowlist entries above, and the Stage A / study-file keep-or-remove decision.

Gate (tools/ci/check_site_paths.py): scan experiments/, scaling-study/,
requirements/, .github/ and root-level files; match grand/gpfs/lustre roots;
count internal identities (usernames, old repo name) on the same ratchet.

Scrub: bioreason/GRPO scripts, DAOS/interactive launch scripts,
universal_evaluator, convert_parquet_to_webdataset, prism_designs.yaml, KEGG
docs, the multi-agent skill and the _repo_paths clone hint now use PRISM_*
variables or placeholders. Remove three study-record experiment files; mark
the rest with TODO(release). Document PRISM_CONDA_ENV / PRISM_REPO_ROOT.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@n-getty
n-getty requested a review from a team as a code owner October 27, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Open-source readiness: PRISM v0.1.0

2 participants