Skip to content

feat(profiling): add a compile-time axis to campaign coverage - #219

Merged
Jammy2211 merged 3 commits into
mainfrom
feature/compile-axis-campaign-coverage
Aug 10, 2026
Merged

feat(profiling): add a compile-time axis to campaign coverage#219
Jammy2211 merged 3 commits into
mainfrom
feature/compile-axis-campaign-coverage

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Closes #218. Phase 1 of 3 of the compile-axis arc.

What this fixes

The Profiling Agent could not see the compile-time corpus autolens_profiling is
already producing. AGENTS.md listed "JAX compilation-time profiling of likelihood
functions" under Future modes, _profiling.py had zero occurrences of "compile",
and all three modes read only results/runtime/ — while scripts/misc/jax_compile/
held 93 committed probe records that nothing cross-referenced against the science
grid.

campaign --axis compile now answers how much of the grid has compile data on a tier:

Cell/transform runs:  11 done · 136 missing (21 grid cells × 7 transforms)
Off-grid records (real measurements, not grid cells):
  datacube_img/mge/hst: 4 · imaging/delaunay_matern/hst: 12 · imaging/knn/hst: 11
Other hardware (neither tier):
  local_gpu_NVIDIA_GeForce_RTX_2060_with_Max-Q_Design: 2 record(s)
Malformed records:    4

That is the number the arc's other two phases are justified by: 2 of 21 grid cells
have any compile coverage, hst only, with no interferometer, datacube, jwst or ao
row anywhere.

Design decisions worth review

Records are placed by their own fields, not by path. Results are filed under
<hardware>/<model_type>, which drops the dataset class and instrument entirely, so
placement uses the in-record (dataset_class, model_type, instrument). A test files an
interferometer record inside mge.json to pin this.

The tier map is deliberately not TIER_CONFIGS. That map keys off sweep config
names which fold precision into the name (local_cpu_fp64 / local_cpu_mp), whereas a
compile record carries a raw hardware string plus a separate mixed_precision
bool. Reusing it would mis-bucket every row. A test pins that an fp64 and an mp record
on one machine are one cell/transform run.

other is a real answer, not a fallback. The corpus holds RTX-2060 rows belonging
to neither tier; folding them into local would report coverage on hardware nobody
asked about. Same reasoning for the off-grid bucket — knn, delaunay_matern and
the datacube_img* multi-band classes are genuine Prodigy-census and multi-band
measurements, so they are neither grid coverage nor noise.

Malformed records are surfaced, not skipped — with file and index, exposing the 4
that carry hardware, dataset_class and instrument all null.

Coverage only; it never compares two timings. jax_compile/README.md records the
first measurements being wrong by up to (851 s vs 117 s for the same compile)
purely from host load, because XLA compiles on the host cores. Rows are comparable only
within (hardware, jax_version, mixed_precision, cache state), and comparison waits on
the pins in phase 2. A test asserts no timing value can leak into a coverage decision.

ingest / triage reject --axis compile with exit 5 rather than ignoring the
flag, so a compile flag can never silently return a runtime answer.

The transform axis is read from the workspace, via the same ast route load_grid
uses for CELLS, so the Brain cannot drift from probe.py's TRANSFORMS.

Verification

  • 15 new tests in tests/test_profiling_conductor.py. Profiling was the only
    conductor without a test file, so the "runtime axis unchanged" requirement had nothing
    to assert against — the file is part of the work, not a bonus. Hermetic synthetic
    workspace fixtures on the test_hygiene_conductor.py pattern; stdlib + pytest only.
  • Runtime axis verified byte-identical against the real workspace (stashed diff on
    both campaign and ingest), plus tests asserting the runtime decision shape gains no
    axis key and that the two tier vocabularies never cross.
  • Full suite: 304 passed. Two test_skill_install.py failures reproduce on clean
    main in this environment (they assert Codex-skill install behaviour that a
    web-github / ci-only environment reports differently) and are untouched by this
    change.

Not in scope

Any judgement about whether a compile time is good (phase 3), any pin/baseline/
dashboard (phase 2), and executing probe.py — the agent reasons and emits a plan,
never runs the workspace.

Follow-ups

  • Phase 2compile_warm_baseline_dashboard.md: warm rows are not machine-
    identifiable today (warmness lives only in a free-text tag, ~40 ad-hoc values, and
    cache_dir is non-empty on cold rows too), so probe.py needs an explicit
    cache_state before any warm-compile tracking is possible.
  • Phase 3compile_axis_triage_drift.md: classify drift, and close the arc by
    moving compile-time profiling out of Future modes.

Generated by Claude Code

claude added 3 commits August 10, 2026 13:12
The Profiling Agent could not see the compile-time corpus autolens_profiling
is already producing. AGENTS.md listed "JAX compilation-time profiling of
likelihood functions" under Future modes, _profiling.py had no occurrence of
"compile", and all three modes read only results/runtime/ -- while
scripts/misc/jax_compile/ held 93 committed probe records that nothing
cross-referenced against the science grid.

campaign --axis compile answers how much of the grid has compile data on a
tier. Records are placed by their own (dataset_class, model_type, instrument)
rather than by path, since results are filed under <hardware>/<model_type> and
the path drops the class and instrument entirely.

Tier mapping is deliberately NOT TIER_CONFIGS. That map keys off sweep config
names which fold precision into the name (local_cpu_fp64 / local_cpu_mp),
whereas a compile record carries a raw hardware string plus a separate
mixed_precision bool; reusing it would mis-bucket every row. "other" is a real
answer rather than a fallback -- the corpus holds RTX-2060 rows belonging to
neither tier.

Off-grid records (knn, delaunay_matern, the datacube_img* multi-band classes)
and non-tier hardware get their own buckets: real measurements that are
neither grid coverage nor noise. Malformed records are reported with file and
index rather than skipped, surfacing the 4 that carry null hardware, class and
instrument.

The mode reports coverage only and never compares two timings. Compile timings
are host-load-sensitive -- jax_compile/README.md records the first
measurements being wrong by up to 7x (851s vs 117s for the same compile)
because XLA compiles on the host cores -- so rows are comparable only within
(hardware, jax_version, mixed_precision, cache state). Comparison waits on the
pins in phase 2.

ingest and triage reject --axis compile with exit 5 rather than ignoring it,
so a compile flag can never silently return a runtime answer.

The transform axis is read from probe.py's TRANSFORMS literal via the same ast
route load_grid uses for CELLS, so the Brain cannot drift from the instrument.

Adds tests/test_profiling_conductor.py -- profiling was the only conductor
without a test file, so the "runtime axis unchanged" requirement had nothing
to assert against. Hermetic: synthetic workspace fixtures, no real checkout.
Runtime-axis output verified byte-identical against the real workspace.

Phase 1 of 3; PyAutoMind draft/feature/profiling/.

Closes #218

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
…atch

The first glob never matched (jax_compile/ has no hpc/ dir) so the fallback
always won, but had it matched it would have printed hpc/batch_gpu/<name>
for a file that lives elsewhere. Use the same path the runtime campaign does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
jax_compile/ hosts export_probe.py and trace_profile.py, which append their
own schema into the SAME results/<hardware>/ tree probe.py writes to. Their
records have no hardware/dataset_class/instrument because they are a different
record kind, not because they are corrupt -- so the 4 the mode was reporting
as malformed would have sent someone to fix two files that work correctly.

Split the two: missing the whole identity triple is a sibling instrument
(reported per file, under its own bucket); missing only some key fields is
genuine corruption and stays malformed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
@Jammy2211
Jammy2211 merged commit cd79005 into main Aug 10, 2026
2 checks passed
Jammy2211 pushed a commit that referenced this pull request Aug 10, 2026
# Conflicts:
#	agents/conductors/profiling/AGENTS.md
#	agents/conductors/profiling/_profiling.py
#	tests/test_profiling_conductor.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(profiling): add a compile-time axis to campaign coverage

2 participants