feat(profiling): ingest --axis compile — warm-pin drift detection - #220
Merged
Conversation
The Profiling Agent could not see the compile-time corpus autolens_profiling is already producing. AGENTS.md listed "JAX compilation-time profiling of likelihood functions" under Future modes, _profiling.py had no occurrence of "compile", and all three modes read only results/runtime/ -- while scripts/misc/jax_compile/ held 93 committed probe records that nothing cross-referenced against the science grid. campaign --axis compile answers how much of the grid has compile data on a tier. Records are placed by their own (dataset_class, model_type, instrument) rather than by path, since results are filed under <hardware>/<model_type> and the path drops the class and instrument entirely. Tier mapping is deliberately NOT TIER_CONFIGS. That map keys off sweep config names which fold precision into the name (local_cpu_fp64 / local_cpu_mp), whereas a compile record carries a raw hardware string plus a separate mixed_precision bool; reusing it would mis-bucket every row. "other" is a real answer rather than a fallback -- the corpus holds RTX-2060 rows belonging to neither tier. Off-grid records (knn, delaunay_matern, the datacube_img* multi-band classes) and non-tier hardware get their own buckets: real measurements that are neither grid coverage nor noise. Malformed records are reported with file and index rather than skipped, surfacing the 4 that carry null hardware, class and instrument. The mode reports coverage only and never compares two timings. Compile timings are host-load-sensitive -- jax_compile/README.md records the first measurements being wrong by up to 7x (851s vs 117s for the same compile) because XLA compiles on the host cores -- so rows are comparable only within (hardware, jax_version, mixed_precision, cache state). Comparison waits on the pins in phase 2. ingest and triage reject --axis compile with exit 5 rather than ignoring it, so a compile flag can never silently return a runtime answer. The transform axis is read from probe.py's TRANSFORMS literal via the same ast route load_grid uses for CELLS, so the Brain cannot drift from the instrument. Adds tests/test_profiling_conductor.py -- profiling was the only conductor without a test file, so the "runtime axis unchanged" requirement had nothing to assert against. Hermetic: synthetic workspace fixtures, no real checkout. Runtime-axis output verified byte-identical against the real workspace. Phase 1 of 3; PyAutoMind draft/feature/profiling/. Closes #218 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
…atch The first glob never matched (jax_compile/ has no hpc/ dir) so the fallback always won, but had it matched it would have printed hpc/batch_gpu/<name> for a file that lives elsewhere. Use the same path the runtime campaign does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
jax_compile/ hosts export_probe.py and trace_profile.py, which append their own schema into the SAME results/<hardware>/ tree probe.py writes to. Their records have no hardware/dataset_class/instrument because they are a different record kind, not because they are corrupt -- so the 4 the mode was reporting as malformed would have sent someone to fix two files that work correctly. Split the two: missing the whole identity triple is a sibling instrument (reported per file, under its own bucket); missing only some key fields is genuine corruption and stays malformed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Completes the Brain side of phase 2, and is the surveillance the arc exists for: the persistent cache and --xla_gpu_autotune_level=0 are SETTINGS, so a config drift or an XLA_FLAGS clobber (the PyAutoNerves#127 failure that went undetected for two months) puts the worst case back with nothing failing. Reads the workspace's jax_compile/pins.json and reports warm rows that are unpinned or have drifted. Every comparison happens strictly inside one (hardware, hostname, jax_version, mixed_precision, cache_state); cross-key pairs are never a regression. A jax_version bump recompiles ONCE BY DESIGN, so it surfaces as a new unpinned key rather than as drift. Two corrections found by running it against the real corpus rather than trusting the design: 1. Rows PREDATING their pin are not drift. The first run flagged four, all of them July-16 measurements the July-28 pin had been chosen over -- i.e. it reported the improvement that set the pin as though it were a regression. Drift now requires a row newer than its pin. 2. That exposed the deeper flaw, fixed in the workspace: pins must be sticky. With "most recent wins", re-deriving pins after a cache regression would have quietly baked the regression in and the surveillance would report all-clear forever. Thresholds are deliberately generous and require BOTH gates -- >= 2.0x the pin AND >= 1.0s absolute. The ratio alone screams about sub-second cells where 100ms of jitter is 3x; the floor alone misses a cheap cell degrading by an order of magnitude. Host load alone has produced 7x errors in this corpus, and an alarm that cries wolf gets ignored. Against the real corpus: 25 pins, 0 drifted, 0 unpinned. 11 more tests, including a synthetic warm-reverting-to-cold row proving the alarm fires, each comparability field proving it does not fire across the key, and a guard that the Brain's mirrored key definition matches the workspace's pins.py (mirrored rather than imported, since importing would drag the JAX stack into the Brain). Stacked on feature/compile-axis-campaign-coverage (needs its load_compile_corpus/compile_tier_of). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
This was referenced Aug 10, 2026
# Conflicts: # agents/conductors/profiling/AGENTS.md # agents/conductors/profiling/_profiling.py # tests/test_profiling_conductor.py
Jammy2211
changed the base branch from
feature/compile-axis-campaign-coverage
to
main
August 10, 2026 15:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the Brain side of phase 2 (autolens_profiling#103). Pairs with
autolens_profiling#104, which owns the pins this reads.
Why
This is the surveillance the whole arc exists for. The persistent compilation cache
(117.0 s → 2.3 s CPU; 5517.8 s → 937.1 s A100) and
--xla_gpu_autotune_level=0(17×on the pathological cold probe) are settings. A config drift or an
XLA_FLAGSclobber — the PyAutoNerves#127 failure that went undetected for two months — puts the
70-minute worst case straight back with nothing failing.
What it does
Reads the workspace's
jax_compile/pins.jsonand reports warm rows that areunpinned or have drifted. Every comparison happens strictly inside one
(hardware, hostname, jax_version, mixed_precision, cache_state). Ajax_versionbump recompiles once by design, so it surfaces as a new unpinned key, never as
drift.
Against the real corpus: 25 pins, 0 drifted, 0 unpinned.
Two corrections that came from running it, not from designing it
1. Rows predating their pin are not drift. The first run flagged four — all
July-16 measurements that the July-28 pin had been chosen over. It was reporting
the improvement that set the pin as though it were a regression. Drift now requires a
row newer than its pin.
2. That exposed the deeper flaw, fixed in autolens_profiling#104: pins must be
sticky. With "most recent warm row wins", re-deriving pins after a cache
regression would have moved the pin onto the regressed value and every later check
would have reported all-clear forever. An existing pin now never moves without an
explicit
--repin.I would not have found either from the design alone — both only showed up against
real data.
Thresholds
Drift needs both gates: ≥
2.0×the pin and ≥1.0 sabsolute. The ratioalone screams about sub-second cells where 100 ms of jitter is 3×; the floor alone
misses a cheap cell degrading by an order of magnitude. Both are generous because
host load alone has produced 7× errors in this corpus, and an alarm that cries
wolf gets ignored.
Tests
11 more (27 total in the file), including:
jax_versionbump classified as a new key rather than drift;pins.py— mirrored rather than imported, since importing the workspace would drag the JAX
stack into the Brain.
Full suite: 316 passed. The 2
test_skill_install.pyfailures reproduce on cleanmainin this environment and are untouched.triage --axis compilestill exits 5 — classification is phase 3.Generated by Claude Code