Skip to content

feat(profiling): triage --axis compile — classify compile drift, close the arc - #222

Merged
Jammy2211 merged 6 commits into
mainfrom
feature/compile-triage-drift
Aug 10, 2026
Merged

feat(profiling): triage --axis compile — classify compile drift, close the arc#222
Jammy2211 merged 6 commits into
mainfrom
feature/compile-triage-drift

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Closes #221. Phase 3 of 3 — this closes the compile-axis arc.

Stacked on #220, which is itself stacked on #219. Merge order: #219 → #220 →
this. Each retargets cleanly as the one below lands. Review the
single-commit diff.

Why

Phases 1–2 make compile drift visible; this makes it actionable. ingest says a
warm compile moved. That alone doesn't tell anyone what to do — a cache that stopped
being hit, a compiler flag that stopped reaching XLA, a busy laptop and a genuine
library regression all look identical as a number. triage separates them and names
the owner.

Classifications

classification signal actionable
cache-regression warm compile returned to its own cold scale yes — config/stack, never the library
autotune-regression GPU compile up ≥10× with no cold-scale match yes — check the flag reaches XLA
library-regression growth on an unchanged key, no other explanation yes — bug/ via intake
host-load measuring host's 1m load average was high no — re-measure idle first
expected-recompile key differs from a pin only by jax_version no — one recompile is by design
new-machine / new-precision / new-cell key is simply unpinned no — pin it

The cold-scale comparison is what makes cache-regression a measurement rather than
a guess. 25 of 32 cell/transform keys in the corpus carry both a warm and a cold
row, so the yardstick is real data from the same machine.

Verified by injecting a synthetic regression into a copy of the real workspace —
a warm vag row moved from 1.622 s to its own 34.592 s cold cost:

[cache-regression] imaging/pixelization/hst [vag] @ local_cpu/DESKTOP-H143S82 jax0.10.2 warm
  evidence: warm 34.592s has returned to its own cold scale (34.592s) — the cache is not being hit
  -> config/stack, NOT the library: check jax_compilation_cache_dir ... XLA_FLAGS (PyAutoNerves#127)

Against the untouched corpus: 0 findings.

A deviation from the prompt worth knowing about

The prompt specified five categories. Two of them — drift from a jax_version bump and
drift from changed measurement conditions — cannot reach triage as drift by
construction: both are different comparability keys, so ingest reports them as
unpinned, never as drifted. That's the design working rather than a gap. They're
still classified so nothing vanishes, but they're never regressions and don't count as
actionable.

host-load was added in their place. The prompt didn't list it, but it's exactly what
host_state was added for in phase 2 — and given host load alone has produced 7×
errors in this corpus, a classifier that couldn't say "your laptop was busy" would send
people chasing phantom regressions.

Closing the arc

  • AGENTS.md: compile-time profiling moves out of Future modes into the Modes
    table — all three modes now serve --axis compile.
  • Boundaries records that release-validation script cost stayed with the hygiene
    conductor. It had already been moved out of this agent once; the note exists so the
    question isn't re-opened a third time.

Tests

11 more (38 in the file), including one asserting triage writes nothing to the
workspace (mtime comparison over every file) — the conductor reasons and delegates.

Full suite: 327 passed. The 2 test_skill_install.py failures reproduce on clean
main in this environment and are untouched.

Generated by Claude Code

claude added 6 commits August 10, 2026 13:12
@claude
feat(profiling): add a compile-time axis to campaign coverage
32ecb8d 
The Profiling Agent could not see the compile-time corpus autolens_profiling
is already producing. AGENTS.md listed "JAX compilation-time profiling of
likelihood functions" under Future modes, _profiling.py had no occurrence of
"compile", and all three modes read only results/runtime/ -- while
scripts/misc/jax_compile/ held 93 committed probe records that nothing
cross-referenced against the science grid.

campaign --axis compile answers how much of the grid has compile data on a
tier. Records are placed by their own (dataset_class, model_type, instrument)
rather than by path, since results are filed under <hardware>/<model_type> and
the path drops the class and instrument entirely.

Tier mapping is deliberately NOT TIER_CONFIGS. That map keys off sweep config
names which fold precision into the name (local_cpu_fp64 / local_cpu_mp),
whereas a compile record carries a raw hardware string plus a separate
mixed_precision bool; reusing it would mis-bucket every row. "other" is a real
answer rather than a fallback -- the corpus holds RTX-2060 rows belonging to
neither tier.

Off-grid records (knn, delaunay_matern, the datacube_img* multi-band classes)
and non-tier hardware get their own buckets: real measurements that are
neither grid coverage nor noise. Malformed records are reported with file and
index rather than skipped, surfacing the 4 that carry null hardware, class and
instrument.

The mode reports coverage only and never compares two timings. Compile timings
are host-load-sensitive -- jax_compile/README.md records the first
measurements being wrong by up to 7x (851s vs 117s for the same compile)
because XLA compiles on the host cores -- so rows are comparable only within
(hardware, jax_version, mixed_precision, cache state). Comparison waits on the
pins in phase 2.

ingest and triage reject --axis compile with exit 5 rather than ignoring it,
so a compile flag can never silently return a runtime answer.

The transform axis is read from probe.py's TRANSFORMS literal via the same ast
route load_grid uses for CELLS, so the Brain cannot drift from the instrument.

Adds tests/test_profiling_conductor.py -- profiling was the only conductor
without a test file, so the "runtime axis unchanged" requirement had nothing
to assert against. Hermetic: synthetic workspace fixtures, no real checkout.
Runtime-axis output verified byte-identical against the real workspace.

Phase 1 of 3; PyAutoMind draft/feature/profiling/.

Closes #218

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
@claude
fix(profiling): drop the dead jax_compile/hpc glob from the a100 disp…
854706a 
…atch

The first glob never matched (jax_compile/ has no hpc/ dir) so the fallback
always won, but had it matched it would have printed hpc/batch_gpu/<name>
for a file that lives elsewhere. Use the same path the runtime campaign does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
@claude
fix(profiling): don't report sibling-instrument records as malformed
d17ecb3 
jax_compile/ hosts export_probe.py and trace_profile.py, which append their
own schema into the SAME results/<hardware>/ tree probe.py writes to. Their
records have no hardware/dataset_class/instrument because they are a different
record kind, not because they are corrupt -- so the 4 the mode was reporting
as malformed would have sent someone to fix two files that work correctly.

Split the two: missing the whole identity triple is a sibling instrument
(reported per file, under its own bucket); missing only some key fields is
genuine corruption and stays malformed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
@claude
feat(profiling): ingest --axis compile — warm-pin drift detection
73dac14 
Completes the Brain side of phase 2, and is the surveillance the arc exists
for: the persistent cache and --xla_gpu_autotune_level=0 are SETTINGS, so a
config drift or an XLA_FLAGS clobber (the PyAutoNerves#127 failure that went
undetected for two months) puts the worst case back with nothing failing.

Reads the workspace's jax_compile/pins.json and reports warm rows that are
unpinned or have drifted. Every comparison happens strictly inside one
(hardware, hostname, jax_version, mixed_precision, cache_state); cross-key
pairs are never a regression. A jax_version bump recompiles ONCE BY DESIGN,
so it surfaces as a new unpinned key rather than as drift.

Two corrections found by running it against the real corpus rather than
trusting the design:

1. Rows PREDATING their pin are not drift. The first run flagged four, all of
   them July-16 measurements the July-28 pin had been chosen over -- i.e. it
   reported the improvement that set the pin as though it were a regression.
   Drift now requires a row newer than its pin.

2. That exposed the deeper flaw, fixed in the workspace: pins must be sticky.
   With "most recent wins", re-deriving pins after a cache regression would
   have quietly baked the regression in and the surveillance would report
   all-clear forever.

Thresholds are deliberately generous and require BOTH gates -- >= 2.0x the pin
AND >= 1.0s absolute. The ratio alone screams about sub-second cells where
100ms of jitter is 3x; the floor alone misses a cheap cell degrading by an
order of magnitude. Host load alone has produced 7x errors in this corpus, and
an alarm that cries wolf gets ignored.

Against the real corpus: 25 pins, 0 drifted, 0 unpinned.

11 more tests, including a synthetic warm-reverting-to-cold row proving the
alarm fires, each comparability field proving it does not fire across the key,
and a guard that the Brain's mirrored key definition matches the workspace's
pins.py (mirrored rather than imported, since importing would drag the JAX
stack into the Brain).

Stacked on feature/compile-axis-campaign-coverage (needs its
load_compile_corpus/compile_tier_of).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
@claude
feat(profiling): triage --axis compile — classify drift, close the arc
1765cf1 
Phases 1-2 make compile drift visible; this makes it actionable, and answers
the question that decides what anyone does next: who owns this?

Seven classifications, three of them actionable:

- cache-regression  -- warm compile has returned to its own COLD scale. The
                       alarm the whole arc exists for; routed to config/stack,
                       explicitly NOT to the library.
- autotune-regression -- GPU compile up >=10x with no cold-scale match, the
                       shape of --xla_gpu_autotune_level=0 not reaching XLA.
- library-regression -- growth on an unchanged key with no cache, autotune or
                       host-load explanation; routed to bug/ via intake and
                       never debugged inside the profiling repo.
- host-load         -- the measuring host's load average was high. Not a
                       regression until re-measured; this is what host_state
                       was added for.
- expected-recompile / new-machine / new-precision / new-cell -- bookkeeping.

The cold-scale comparison makes cache-regression a measurement rather than a
guess: 25 of 32 cell/transform keys in the corpus carry BOTH a warm and a cold
row, so the yardstick is real data from the same machine. Verified by injecting
a synthetic regression into a copy of the real workspace -- a warm vag row
moved from 1.622s to its own 34.592s cold cost and classified as
cache-regression with that evidence quoted.

Two categories cannot reach triage as drift by construction, and that is the
design working: a jax_version bump or a changed host is a different
comparability key, so ingest reports it as UNPINNED, never as drifted. They
are still classified here so nothing vanishes, but they are never regressions
and do not count as actionable.

Internal key tuples are stripped before emit, so the decision surface stays
the documented shape.

Closes the arc in AGENTS.md: compile-time profiling moves out of "Future
modes" into the Modes table, and the Boundaries section records that
release-validation script cost stayed with the hygiene conductor -- it had
already been moved out of this agent once, so the note exists to stop the
question being re-opened a third time.

11 more tests (38 in the file), including one asserting triage writes nothing
to the workspace at all.

Stacked on feature/compile-ingest-pins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
@claude
Merge remote-tracking branch 'origin/main' into feature/compile-triag…
3e06455 
…e-drift

# Conflicts:
#	agents/conductors/profiling/AGENTS.md
#	agents/conductors/profiling/_profiling.py
#	tests/test_profiling_conductor.py
@Jammy2211
Jammy2211 changed the base branch from feature/compile-ingest-pins to main August 10, 2026 15:14
@Jammy2211
Jammy2211 merged commit a50efc3 into main Aug 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(profiling): triage --axis compile — classify compile drift, close the arc

2 participants

@Jammy2211 @claude