feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced

[uipreliga]

What

Adds Dataset.split_field + a --split row filter, ships a seventh plugin skill /coder-eval:optimize-skill, and dogfoods the whole loop in a new tutorial — which is where most of the value in this PR came from.

Implements c/2026-08-12-skill-optimize-p0-p1.md, phase by phase.

The feature

--split (P0). Label dataset rows train / test and select one at run time. The filter runs before either sampler — sampling first would leave an unpredictable (possibly zero) number of rows per split, destroying the comparison the split exists to protect. Unlabelled tasks pass through untouched, so --split is safe in a multi-task run; a labelled task with no matching row raises.

/coder-eval:optimize-skill (P1). A/B tests candidate edits to a skill as experiment variants, promoting only what beats run-to-run noise and then survives a held-out split. Two tracks:

• Activation — the frontmatter description, measured against an activation suite. Does it fire when it should?
• Execution — the skill body, measured against an outcome suite with real success criteria. Having fired, does it do the job?

Explicit-invocation only — it spends real money.

Why the two gates differ (worth a reviewer's eye)

The tracks share splits, snapshots, reachability, replicate discipline and the ledger. What differs is the instrument, and it is load-bearing:

• Activation compares F1, which a pooled suite.json cannot report per replicate — hence three separate invocations.
• Execution compares per-row weighted_score, which paired_comparison already computes correctly over replicates it averages per row before pairing — hence --repeats 3 on exactly two variants.

So the paired block that is mere corroboration on the activation track is the primary instrument on the execution track: tested code instead of arithmetic by hand. Unifying the gates would swap in an instrument that cannot see the metric, so both halves are pinned by sensors.

Two smaller inversions, also sensored: activation rows must never name the skill (that tests obedience, not activation) while execution rows must invoke it by slash command — /plugin:skill at the top of initial_prompt — to hold activation constant; and disable-model-invocation: true now routes to the execution track rather than hard-stopping, so init and ci are optimizable after all.

The slash form is not a stylistic preference. A disable-model-invocation skill is not offered to the model at all, so asking in prose returns "no such skill is available" and the row measures nothing; the slash form loads it, emits a real Skill tool call, and is detected by skill_triggered. Both halves verified live (see below).

Validation

Beyond the test suite, the skill itself was live-tested in three layers, because most of its failure modes are silent rather than loud:

1. Static — every ${CLAUDE_PLUGIN_ROOT} reference, slash command and CLI flag it emits resolves; the experiment YAML in its body validates against ExperimentDefinition.
2. Cold agents — two agents with no context executed the skill end to end, one per track, on a zero-run budget, building real snapshots and experiment YAML for inspection.
3. Live probe — one real agent run to settle whether skill_triggered detects a slash-command invocation (it does; scored 1.0).

That surfaced 15 problems, fixed in 9ac72f6. The two worth a reviewer's attention were silent confounds in the snapshot step: the diagram showed only skills/ while the prose said "everything that source contained" (a diagram-faithful snapshot mounts skills whose bundled reference/ files are missing — invisible on the activation track), and omitting .claude-plugin/plugin.json makes the namespace default to the arm's directory name, so arms would differ in the listing name as well as the text under test. Neither would have errored; both would have produced confident, wrong numbers.

It also produced one piece of guidance now in the skill: before proposing an A/B, check whether a lint rule already answers the question for zero runs. The complaint that prompted the ci test — "emits workflows that miss a step" — overlaps CE026 almost exactly, and ~300 agent runs is the wrong way to learn something a static rule catches permanently.

Three bugs caught before shipping

Two were errors in the plan, found by review; the third was already shipped on main.

1. The sibling-regression gate read the wrong metric. Annexation makes the sibling's criterion expected=yes, observed=no — a false negative. Since precision = tp/(tp+fp), it stays pinned at 1.0 however many requests are stolen. The gate would have been gating on a constant. Now reads recall.yes.

2. A one-skill candidate snapshot empties the sandbox. A variant's plugins block replaces the task's, so the snapshot is the arm's only skill source. Snapshot one skill and every sibling criterion silently observes no — the sibling gate "passes" by measuring nothing. Snapshots now carry the whole skills tree.

3. The shipped reachability guidance was wrong (pre-existing). A local plugin path must be a plugin root holding skills/; the template told users to point at the bare skills directory, which loads nothing. Proven by probing three layouts against the real CLI. Every suite check-skill generated would have reported recall 0.0 — exactly what the template's own comment calls "reads exactly like a broken skill". Corrected in 7 surfaces, including ci, which was writing the broken path into users' CI workflows.

The tutorial (docs/tutorials/08)

Runs the loop for real against this repo's own skills, on Sonnet, and reports what happened rather than a tidied version:

• lint-tasks measured at ceiling (F1 1.000 both splits), so the loop declined to spend ~224 runs chasing an unreachable number. That also closed the plan's open residual: the earlier 66-character description trim is now measured safe.
• A task misfire that reproduced on both splits turned out to be 2-in-3 variance under replicates. Two agreeing runs were not evidence.
• The real headroom was a sibling: analyze missed "what regressed" deterministically. Full three-stage A/B → a-regression promoted, gated on the train split (1.000 vs 0.667, non-overlapping, three invocations) and confirmed on the test split (1.000 vs 0.909).
• Two comparisons were invalidated mid-run and thrown out — one candidate ranked on an eroded denominator, and a confident p = 0.038 that was a billing limit eating one arm harder than the other. completion_rate is what caught both.
• Bare-name collision hazard: skill_triggered strips plugin: prefixes, and Claude Code ships its own unscoped init. A skill_name: "init" criterion would silently score a different skill. Documented in check-skill.

Notes for review

• Listing budget held flat at 1,574/1,600 with a seventh skill added — five descriptions trimmed rather than raising the ceiling, since the budget is shared with every skill the user has installed.
• analyze's description change is the one behavioural edit to an existing skill, and it is measured.
• 3 new lint sensors (mutation-tested); 1 harness gap deferred to .claude/harness-candidates.md — an all-skipped run exits 0, now reachable by a one-character --split typo. Fixing it changes exit semantics for every skipped-task path, so it wants a decision rather than a drive-by.

make verify green: 4071 passed, coverage 91.57%.

🤖 Generated with Claude Code

Late changes (after the first review pass)

Two follow-ups from using the thing, both pushed since the PR opened:

Split values renamed tune/holdout → train/test. Docs and content only — split values are open strings, so no schema change and nothing to migrate; split_field still defaults to "split". 215 replacements, followed by a pass disambiguating bare "test" back to "test split" wherever it named the data half rather than a check, since in a testing framework "without a test" parses exactly wrong. (Pedantic but recorded: nothing is trained here — you hand-select a candidate, which is a form of fitting, so "train" is defensible in the ML sense and far more recognisable than "tune".)

The skill now labels the splits itself rather than asking the user to. It previously offered to add labels, leaving a mechanical JSONL edit and a balance judgement to the reader. It now writes them and shows the counts for objection, under two rules that are easy to get wrong by eye and expensive when you do: stratify within each polarity so both halves carry positives and distractors, and assign deterministically so the split never re-rolls between rounds — a reshuffled split is not a test split, because rows already tuned against leak into it.

Not done, deliberately: a hash-based auto-split (--split train --split-ratio 0.6, membership derived from row_id, no labels in the file) would remove the last of the friction, but it conflicts with a load-bearing behaviour — unlabelled tasks currently pass through unfiltered, which is what makes --split safe in a multi-task run. Auto-splitting them would silently change that, so it needs an opt-in flag and its own tests. Left as a follow-up rather than smuggled in here.

---

Late changes (second pass): the execution track measured for real — and a criterion bug it uncovered

The execution track had never been run end to end. Doing that turned up a bug in skill_triggered that invalidates two claims made earlier in this description, so those are corrected below rather than quietly edited.

⚠️ Behaviour change reviewers should look at

skill_triggered no longer counts an errored Skill call as engagement (src/coder_eval/criteria/skill_triggered.py).

Any suite where a Skill call failed previously scored yes and will now score no. That is the correction — a refused call means the skill's body never loaded — but an existing green activation suite that was passing on refused calls will go red, and correctly so. The file-read signal is deliberately not gated the same way: a refused call loaded nothing, whereas a path reference means the SKILL.md was actually opened. Both cases are pinned by tests.

What the run found

An outcome suite was built to A/B ci's body: 4 arms differing only in that body, 24 rows. All four tied exactly on every criterion — which is a bug report, not a result.

ci sets disable-model-invocation: true, and the Skill tool refuses such a call outright:

<tool_use_error>Skill coder-eval:ci cannot be used with Skill tool
due to disable-model-invocation</tool_use_error>

24 of 24 calls failed this way, result_status: "error" on every one, and no row read the SKILL.md off disk either. The body never entered context; the agent answered from its own knowledge of GitHub Actions, plausibly enough that nothing downstream looked wrong. The tell was in the output the whole time:

body NOT loaded:  uses: anthropics/coder-eval-action@v1   <- does not exist
body loaded:    - uses: UiPath/coder_eval@v0             <- what the body specifies

Re-scoring those same artifacts with the fixed criterion gives 0/6 engagement per arm where the run reported 6/6.

Corrections to claims made above

• "asking in prose returns no such skill is available" — wrong. Measured across the same six rows, by how often the model attempted the call: slash form alone 3/6, prose instruction alone 5/6, slash + explicit imperative 6/6. Prose works better than the slash form. Nothing in coder-eval expands a slash command; it arrives as plain text the model may ignore.
• "Live probe — … whether skill_triggered detects a slash-command invocation (it does; scored 1.0)" — that 1.0 was the false positive. The call had errored.
• "init and ci are optimizable after all" — true, but only with the snapshot fix: delete the disable-model-invocation: line in every arm's snapshot, which is what a real user's slash command effectively does. Verified — the same rows then score 1.000 with the action reference correct.

Also in this pass

• Verdict on ci: no promotion, a verified ceiling. With the body loaded, the train split scores 1.000 on every row and every criterion (18/18 assertions, completion_rate 1.0). The three candidates were solving a problem that did not exist. Stages B and C were correctly never reached. ~55 Sonnet runs, ~$20.
• The outcome-suite template (reference/templates/outcome.yaml) + a checked-in worked example (tasks/skills/ci-outcome.yaml), runnable in two documented lines.
• Engagement is now actually gated (recall.yes: 1.0) in both — they described it as a gate and carried no threshold, which is how a zero-engagement run exited 0.
• CE035 (partly-labelled split datasets) and CE036 (a row's prompt must not contain the string its criteria grade — four scenarios here had that leak).
• Tutorial 09, written from the real numbers, plus tutorial 08 made reproducible (snapshot layout, per-stage experiment YAML, the -e command lines) and restructured into two parts.
• docs/TASK_DEFINITION_GUIDE.md updated — it documented the pre-fix detection rule.

make verify green: 4074 passed, coverage 91.57%.

---

Third pass — the review's P0/P1/P2 fixes, then a tutorial audit

Implements c/2026-08-13-optimize-skill-review-v2-fixes.md, one commit per phase.

Two behaviour changes to look at first

skill_triggered now requires the signal to have actually delivered. The early-stop watcher evaluates on the tool call, where result_status is still None, so an in-flight or crash-force-closed Skill call live-passed while the frozen check scored it no — live and final verdicts disagreed on the one criterion that decides whether a paid round measured the skill at all. Engagement is now an allowlist: a Skill call counts only on "success" (for that tool the body is the tool result), and a Read/Glob/Grep that errored or has not resolved does not count — the path is in its parameters, but nothing loaded. Bash stays ungated, because cat … | grep exits non-zero after genuinely reading the file. Historical activation P/R/F1 is not directly comparable if a run contained those shapes; two golden-corpus entries were appended to force that acknowledgement rather than editing existing ones.

This reverses a documented design decision, so it is worth a reviewer's eye. Seven tests in tests/test_early_stop.py pinned the opposite semantics as deliberate; each is inverted, not deleted, with the reasoning in its comment, plus a Bash twin for the path that still decides on the call. The cost is stated in the test that pays it: a Skill call whose result never arrives no longer stops the run, so that run continues to its turn cap. The old behaviour stopped it and scored yes — crediting a skill whose body never reached the agent.

A mistyped --split now exits 2 instead of 0. It used to raise, get caught like any load failure, and land in skipped_tasks: one yellow line, zero evaluations, exit 0 — a CI gate reporting success for a one-character typo. SplitSelectorError separates a malformed file (still demoted, so one bad task cannot abort a suite) from a malformed invocation (re-raised — the same selector applies to every task in the run). No new CLI code was needed: the existing except ValueError -> typer.BadParameter supplies the exit.

The rest

• coder-eval plan expands datasets and takes --split — prints total/selected row counts and catches a bad ${row.*} at plan time rather than per-row after the sandbox is built. The pre-spend check now costs nothing. It is also louder: a task with a missing dataset JSONL now fails a plan it previously passed.
• Partial split labelling stops being silent — a WARNING with the drop count from expand_dataset, a yellow ⚠ from plan, and a check in lint-tasks for adopters CE035 cannot reach. All three call row_split_label, so the rule cannot drift even though the messages differ.
• Row-id validation covers the whole dataset, before --split and sampling narrow it. A malformed row in the test half used to validate under every --split train run and surface at promotion time — the most expensive moment to learn it.
• The bundled activation.yaml caps and isolates (run_limits, setting_sources: []), mirroring the checked-in worked example key for key. It preached both and shipped neither, so a user who copied it got the opposite of the advice they had just read.
• CE036 gets real fixtures and a skill_name exemption. Its only test was a repo scan that passed whether or not the rule could still detect anything; detection now lives in a helper the scan and six fixtures share. CE036_LOCATOR_FIELDS is the source and CLAUDE.md the derived surface, with a both-directions parity sensor — that list had already drifted once.
• reference/optimize-method.md — the track-invariant method extracted from an 859-line SKILL.md as a byte-identical move (verified against HEAD), the three named seams fixed, and the prose sensor split into procedure-vs-method sets that state which file each token belongs to and why.
• Method caveats the review asked for: promotion verdicts apply to the flag-removed configuration, and Stage B now requires a predeclared primary criterion plus guardrails — scanning every per-criterion aggregate post hoc is uncorrected multiple testing in the rejection direction, so noisy criteria veto real wins.
• ci-outcome's both-triggers row asked for two triggers and graded one, so a schedule-only workflow scored 1.000 — and it graded the same string as its sibling, discriminating nothing. Fixed without touching any train row, so the file's own measured "the train split scores 1.000 on every row" claim stays true by construction.

Tutorial audit

All nine checked mechanically — every path, relative link, cross-doc anchor, CLI flag against live --help, and every complete YAML snippet against the real models. All clean. By content, three defects, every one the same shape: a hand-maintained number describing a set the code derives.

• "14 criterion types" (×3) against a registry of 15. CLAUDE.md's heading was right, which is why nothing noticed — it was the only guarded surface.
• "six commands" in tutorial 07, omitting optimize-skill entirely. A skill-count sensor already existed; this file was not one of the four surfaces it read, and the tutorial's phrasing ("commands") was a fourth the matcher did not know.
• Tutorial 09 quoting ci-outcome.yaml excerpts this PR had just changed — a reader copying the shown row would have built a suite that raises at expansion.

Each is now derived rather than asserted, and tutorial 07 joins SKILL_DOC_SURFACES.

Tutorial 08 Part 1 was re-measured rather than annotated — 62 Sonnet runs, three replicates of the train split plus one of the test split. lint-tasks still holds at F1 1.000 on a suite three rows larger. analyze reads 1.000 where Part 1 recorded 0.000, because the description this PR promotes is committed (4c7481c): the baseline is unreproducible precisely because the loop worked. And hard-3 is still unstable at exactly the 2-in-3 Step 6 measured — months later, on a different revision of the suite, which is the page's own "two agreeing runs are not evidence" lesson arriving unprompted. The page now states what a reader gets today and why it differs, instead of quietly refreshing the numbers.

Notes for review

• 6 new sensors + 6 CE036 fixtures, all mutation-verified — each was confirmed to fail when the property it guards is reverted.
• 1 harness gap deferred: "the ToolStart seam decides" is now a per-criterion property rather than a global invariant. Not mechanically detectable — it is about what a live_verdict reads, which no AST rule can infer.
• No new Pydantic model, criterion, agent, config key or MergeField; one new exception type (SplitSelectorError), one module logger, one CLI option on an existing command.

make verify green: 4115 passed, 390 lint checks, coverage 91.63%.

[github-actions]

Claude finished @uipreliga's task in 1m 29s —— View job

---

Code Review in Progress

I'm reviewing PR #109 following the guidelines in .github/code_review.md.

Todo List

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Run git diff origin/main...HEAD to see full changes
• Review changed files with full context
• Perform cross-file consistency checks
• Analyze what's missing
• Provide comprehensive review feedback

Starting review now...

[uipreliga]

Review: coder_eval — pr:109

Scope: pr:109 · branch feat/plugin-optimize-skill · f1a684d · 2026-08-13T04:57Z · workflow variant

Change class: complex — introduces a new dataset row-filter control path (--split) with new validation/error semantics that interacts with the existing sampler win-order, plus a new published plugin skill and its lint/doc-surface enforcement

Architecture and security are flawless (both 10/10) and the --split feature is well-designed and thoroughly documented, but the real risk is a cluster of silent-green paths around the new row-selection stage — a mistyped or unmatched --split produces a zero-task run that still exits 0, a split_field naming no row key makes the filter an unannounced no-op over the full dataset, and the CLI plumbing for the flag is dead-code-tolerant (mutation-proven) — so the bottom line is: merge once the selector paths are made loud, since every one of these can report a confident train/test result that was never actually measured.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.6 / 10	0	0	2	4	expand_dataset absorbs the --split filter inline: cyclomatic complexity rises C(15) -> D(22)
2. Type Safety	9.3 / 10	0	0	1	2	A split_field that names no key present in the rows makes --split a silent no-op: the full dataset runs under every split name, with no warning (task_loader.py:437)
3. Test Health	7.9 / 10	0	1	2	1	New --split CLI flag is untested at the CLI layer — mutation-proven: nulling split=split at run_command.py:500 keeps the whole suite green (4301 passed, 14 skipped), and the documented exit-0-on-mistyped-split path is unasserted
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	9.5 / 10	0	0	1	0	load_task's blanket except Exception (task_loader.py:68, wrapping the try opened at line 60) relabels programming errors as ValueError, defeating the narrow-catch contract documented at experiment.py:632-637 and demoting first-party regressions to SkippedTask
7. API Surface & Maintainability	8.4 / 10	0	1	1	1	Mistyped/unmatched --split yields a zero-task run that still exits 0 (no run-level guard; asymmetric with the zero-task-files fatal path)
8. Evaluation Harness Quality	9.4 / 10	0	0	1	1	Published activation template ships no run_limits, while the PR's own new in-tree suite documents caps as load-bearing for its metric denominator

Overall Score: 9.1 / 10 · Weakest Axis: Test Health at 7.9 / 10
Totals: 🔴 0 · 🟠 2 · 🟡 8 · 🔵 9 across 8 axes.

Blockers

1. [Axis 3] New --split CLI flag is untested at the CLI layer — mutation-proven: nulling split=split at run_command.py:500 keeps the whole suite green (4301 passed, 14 skipped), and the documented exit-0-on-mistyped-split path is unasserted (tests/test_dataset_expansion.py:742) — Two halves, both proven by direct experiment in the PR-HEAD worktree.

(a) PLUMBING. src/coder_eval/cli/run_command.py:500 reads split=split, (inside the config = BatchRunConfig( call at line 490). I mutated exactly that line to split=None, and ran the full suite: 4301 passed, 14 skipped in 39.06s — zero failures. So the flag can be completely dead (every run silently expands the whole dataset instead of the requested split, and the reported recall/precision/F1 is pooled over train+test while the operator believes they measured one half) with a green make verify. grep -rn 'split=' tests/*.py shows the deepest entry point any test exercises is BatchRunConfig(run_dir=..., split="train") (tests/test_dataset_expansion.py:690, :732, :756) — nothing calls run_command/_run_all_tasks with a split, unlike the CLI-level precedents in tests/test_cli_type_flag.py and tests/test_run_command_junit.py.

(b) EXIT CODE. test_unmatched_split_demotes_the_suite_to_a_skipped_task (line 742) stops at the resolver: it asserts resolved == [] and "no rows in split 'holdou'" in skipped[0].reason (lines 758-761). It does not assert the consequence CLAUDE.md and the skill both state: with zero tasks, summary.tasks_failed/tasks_error/failed_suite_gates are all 0, so run_command.py:560 (if summary.tasks_failed > 0 or summary.tasks_error > 0 or failed_suite_gates > 0:) never raises typer.Exit(1) → a green CI run over zero rows. The only user-visible signal is the console.print at run_command.py:691, which the routed coverage report lists as an uncovered line.

Fix: add a _run_all_tasks(..., split="test") test asserting the resolved/run task set (kills mutation (a)), plus one asserting a mistyped split exits 0 with the suite in run.json skipped_tasks (pins (b) as intended rather than accidental). Also pin the one prose defense in tests/test_custom_lint.py::test_optimize_skill_keeps_its_load_bearing_instructions — its ~18-token list omits plugins/coder-eval/skills/optimize-skill/SKILL.md:249 ("Check the resolved row count, not just the exit code. A mistyped split name (--split holdou) is reported as a skipped task and the run still exits 0 — a green run of zero rows."), which is the only thing standing between a typo and a fabricated result.
2. [Axis 7] Mistyped/unmatched --split yields a zero-task run that still exits 0 (no run-level guard; asymmetric with the zero-task-files fatal path) (src/coder_eval/cli/run_command.py:560) — The run's exit gate keys only on executed-task outcomes — run_command.py:559-561:

    # Exit with non-zero code if any tasks failed, errored, or any suite failed its thresholds.
    if summary.tasks_failed > 0 or summary.tasks_error > 0 or failed_suite_gates > 0:
        raise typer.Exit(1)

A labelled suite with no row in the requested split raises in expand_dataset (task_loader.py:441-446), which resolve_all_tasks catches into skipped (experiment.py:632-639). With every task file demoted, tasks_failed == tasks_error == failed_suite_gates == 0, so coder-eval run <suite> --split trian prints one yellow line and exits 0. The CLI is already inconsistent about this: zero task files is fatal — run_helpers.py:92-94 if not all_task_files: console.print("[red]No task files found![/red]"); raise typer.Exit(1) — while zero resolved tasks is green, so the same user-visible outcome ("nothing ran") returns two different exit codes depending on which stage produced the emptiness. The hazard is documented (docs/DATASETS.md:154-155: "a mistyped selector (--split holdou) produces a run of zero tasks that still exits 0") and deferred in .claude/harness-candidates.md:317-330, but --split is precisely what makes it reachable from a typo rather than a broken YAML, and the optimize-skill workflow it exists to serve passes --split on every one of its ~10 invocations (plugins/coder-eval/skills/optimize-skill/SKILL.md:466, 495-497, 548) — a truncated selector there reads as "the stage ran and nothing regressed". Narrow fix that avoids re-litigating skip: true semantics: after resolve_all_tasks, fail non-zero when a CLI selector (config.split, --tags/--exclude-tags) was supplied and resolved is empty — unambiguously a user error rather than repo state. Note line 691 (the skipped banner, the only on-screen signal) is itself uncovered per the routed coverage report, so add a CLI test for the all-skipped-under---split path either way.

Non-blocking, but please consider before merge

1. [Axis 1] expand_dataset absorbs the --split filter inline: cyclomatic complexity rises C(15) -> D(22) (src/coder_eval/orchestration/task_loader.py:361) — Routed radon signal, quantified against the baseline: radon cc -s on origin/main's copy reports F 341:0 expand_dataset - C (15); on the PR HEAD copy it reports F 361:0 expand_dataset - D (22) — this PR added 7 of the 22 and pushed the function out of the C band. (The other routed offenders are untouched here: resolve_all_tasks is E (36) on BOTH main and PR HEAD — the PR adds only split=config.split at experiment.py:628 — and cli/run_command.py::run_command is C (12) on both.) The growth is a self-contained filter block that the module's own idiom says should be a helper: _load_dataset_rows, _stratified_sample, and the PR's own _reject_duplicate_row_ids (line 313) are all extracted, but this one is inlined at lines 431-446:

    if split is not None:
        field = task.dataset.split_field
        ...
        labelled = [r for r in rows if r.get(field) not in (None, "")]
        if labelled:
            rows = [r for r in labelled if str(r[field]) == split]
            if not rows:
                raise ValueError(...)

Extract it as _filter_by_split(rows, field, split, task_id) -> list[dict[str, Any]] beside _stratified_sample, which restores expand_dataset to roughly its pre-PR complexity, moves the 4-line 'unlabelled means absent/null/""' rationale comment onto the helper's docstring, and gives the semantics a unit-testable seam of its own. While there, consider narrowing _reject_duplicate_row_ids(rows, task) to (rows, id_field, task_id) so its assert task.dataset is not None disappears.
2. [Axis 1] New lint test duplicates the JSONL row-loader (and a function-local import json) verbatim from the test directly above it (tests/test_custom_lint.py:1422) — test_activation_rows_split_both_polarities_both_sides (line 1416) copies the loader from test_activation_rows_have_both_polarities (line 1404) character for character — lines 1422-1428 repeat lines 1405-1411:

        import json

        rows = [
            json.loads(line)
            for line in (self.TEMPLATES / "activation-rows.jsonl").read_text(encoding="utf-8").splitlines()
            if line.strip()
        ]

That is two copies of the same 6 lines in adjacent methods of one class, including a redundant function-local import json in each (the module imports re and Path at the top, lines 13-14, so there is no import-cost reason for the locals). Hoist a single def _template_rows(self) -> list[dict[str, Any]] helper onto TestPluginArtifacts (it already carries self.TEMPLATES) and move import json to the module header; both tests then read as the assertion they are about. Also note the pattern will recur — this is the second reader of that template file, so a third would copy it again.
3. [Axis 2] A split_field that names no key present in the rows makes --split a silent no-op: the full dataset runs under every split name, with no warning (task_loader.py:437) (src/coder_eval/orchestration/task_loader.py:437) — The unlabelled state is inferred from row data rather than declared, and the two very different causes collapse to the same value domain. task_loader.py:437-446:

labelled = [r for r in rows if r.get(field) not in (None, "")]
if labelled:
    rows = [r for r in labelled if str(r[field]) == split]
    ...
# else: no row in this task carries a split label -> --split does not apply here.

field comes from Dataset.split_field (src/coder_eval/models/tasks.py:261), a free str with no min_length and no cross-check against the row keys — pyright and pydantic both accept any value. Verified empirically against the PR worktree with a 2-row labelled dataset ({'id':'a','split':'train'}, {'id':'b','split':'test'}):

• split_field='spilt' (typo) + --split test → ['a', 'b'] (both rows, filter silently inert)
• split_field='' + --split test → ['a', 'b'] (same)

So a user whose JSONL column is set/fold, or who typos either the YAML value or the JSONL key, gets the full dataset under both --split train and --split test with no warning — exactly the tutorial-08 / optimize-skill workflow, where the two arms would then be identical full-dataset runs reported as "confirmed on held-out data". Contrast the mistyped-selector case, which DOES surface (documented in docs/DATASETS.md); the mistyped-field-name case fails in the opposite, silent direction.

Fix: make the inert case observable rather than inferred. Cheapest form — when split is not None and a dataset-backed task took the else branch at line 446, logger.warning("--split %r: task %r has no row carrying split_field %r; running all %d rows", split, task.task_id, field, len(rows)). Optionally add min_length=1 to Dataset.split_field so the empty-string form is rejected at load. A CE-style lint rule can also assert that any tasks/**.yaml setting split_field names a key present in its dataset.paths rows.
4. [Axis 3] Split label vocabulary (train/test) is pinned on the prose side but not the data side — renaming the shipped template's labels keeps all 344 lint tests green while every documented --split command hardcodes the old names (tests/test_custom_lint.py:1437) — The new guard deliberately accepts any two names: line 1437 is assert len(by_split) >= 2, f"template rows collapsed to a single split: {sorted(by_split)}", and line 1429 only checks assert all(r.get("split") for r in rows). Meanwhile the consuming surfaces hardcode the names: plugins/coder-eval/skills/optimize-skill/SKILL.md:247 (coder-eval run <suite> --split train -D run_limits.stop_early=false), :466, :495-497, :548, :581 (--split test --repeats 3); plugins/coder-eval/skills/check-skill/SKILL.md:159 ("every row's split → train or test"); docs/tutorials/08-optimizing-a-skill.md. I verified the gap by rewriting the shipped template rows' labels to fit/eval (plugins/coder-eval/reference/templates/activation-rows.jsonl) and running uv run pytest tests/test_custom_lint.py: 344 passed. With that drift, every suite a user copies raises "no rows in split 'train'" → demoted to skipped_tasks → the zero-row green run of finding #1. This file already has the exact pattern to reuse — test_lint_tasks_does_not_flag_the_shipped_activation_template (line 1581) reads the template with yaml.safe_load and asserts its real contents match what the skill prose claims. Do the same here: assert set(by_split) == {"train", "test"}, or better, derive the expected names from the --split <name> tokens found in optimize-skill/SKILL.md so the two sides can never drift independently.
5. [Axis 3] The repo's own new activation suite gets none of the dataset guards the shipped template gets — five template-pinned guards, zero for tasks/skills/lint-tasks-activation* (tests/test_custom_lint.py:1416) — test_activation_rows_split_both_polarities_both_sides (line 1416) and the three sibling template guards (test_activation_template_expands_to_one_task_per_row line 1270, test_activation_template_thresholds_use_real_metric_keys line 1287, test_activation_template_makes_the_skill_reachable) are all hardcoded to self.TEMPLATES / "activation-rows.jsonl" / activation.yaml. The PR also adds a real suite — tasks/skills/lint-tasks-activation.yaml + tasks/skills/lint-tasks-activation-rows.jsonl (28 rows) — which drives the paid optimize-skill dogfood rounds and has zero guards. Proven: I deleted , "split": "train" from the pos-1 row of tasks/skills/lint-tasks-activation-rows.jsonl and the full suite still reported 4301 passed, 14 skipped — i.e. precisely the state the guard's own message calls "the one bad state: --split keeps the matching rows and drops the unlabelled ones, shrinking the suite the metrics are computed over" (line 1430-1432) is unguarded on the suite that actually runs. Note the only whole-tree task check, the CE034 parametrization at line 2661 (sorted(p for p in (... / "tasks").rglob("*.yaml") ...)), calls load_task (line 2666) but never expand_dataset, so a bad dataset.paths entry, a duplicate row id, or an unsubstituted ${row.…} in the new suite is also uncaught. Fix: parametrize the split-labelling/balance guard (and an expand_dataset smoke assertion) over both jsonl/yaml pairs instead of the template alone.
6. [Axis 6] load_task's blanket except Exception (task_loader.py:68, wrapping the try opened at line 60) relabels programming errors as ValueError, defeating the narrow-catch contract documented at experiment.py:632-637 and demoting first-party regressions to SkippedTask (src/coder_eval/orchestration/task_loader.py:68) — Pre-existing, but load-bearing for the error routing this PR extends, and it contradicts a comment in another in-scope file. task_loader.py:62-69:

try:
    task = TaskDefinition(**task_data)
    task = resolve_template_paths(task, task_file.parent)
    task = resolve_initial_prompt_file(task, task_file.parent)
    task = resolve_system_prompt_files(task, task_file.parent)
    task = resolve_dockerfile_path(task, task_file.parent)
    return task, raw_yaml
except Exception as e:
    raise ValueError(f"Invalid task definition: {e}") from e

Meanwhile experiment.py:632-636, the handler that consumes it, claims the opposite:

# Narrow set: real load failures only. We deliberately don't catch
# AttributeError / TypeError / ImportError — those signal a regression
# in load_task / expand_dataset and should crash loudly rather than
# silently demote every task to "skipped".

The narrowing is defeated for the whole load_task half: an AttributeError/TypeError/ImportError raised inside any of those four path resolvers is converted to ValueError at line 68, matches except (... ValueError ...) at experiment.py:637, and is demoted to a SkippedTask. With no all-skipped guard on that block (finding #1), a first-party regression in path resolution therefore produces a green, zero-task run rather than a crash — precisely the outcome the comment says is prevented.

Fix: narrow line 68 to the exception classes that genuinely mean "invalid task definition" — except (ValidationError, ValueError, FileNotFoundError, OSError) as e: — so programming errors propagate uncaught and reach the deliberately-unhandled path. This is grep/AST-shaped and a good CEnnn candidate: forbid except Exception in orchestration/ load paths whose callers document a narrow catch set. n/a
7. [Axis 7] optimize-skill cost table references undefined M_tune / M_holdout symbols after the train/test rename (SKILL.md:444) (plugins/coder-eval/skills/optimize-skill/SKILL.md:444) — Technique-1 rename grep over src/ tests/ docs/ tasks/ plugins/ experiments/ CLAUDE.md README.md .claude/ finds the rename otherwise complete, with three stragglers — the first is a real broken reference. SKILL.md:440 defines the symbols:

`M_train` train rows and `M_test` test rows:

but the table immediately below uses the retired names (SKILL.md:444-447):

| Step 6 baseline | `M_tune` |
| Stage A — triage | `(N+1) × M_tune` |
| Stage B — gate | `3 × (S+1) × M_tune` |
| Stage C — confirm | `6 × M_holdout` |

M_tune and M_holdout are defined nowhere in the file, and this table is load-bearing: the skill requires the agent to "State the projected run count before each stage and ask" before spending real money. Rename all four to M_train / M_test. Two lower-impact sites of the same class: src/coder_eval/models/tasks.py:264 offers a third vocabulary word the project no longer teaches — "Row field naming the row's split (e.g. 'train' / 'test' / 'holdback'). " (this Field description is the schema's public documentation; use 'train' / 'test'), and docs/DATASETS.md:154 illustrates a typo with `--split holdou`, a truncation of the retired holdout — switch to a truncation of a label the docs actually teach (e.g. --split trian) so a reader cannot mistake it for a valid split name.
8. [Axis 8] Published activation template ships no run_limits, while the PR's own new in-tree suite documents caps as load-bearing for its metric denominator (plugins/coder-eval/reference/templates/activation.yaml:41) — The in-tree suite added by this PR caps every row — run_limits: {max_turns: 2, turn_timeout: 120, task_timeout: 300} (tasks/skills/lint-tasks-activation.yaml:39-42) — and explains why in the comment above it: "Without caps the agent spends turns exploring a sandbox that deliberately holds no eval files, and a row that times out is EXCLUDED from the confusion matrix rather than scored — shrinking the denominator the metrics are computed over (watch completion_rate in the rollup)." That mechanism is real: reports.py:848-850 slices row.result.success_criteria_results[i] only if i < len(...), so a row that errored before criteria ran silently leaves per_rows, and .claude/shared/run-layout.md:34-36 documents it as rows_excluded. The template that every check-skill-generated suite is copied from has an agent: block at line 41-44 and a dataset: block at 46, and no run_limits: anywhere — so generated suites inherit experiments/default.yaml's max_turns: 20 / task_timeout: 600 (10× the cost per activation probe) and gate recall.yes/precision.yes (lines 64-66) over a denominator that timeouts can erode with nothing failing. Add the same run_limits block to the template with the real task's comment, and consider adding completion_rate: 1.0 to its suite_thresholds — run-layout.md:39-42 documents that exact gate for this purpose. (setting_sources: [] is a second divergence — the real task calls the host CLAUDE.md "expensive and a confound" — but it needs type: "claude-code", which would pin the template to one agent, so that half is a judgement call.) n/a

Nits

1. [Axis 1] "Three requirements specific to this track" is followed by four bullets (plugins/coder-eval/skills/optimize-skill/SKILL.md:163) — Line 163 reads "Three requirements specific to this track:" and the list under it has four top-level bullets: line 165 "Invoke the skill from the prompt.", line 187 "Assert engagement, do not assume it.", line 191 "Score outcomes, not prose.", line 196 "Cover what already works, not just what is broken.". An agent executing the skill counts requirements to check it has satisfied them all, so a miscount invites treating the fourth as optional commentary. Change to "Four requirements" (the parallel enumerations elsewhere in the file are correct — "Five facts" at line 414 has exactly five bullets, "Two rules" at line 219 has two), or drop the count and let the list speak for itself so a future bullet cannot make it stale again.
2. [Axis 1] Two-topic comment block sits above the wrong statement, and a comment marks code that is no longer there (src/coder_eval/orchestration/task_loader.py:422) — Lines 422-428 are one unbroken 7-line comment covering two unrelated decisions, and the half that comes first describes a statement nine lines below it. The line it actually sits on top of (429) is the duplicate check, not the split filter:

    # --split filters BEFORE either sampler below: sampling first would leave an
    # unpredictable (possibly zero) number of rows per split, destroying the
    # train/test comparison the split exists to protect.
    # Duplicate ids are a property of the DATASET, so check the whole row set BEFORE any
    # ...
    _reject_duplicate_row_ids(rows, task)

Split it: keep the duplicate-id rationale (lines 425-428) above line 429, and move the ordering rationale (lines 422-424) down onto the if split is not None: block at line 431 (or onto the _filter_by_split helper suggested in the complexity finding). Relatedly, lines 479-480 leave a comment where the deleted per-row check used to be — "# Uniqueness is already enforced across the whole dataset by / # _reject_duplicate_row_ids, before filtering narrowed rows." — inside the hot per-row loop, annotating absent code; the docstring's Raises section already states it, so this can go.
3. [Axis 1] The skill tells users to write split_field: "split", which only restates the model default — now duplicated across four surfaces (plugins/coder-eval/skills/optimize-skill/SKILL.md:217) — Dataset.split_field defaults to "split" (src/coder_eval/models/tasks.py:261-262), so the key is a no-op whenever rows use the conventional field name — yet line 217 instructs the agent to add it unconditionally: "Rows unlabelled → add split_field: \"split\" to the dataset: block and write a \"split\" into every row". Following that guidance has already seeded four copies of the redundant line — plugins/coder-eval/reference/templates/activation.yaml:51, tasks/skills/lint-tasks-activation.yaml:47, docs/tutorials/08-optimizing-a-skill.md:100 — all with the literal default value, and none of the repo's YAML sets a non-default (grep -rn split_field shows every occurrence is "split"). It also implies the key is required for --split to work, which it is not. Reword line 217 to "write a split into every row (dataset.split_field only needs setting if your rows name the field something else)" and drop the key from the template/task/tutorial snippets; if no dataset ever needs a custom name, the knob itself is a candidate for removal (it mirrors stratify_field, which likewise has no non-default user outside tests).
4. [Axis 1] New SKILL.md is 626 lines / 5,647 words — roughly double the next-largest skill, with no use of the plugin's reference/ progressive-disclosure pattern (plugins/coder-eval/skills/optimize-skill/SKILL.md:1) — wc -l plugins/coder-eval/skills/*/SKILL.md gives 626 for optimize-skill against 330 (analyze), 256 (check-skill), 243 (task), 220 (ci), 218 (lint-tasks), 132 (init); by wc -w it is 5,647 vs 2,444 for the next largest — 2.3x. The whole body loads into context on invocation, and several sections are self-contained deep detail that only one step needs: Step 10's three-stage gate reasoning (lines 437-596, ~160 lines, including the per-track gate rationale and the recall-vs-precision derivation at 520-527) and Step 8's snapshot-layout rules (lines 311-385). The plugin already has the mechanism for this — skills read ${CLAUDE_PLUGIN_ROOT}/reference/*.md on demand (this file does so at lines 39, 60, 194, 263). Consider moving the stage/gate detail to reference/optimize-gates.md and keeping the step body as the decision plus a pointer, which also shrinks what a reviewer must re-read when one stage changes. Not urgent — the file is well organized and readable as-is.
5. [Axis 2] split: str | None admits "", the exact sentinel the row side treats as unlabelled, making --split "" unsatisfiable by construction (src/coder_eval/cli/run_command.py:300) — src/coder_eval/cli/run_command.py:300 declares split: str | None = typer.Option(None, "--split", ...) and threads it unchanged through BatchRunConfig.split (src/coder_eval/orchestration/config.py:89, also str | None) into expand_dataset(split=...). The type therefore admits "", but task_loader.py:437 defines a row as unlabelled precisely when its value is None or "" — so no labelled row can ever match split="". Verified in the PR worktree: expand_dataset(mk([{'id':'a','split':'train'},{'id':'b','split':'test'}]), Path('.'), split='') raises Dataset for task 'suite' has no rows in split '' (split_field='split'); labelled splits present: ['test', 'train'], and since resolve_all_tasks catches ValueError into skipped_tasks, coder-eval run --split "" yields a zero-task run that exits 0. Narrow the value at the boundary instead of letting an impossible value travel the whole chain: either reject it in the CLI (a Typer callback raising typer.BadParameter("--split requires a non-empty name")) or normalize split or None before building BatchRunConfig.
6. [Axis 2] _reject_duplicate_row_ids takes a wider type than its precondition and re-narrows with assert, unlike its sibling helper one function up (src/coder_eval/orchestration/task_loader.py:313) — The new helper's signature is def _reject_duplicate_row_ids(rows: list[dict[str, Any]], task: TaskDefinition) -> None: (line 313) and its first statement is assert task.dataset is not None (line 321) purely to narrow before id_field = task.dataset.id_field. It uses only two things from task: task.dataset.id_field and task.task_id. The sibling helper directly above already models this correctly — def _load_dataset_rows(dataset: Dataset, task_file_dir: Path) -> list[dict[str, Any]]: (line 276) takes the narrowed Dataset so no narrowing assert is needed for the parameter itself. Prefer _reject_duplicate_row_ids(rows: list[dict[str, Any]], dataset: Dataset, task_id: str) -> None and call it as _reject_duplicate_row_ids(rows, task.dataset, task.task_id) at line 429, where task.dataset is already narrowed by the if task.dataset is None: return [task] guard at line 415. That deletes the assert (which python -O strips) and makes the precondition expressible in the type rather than at runtime.
7. [Axis 3] Dead assertion clauses in the new reachability guard — one can never fail, the other would raise IndexError if it ever fired (tests/test_custom_lint.py:1361) — In the PR-added test_reachability_guidance_names_the_plugin_root_layout:

• Line 1361: if "SKILL_SOURCE_PATH=" in line or "SKILL_SOURCE_PATH=" in line.replace('"', ""): — the second clause can only be true when the first is false if the raw line embeds a quote inside the token (e.g. SKILL_SOURCE_PATH"="/x/skills). In that case line 1362 (value = line.split("SKILL_SOURCE_PATH=", 1)[1]...) splits the unmodified line, yielding a 1-element list and an IndexError instead of the intended assertion message. Drop the or clause, or normalize once (stripped = line.replace('"', "")) and split the same string you tested.
• Line 1377: assert ".claude/skills" in text and ".claude" in text, ( — the second conjunct is implied by the first and can never fail independently; it reads as two checks but is one. Drop it (or make it a distinct check, e.g. that the surface also shows the correct .claude form on its own).

8. [Axis 7] Skipped-tasks banner enumerates only two of the now-three causes, misdirecting the operator on a mistyped --split (src/coder_eval/cli/run_command.py:693) — run_command.py:690-694 is the only on-screen signal that a --split selector matched nothing:

    if skipped:
        console.print(
            f"[yellow]⚠[/] {len(skipped)} task file(s) skipped "
            + "(load errors or `skip: true` — see run.json `skipped_tasks` for reasons)"
        )

The parenthetical names two causes; this PR adds a third (a labelled dataset with no row in the requested split), so an operator who typo'd --split is told to look for a broken YAML or a quarantined task. Either add the third cause to the string, or drop the enumeration and print each SkippedTask.reason inline — the reason strings already carry the full ValueError text from expand_dataset (experiment.py:637: reason = f"{type(exc).__name__}: {exc}"[:500]), which names the splits that do exist. Line 691 is also flagged uncovered in the routed coverage report, so this whole branch is untested.
9. [Axis 8] The row-selection selector is absent from run.json / suite.json, the documented consumer contract (src/coder_eval/orchestration/config.py:89) — BatchRunConfig.split (config.py:89-96) joins max_rows and sample_per_stratum as a third knob that changes which rows enter a suite metric, and none of the three is echoed into RunSummary (models/results.py:992-1058, which does echo max_parallel at line 1044 as precedent) or into SuiteRollup. So a suite.json read in isolation — the shape an external consumer reads — cannot distinguish a --split test rollup from a full-suite one; only rows_total (a number) and the per-row task_ids hint at it. It is not lost entirely: compute_run_fingerprint dumps the whole config (return config.model_dump(mode="json") | {...}, batch.py:485) into resume_fingerprint.json, but that file is named for --resume and is not mentioned anywhere in .claude/shared/run-layout.md or its plugin mirror. Either surface split on SuiteRollup/RunSummary, or document resume_fingerprint.json in the run-layout contract as the run's config record. Note the tune/holdout → train/test rename is not a cross-repo risk: git grep holdout origin/main returns nothing, so the old labels never shipped and no downstream query can be pinned to them. n/a

What's Missing

Daily/nightly:

• 🟠 Blast radius of the new 28-row PAID suite is unstated. It lands in tasks/ with no skip: true (the repo's existing convention for an environment-dependent suite — tasks/agents/codex_disallowed_tools_test.yaml) and no tag that excludes it from anything. Zero-arg coder-eval run is documented as recursive discovery over tasks/ (cli/run_helpers.py::discover_default_tasks), and any external nightly / eval-runner that globs tasks/** now picks up 28 full agent runs x 3 skill_triggered criteria. Worse, its plugin path is $SKILL_SOURCE_PATH: unset — the default anywhere the operator did not hand-export it — every row scores recall 0, the recall.yes/precision.yes: 0.7 gates fail, and the run exits non-zero. That is a permanent red indistinguishable from the real activation drift the suite exists to detect (the same failure mode the ci skill warns about at SKILL.md:159-168). The PR should say what the nightly/tasks/** consumers do with this suite, or gate it (skip: true / a tag) and document the opt-in. (trigger: tasks/skills/lint-tasks-activation.yaml)

Downstream consumers:

• 🟡 suite_thresholds are evaluated over whatever denominator the split leaves, and nothing recalibrates them per split. The PR's own suite gates recall.yes: 0.7 (line 60-62) over 5 lint-tasks positives on train (4/5 = 0.80 passes) but only 3 on test — where 2/3 = 0.667 fails, so the held-out gate silently demands 100%. The shipped template is worse: after this PR's labelling it has 2 positives on train and exactly 1 on test, so activation.yaml's recall.yes: 0.7 on the test half is decided by a single coin flip. optimize-skill Stage C ("confirm on the test split") will therefore report a threshold failure at a level train tolerated, reading as a regression that is pure granularity. check-skill:132-134 does warn that "recall over 3 rows moves in 33-point jumps" and DATASETS.md:206-208 says budget double the rows, but neither the template's own gate values, the new in-tree suite, nor Stage C's promotion checklist tells the user to re-derive the thresholds for the smaller half. (trigger: tasks/skills/lint-tasks-activation.yaml)
• 🟡 No output artifact records which split produced a run, so every downstream consumer of the changed row-selection logic silently conflates halves: (a) reports_junit.py names testcases per row-task and the suite gate as variant/suite_id (lines 253, 343), so a --split train run's JUnit XML shows the test rows as vanished tests and files its gate result into the same CI-history bucket as a full-suite or test-split run; (b) /coder-eval:analyze (SKILL.md, split-unaware — zero mentions) and the evalboard read suite.json and cannot tell an in-sample train rollup from a held-out one; (c) this PR's brand-new run-layout contract section (.claude/shared/run-layout.md:17-63 and its plugin mirror) documents rows_total / rows_excluded / completion_rate in detail but never mentions that a CLI selector may have chosen the rows. The only record is resume_fingerprint.json, which the contract doc does not list, plus whatever the operator encoded in --run-dir by hand. (trigger: .claude/shared/run-layout.md) (restates: Axis 8: split absent from run.json / suite.json)

Parallel paths:

• 🟡 The CI surface was not extended for the split concept, so the workflow ci emits is in-sample after an optimize-skill round. optimize-skill tunes prose against --split train (SKILL.md:247, 466, 495-497, 548 — all six of its coder-eval run invocations pass --split), and ci then writes the weekly gate that runs the suite with no selector at all, i.e. train+test pooled — the gate's numbers partly re-measure the rows the description was fitted to, which is exactly what the train/test discipline exists to prevent. It is expressible today (extra-args: --split test via action.yml's extra-args input; there is no dedicated split input), but the ci skill's diff here touches only SKILL_SOURCE_PATH and says nothing about splits, and neither does check-skill's new split section when it hands off to CI. (trigger: plugins/coder-eval/skills/ci/SKILL.md)
• 🟡 The two new split hazards are documented in prose only; the plugin's own linter surface — the natural mechanical home for them — was not extended. /coder-eval:lint-tasks (and the shared reference/task-rubric.md it reads with task) gained no rule for (a) a PARTIALLY split-labelled dataset, which check-skill:165-168 now calls "the one state to avoid" because --split silently drops the unlabelled rows and shrinks the gate's denominator, or (b) a split_field naming a key no row carries, which makes --split an inert no-op that runs the full dataset under both selectors. Both are decidable from the task YAML plus its dataset.paths rows — the same inputs lint-tasks already loads — so the linter that ships to users, not just an in-repo CE rule, should refuse them. (trigger: plugins/coder-eval/skills/lint-tasks/SKILL.md) (restates: Axis 2: split_field naming no key present in the rows makes --split a silent no-op)
• 🟡 The new tasks/skills/ directory is not wired into either repo surface that inventories tasks. tasks/README.md calls itself "the map" and documents every sibling directory (agents/, datasets/, internal/, samples/, ...) including — for agents/ — precisely the caveat this suite needs ("several need a specific backend or extra ... so they are not in the CI smoke buckets"); it gained no skills/ entry and no note that the suite needs SKILL_SOURCE_PATH exported. And make run ("Run coder-eval on all tasks", Makefile:89-90) enumerates subdirectories explicitly as tasks/*.yaml tasks/agents/*.yaml and was not extended with tasks/skills/*.yaml — so either the target's "all tasks" claim is now false or the exclusion is deliberate and unrecorded. (trigger: tasks/skills/lint-tasks-activation.yaml)
• 🔵 No cheap surface previews a split selection. coder-eval plan never expands datasets at all (it stops at load_task + resolve_task_for_variant — no dataset/row handling anywhere in plan_command.py) and gained no --split, so the verification optimize-skill Step 6 demands — "Check the resolved row count, not just the exit code" (SKILL.md:250) — is only obtainable from a paid run or a hand-written expand_dataset snippet like the one in pr-checks.yml. A plan --split <name> that printed the resolved row count per suite (or a run --dry-run) would make the mistyped-selector hazard visible for free. (trigger: src/coder_eval/cli/run_command.py)
• 🔵 The skill's second track ships with no worked surface and nothing to dogfood it against. Tutorial 08 (535 new lines) explicitly walks the activation track only ("This page walks the activation track.", line 17), and the PR's in-tree dogfood suite tasks/skills/lint-tasks-activation.yaml is likewise activation-only — so the execution track (~160 lines of SKILL.md: Stage B's paired-comparison instrument, completion_rate parity, per-criterion regression rules) has neither a tutorial nor an outcome suite in the tree that has ever exercised it. At minimum, say in the tutorial or PLUGIN.md that the execution track is documented-but-unwalked, so a reader knows which half has been run for real. (trigger: docs/tutorials/08-optimizing-a-skill.md)

Tests:

• 🟡 The out-of-tree scaffold assert — the one check that exercises the shipped template the way a user's copy sees it (pr-checks.yml:246-268, deliberately run with no source tree, no experiments/) — was not extended when the PR added split to that template. It still asserts only len(rows) == 6 and the expected_skill label set via expand_dataset(task, Path(".")) with no split= argument, so the headline user path this PR ships (copy template -> coder-eval run --split train) is verified nowhere end to end: a template shipped with absent, partial, or renamed split labels passes both that CI step and make lint. (trigger: plugins/coder-eval/reference/templates/activation-rows.jsonl) (restates: Axis 3: split label vocabulary is pinned on the prose side but not the data side)
• 🔵 The resume drift guard is untested for the new field and has a hole for it. BatchRunConfig.split (config.py:89) is picked up by compute_run_fingerprint only because it dumps the whole config (batch.py:485), and no test in tests/test_resume.py asserts split participates in the drift warning. Worse, fingerprint_diff (batch.py:512-518) compares only keys present in the PRIOR stamp, so resuming a run directory stamped before this feature with --split test produces no warning at all and folds pooled prior results into a split subset's run.json. (trigger: src/coder_eval/orchestration/config.py)
• 🔵 The exact invocation the new skill's Stage C prescribes — --split test --repeats 3 (SKILL.md:581) — has no test. Split filtering is covered alone (TestSplitFilter, lines 306-435) and replicate fan-out is covered alone (test_rows_fan_out_times_repeats, line 887), but nothing asserts row-filtering composes with replicate indexing (per-replicate run dirs, rows_total = repeats x filtered_rows in the pooled rollup), which is the combination every paid confirm run will use. (trigger: tests/test_dataset_expansion.py)

Display & mapping dicts:

• 🔵 The hardcoded cause enumeration in the skipped-tasks banner (run_command.py:690-694, "load errors or skip: true") was not extended for the third cause this PR adds — a labelled dataset with no row in the requested split — so the one on-screen signal after a mistyped --split points the operator at broken YAML or a quarantined task instead of the selector they just typed. (trigger: src/coder_eval/cli/run_command.py) (restates: Axis 7: skipped-tasks banner enumerates only two of the now-three causes)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE036 — a symbol-shaped token used inside a Markdown table in a shipped skill must be defined in prose outside any table in the same file. New whole-tree doc-surface rule (a @pytest.mark.lint class in tests/test_custom_lint.py, alongside CE026/CE033, not a BaseRule — it reasons over Markdown, not one .py AST). Implementation: for every plugins/coder-eval/**/*.md, extract inline-code spans, split on non-identifier chars, keep tokens matching ^[A-Za-z]_[A-Za-z0-9_]+$; the set used on lines starting with | must be a subset of the set appearing on non-table lines. VERIFIED by executing it on the tree at PR HEAD: exactly one hit — {M_holdout, M_tune} in plugins/coder-eval/skills/optimize-skill/SKILL.md — and zero false positives across every other bundled Markdown file, so migration cost is the fix itself. M_train/M_test are correctly seen as defined by the legend at SKILL.md:440. Prevents: The merged 4-axis finding at plugins/coder-eval/skills/optimize-skill/SKILL.md:444 — M_tune/M_holdout left undefined in the cost table by the tune/holdout→train/test rename (commit 2ebc522 renamed the legend line and missed the table body). That table is what the agent uses to project spend before each paid stage, so a stale symbol is a live cost-estimation surface. Also prevents the general rename-residue class in any bundled skill's legend/math prose, which no existing guard covers: the prose guard at tests/test_custom_lint.py:1614 asserts required tokens are PRESENT, never that used symbols are DEFINED.
• [ce-lint] CE037 — a spelled-out enumeration count must match the number of top-level bullets that follow it. Same doc-surface lane as CE036. Implementation: a line matching ^\**(One|Two|…|Ten)\b[^.:]{0,70}:\**\s*$ opens an enumeration; count the immediately following top-level - /* /N. bullets (skipping indented continuation lines, blank lines and tables) and require equality. Scope plugins/coder-eval/**/*.md, docs/**/*.md, .claude/**/*.md. VERIFIED: the tightened form (the count word must START the line — the loose form false-positived on a prose sentence at docs/TASK_DEFINITION_GUIDE.md:1184) matches 8 enumeration headers across those trees and reports exactly one violation, the true positive. Zero migration cost. Prevents: optimize-skill/SKILL.md:163 — "Three requirements specific to this track:" followed by four bullets (165, 187, 191, 196). An agent executing a skill counts requirements to confirm it satisfied them, so a miscount invites treating the fourth ("Cover what already works, not just what is broken") as optional commentary. It also protects the correct enumerations in the same file ("Five facts" at 414, "Two rules" at 219) from going stale the next time a bullet is added.
• [ce-lint] CE038 — no YAML key in tasks/** or plugins/coder-eval/reference/templates/*.yaml may set a value identical to its Pydantic field default. Whole-tree rule: yaml.safe_load the file, walk the top level against TaskDefinition.model_fields and the dataset: block against Dataset.model_fields, fail when field.get_default() == authored_value (skip None defaults). VERIFIED by executing the check: exactly two violations tree-wide — tasks/skills/lint-tasks-activation.yaml:47 and plugins/coder-eval/reference/templates/activation.yaml:51, both split_field: "split" — so the rule ships green after a two-line fix. It is the dual of CE031 (dead config = a field nothing reads; this = authored config that changes nothing). Prevents: The split_field: "split" finding — Dataset.split_field already defaults to "split" (models/tasks.py:261-262), so the key is a no-op that the skill's own instruction (optimize-skill/SKILL.md:217) tells agents to add unconditionally, and it has already been copied into four surfaces (template:51, tasks/skills/lint-tasks-activation.yaml:47, docs/tutorials/08-optimizing-a-skill.md:100) with the literal default value. It also implies the key is REQUIRED for --split to work, which is false. Caveat: the rule guards generated YAML only — the prose instruction that generates it must be reworded in the same change, or every new user suite re-adds the line and the rule fires on their tree instead of ours.
• [ce-lint] CE039 — a dataset field-name knob must name a key that exists in the referenced rows. Whole-tree rule over tasks/**/*.yaml plus the shipped templates: resolve dataset.paths relative to the task file, load the rows, and assert split_field / stratify_field / id_field each name a key present in at least one row (and, for split_field, that labelling is all-or-nothing). Add min_length=1 to Dataset.split_field in the same change so the degenerate split_field: "" form is rejected at load. Prevents: The src/coder_eval/orchestration/task_loader.py:437 finding — a split_field naming no key present in the rows makes --split a SILENT no-op (reproduced: split_field='spilt' or '' with --split test returns both rows of a 2-row labelled dataset). In the optimize-skill / tutorial-08 workflow that means the train arm and the test arm are the same full-dataset run, reported as "confirmed on held-out data". Recorded boundary: this reaches in-repo suites only — a user's own JSONL is invisible to any lint pass, so it must be paired with harness item (D); and the min_length=1 half closes only the empty-string sub-case, never the typo.
• [ce-lint] CE040 — every dataset-backed task under tasks/** must survive an expand_dataset smoke check. Extend the existing CE034-style tasks/ scanner (tests/test_custom_lint.py:2661), which today stops at load_task — as do the only two other whole-tree task scanners (test_tags.py:133, test_yaml_migration.py:25). Clauses per suite: expand_dataset succeeds; no ${row.…} placeholder survives substitution; row ids are unique; split labelling is all-or-nothing; a labelled suite carries both polarities on every split. Then parametrize the five template-pinned guards (tests/test_custom_lint.py:1270, 1287, 1382, 1404, 1416) over ALL jsonl/yaml pairs instead of self.TEMPLATES alone. Prevents: Two findings. (1) tests/test_custom_lint.py:1416 — the repo's own new paid-dogfood suite (tasks/skills/lint-tasks-activation.yaml + 28 rows) gets NONE of the guards the shipped template gets; mutation-proven twice (deleting "split": "train" from the pos-1 row, and separately breaking both dataset.paths and a ${row.prompt} placeholder, each left the suite green). (2) The data half of the label-vocabulary finding. The duplicate-id clause comes free: _reject_duplicate_row_ids (task_loader.py:313) already enforces it in the loader, it is simply never exercised for tasks/.
• [ce-lint] CE041 — the split labels shipped in the template rows must equal the --split <name> tokens the shipped skills and tutorial hardcode. Derive the expected label set by parsing --split (\w+) out of plugins/coder-eval/skills/optimize-skill/SKILL.md (6 invocations: lines 247, 466, 495, 496, 497, 548) and check-skill/SKILL.md:159,162, then assert {r["split"] for r in activation-rows.jsonl} == that set. Deriving beats asserting the literal {"train","test"} because it makes the two sides unable to move independently. Prevents: tests/test_custom_lint.py:1437 — the new guard is deliberately value-agnostic (assert len(by_split) >= 2) while the PROSE side IS pinned (tests/test_custom_lint.py:1636-1638 asserts the literal --split train / --split test tokens). Reproduced: relabelling the shipped template rows to fit/eval leaves make lint fully green (344 passed) while every documented command still says train/test; a verbatim template copy then yields "no rows in split 'train'" → demoted to skipped_tasks → the zero-row green run of the merged high finding. This PR's own diff performed exactly that rename by hand, with no mechanical guard on either side.
• [ce-lint] CE042 — no except Exception in src/coder_eval/orchestration/ whose handler raises a DIFFERENT exception type. AST rule in tests/lint/rules/, wired into tests/lint/runner.py: flag an ExceptHandler whose type is bare Exception and whose body contains raise <OtherType>(...). The fix at each site is to narrow to the consumer's own tuple — except (ValueError, FileNotFoundError, OSError, yaml.YAMLError), mirroring experiment.py:637 (pydantic v2's ValidationError is already a ValueError subclass, so listing it is redundant). VERIFIED cost: 8 handlers of this shape tree-wide, exactly 2 inside orchestration/ (task_loader.py:68 and experiment.py:123, the same laundering class), so the scoped rule ships after two edits. Log-and-continue handlers (early_stop.py:401/547, batch.py:201/263) are untouched by the raise clause. Prevents: The src/coder_eval/orchestration/task_loader.py:68 finding — except Exception around the four path resolvers relabels AttributeError/TypeError/ImportError as ValueError, which then matches the consumer's tuple at experiment.py:637 and is demoted to a SkippedTask, defeating the narrow-catch contract that consumer documents five lines above it ("we deliberately don't catch AttributeError / TypeError / ImportError — those signal a regression … and should crash loudly"). Combined with the un-reachable all-skipped guard (experiment.py:725 can never fire for load failures, since attempted += 1 is at line 652, after the handler), a first-party regression in path resolution yields a green exit-0 zero-task run instead of a crash.
• [ce-lint] CE043 — a module-level helper must take the narrowed type, not re-narrow a parameter attribute with assert. AST rule: flag a module-level FunctionDef (methods asserting on self are exempt — that shape is genuinely unavoidable, cf. the six in sandbox.py) whose first statements include assert <param>.<attr> is not None. VERIFIED cost: 3 hits tree-wide — task_loader.py:321, task_loader.py:282, experiment.py:502 — so it is adoptable with at most one # noqa debt marker. Prevents: The src/coder_eval/orchestration/task_loader.py:313 finding: _reject_duplicate_row_ids takes TaskDefinition and re-narrows with assert task.dataset is not None (line 321) purely to reach task.dataset.id_field, while its sibling one function up (_load_dataset_rows, line 276) already models it correctly by taking the narrowed Dataset. The assert is stripped by python -O, so the precondition is expressible in the type but enforced at runtime instead. Moving to (rows, dataset, task_id) also removes the coupling flagged in the complexity finding.
• [ce-lint] CE044 — a dataset-backed task gating classification metrics must cap its run time. Whole-tree rule over tasks/** plus plugins/coder-eval/reference/templates/*.yaml: a task whose suite_thresholds name a classification metric (recall.*, precision.*, f1.*, accuracy) must set run_limits.task_timeout, and should gate completion_rate. Mechanism the rule encodes: reports.py:849-851 slices row.result.success_criteria_results[i] only if i < len(...), so a row whose criteria never ran (the task-timeout watchdog cancel at orchestrator.py:492-540, plus crash paths) silently leaves per_rows — documented as rows_excluded in .claude/shared/run-layout.md:34-36. Prevents: The plugins/coder-eval/reference/templates/activation.yaml:41 finding — the published template every check-skill-generated suite is copied from ships NO run_limits, while the in-tree suite this PR added caps every row (max_turns: 2, turn_timeout: 120, task_timeout: 300) and explains in its own comment why the caps are load-bearing for the metric denominator. Generated suites therefore inherit the PACKAGED default_experiment.yaml's task_timeout: 600 (so wheel-installed users too, not just this repo) and gate recall.yes/precision.yes at 0.7 over a denominator timeouts can erode with nothing failing. Note the raised max_turns: 20 is a cost multiplier, not a denominator risk — MAX_TURNS_EXHAUSTED still scores the row; only the timeout path skips criteria, so task_timeout is the clause that matters.
• [ce-lint] CE045 — tests must not hand-roll a reader for a bundled template/dataset file. Rule over tests/**: a path expression reaching plugins/coder-eval/reference/templates/ must come through the single shared fixture/helper (e.g. a _template_rows() on TestPluginArtifacts, which already carries self.TEMPLATES), not an inline json.loads(... .read_text(...).splitlines()). Recorded boundary on the ruff alternatives so this is a deliberate choice, not an omission: ruff ships no adoptable duplicate-code rule, and PLC0415 (import-outside-top-level — which WOULD have flagged the two redundant function-local import json) is unadoptable here, measured at 983 hits in tests/ and 106 in src/, and CE017 positively REQUIRES lazy agent imports in models/. Prevents: tests/test_custom_lint.py:1422 — the PR's new test_activation_rows_split_both_polarities_both_sides (line 1416) copies the 6-line JSONL loader byte-for-byte from the adjacent test_activation_rows_have_both_polarities (1405-1411), including a redundant function-local import json in each (the file's only two json. uses in ~2900 lines). It is the second raw reader of that file, so a third would copy it again. Forcing the shared helper also nudges the parametrization CE040 asks for, since a fixture generalizes over jsonl/yaml pairs where an inlined literal path cannot.
• [ruff] Enable C901 (mccabe) alongside the existing PLR0915/PLR0912 size gates, and add a checked-in per-function complexity ratchet. MEASURED on the current tree: lint.mccabe.max-complexity = 30 → 0 violations (free to adopt today); = 18 → 5 (claude_code_agent.communicate 21, codex_agent._setup_skills 19, docker_runner._build_argv 29, orchestrator._simulation_dialog_loop 22, reports_experiment.generate_variant_report 24), each absorbable by the # noqa debt-marker policy pyproject.toml:188-192 already documents for function size. HONEST LIMIT, recorded rather than glossed: ruff's mccabe scores expand_dataset at 13 (radon cc -s says D(22) — radon counts boolean operators and comprehensions, mccabe does not), so C901 at ANY tree-passing threshold would NOT have fired on this PR. The form that would is a ratchet: a generated tests/lint/complexity_baseline.json of per-function radon scores plus a rule failing when a function exceeds its recorded value, refreshed by make complexity-baseline (same generated-surface pattern as make plugin-reference/CE033). That converts "the repo enforces no complexity gate" into "no function may get worse", at zero migration cost for the ~20 functions already at D or worse. Prevents: The merged A1/A5 finding at src/coder_eval/orchestration/task_loader.py:361 — expand_dataset went from C (15) on main to D (22) at PR HEAD by inlining the --split filter (lines 431-446) that the module's own idiom says should be a helper (_load_dataset_rows, _stratified_sample, and this PR's own _reject_duplicate_row_ids are all extracted). The ratchet catches it; the C901 threshold catches only the next unbounded-growth case. It also fires on the grouped comment/altitude nit at task_loader.py:422, which shares the root cause.
• [pyright] Stop letting an unsatisfiable selector value travel the whole chain; narrow at the CLI boundary and type the selector into the run record. (a) --split is declared str | None at cli/run_command.py:300 and threaded unchanged through BatchRunConfig.split (orchestration/config.py:89) into expand_dataset, but task_loader.py:437 defines a row as UNLABELLED precisely when its value is None or "" — so "" is unsatisfiable by construction (verified: --split "" raises "has no rows in split ''", is caught into skipped_tasks, and exits 0). Reject it in a Typer callback (typer.BadParameter("--split requires a non-empty name")) or normalize split or None before building BatchRunConfig, and give Dataset.split_field min_length=1. (b) Echo split / max_rows / sample_per_stratum onto RunSummary (models/results.py:992-1058 already echoes max_parallel at line 1044 as precedent) and SuiteRollup, so the row-selection knobs are typed members of the consumer contract instead of being reachable only via resume_fingerprint.json (batch.py:485) — a file named for --resume and absent from .claude/shared/run-layout.md and its plugin mirror. Prevents: Directly: the src/coder_eval/cli/run_command.py:300 finding (--split "" unsatisfiable by construction) and the src/coder_eval/orchestration/config.py:89 finding (selector absent from run.json/suite.json, so a --split test rollup is indistinguishable from a full-suite one). Indirectly it is the enabling half of harness item (D): once split is on RunSummary, an inert-selector or zero-row run is attributable after the fact instead of leaving no artifact trace. Recorded boundary: no type system can catch the MAIN half of the split_field finding — distinguishing "field-name typo" from "dataset genuinely has no labels" needs the user's row data, since rows are dict[str, Any]; that half needs CE039 plus the runtime warning.

Harness improvements (not statically reachable):

• Mutation-kill gate on the CLI/orchestration plumbing seam. Add a make mutate-diff target (and a PR CI job) that, for each changed line in src/coder_eval/cli/ and src/coder_eval/orchestration/, nulls keyword-argument values one at a time and asserts at least one test fails. Cheaper first step: a curated kill-list (tests/mutants.yaml) covering the high-traffic seams — BatchRunConfig(...) construction, _run_all_tasks(...) forwarding, the exit gate — checked by one -m slow test. Why not static: No AST pass can distinguish live plumbing from dead plumbing that happens to be spelled correctly; proving a line is OBSERVED by a test requires executing the suite with that line mutated. The obvious static substitute was measured and rejected: a "every typer.Option flag name appears somewhere in tests/" grep PASSES for --split today, because the string occurs in test docstrings and comments (tests/test_dataset_expansion.py:288, 315, 397, 411, 424) — it would have been a false green on exactly this finding. (A separate genuinely useful AST rule — every typer.Option parameter of a CLI command must be forwarded to some call — catches the adjacent DEAD-flag class but not this one; 5 of 26 flags are currently never named in tests at all: --exclude-tags, --input, --log-file, --tags, --task-dir.) Prevents: The high finding at tests/test_dataset_expansion.py:742 half (a): mutating split=split, → split=None, at src/coder_eval/cli/run_command.py:500 leaves the whole suite green (4301 passed, 14 skipped — reproduced independently), i.e. the entire --split flag can be dead while make verify passes, with pooled train+test P/R/F1 reported as if measured on one half.
• A CLI-layer test for every row-selection selector. Call _run_all_tasks(..., split="test") — the seam the repo already pins this way in tests/test_run_command_junit.py:45, tests/test_cli_telemetry.py:93 and tests/test_cli_empty_glob.py:44 — and assert the resolved/run task ids, not just the resolver's return value. Do the same for --sample, --sample-per-stratum, and --tags/--exclude-tags. Why not static: The property under test is which rows actually entered the run, which only exists after the resolver plus batch have executed. A lint rule can see that the kwarg is passed but not that passing it changed the row set. Prevents: The same high finding from the other direction, plus the untested-plumbing class for the other three selectors, whose deepest current coverage is BatchRunConfig(run_dir=..., split="train") handed straight to resolve_all_tasks (tests/test_dataset_expansion.py:690, 732, 756).
• Exit-code matrix test for every "nothing ran" cause, plus the narrow selector-scoped fix. One table-driven test enumerating: zero task files / all skip: true / tag filter matched nothing / --split matched no labelled row / --split "" / every task demoted by a load error — each with its expected exit code AND expected banner text. Then implement the narrow fix already scoped in .claude/harness-candidates.md:320-333: fail non-zero when a CLI SELECTOR (--split, --tags) eliminated everything, which is unambiguously user error rather than repo state, leaving skip: true semantics untouched. While there, fix the banner at run_command.py:690-694, whose parenthetical still names only two of the now-three causes. Why not static: Exit semantics are runtime behavior, and the load-bearing part is a POLICY decision — which flavour of emptiness should be fatal — that no lint rule can make. The asymmetry is the tell: zero task FILES is already fatal (cli/run_helpers.py:92-94, pinned by tests/test_cli_empty_glob.py:15) while zero RESOLVED tasks is green (cli/run_command.py:559-561); same user-visible outcome, two exit codes, and only one of them has a test. Prevents: The merged A6/A7/A8 high finding — reproduced end to end: coder-eval run tasks/skills/lint-tasks-activation.yaml --split trian prints "1 task file(s) skipped" / "Running 0 task(s)" and EXITS 0. All six coder-eval run invocations in optimize-skill/SKILL.md pass --split, so a one-character typo reads as "the stage ran and nothing regressed". Also covers half (b) of the test-health high finding (the exit-0 consequence is unasserted at tests/test_dataset_expansion.py:758-761) and the uncovered banner branch at run_command.py:691.
• Make an inert selector observable at runtime, and record it in the run artifacts. (a) When split is not None and a dataset-backed task takes the unlabelled fallback (task_loader.py:446), emit logger.warning("--split %r: task %r has no row carrying split_field %r; running all %d rows", ...) — note task_loader.py imports no logging at all today, so this is a new seam rather than a missing call. (b) Once split is echoed onto RunSummary/SuiteRollup (see the pyright static entry), add a golden-run assertion that suite.json carries the selector, and document the new fields (or resume_fingerprint.json) in .claude/shared/run-layout.md plus its plugin mirror. (c) Add the missing OVER-count instruction to optimize-skill Step 6: the existing guidance (SKILL.md:250-253, 476-481) warns only about a row count that is too LOW or zero, so a full-dataset run under --split test is named by no surface. Why not static: The inert case depends on the USER's row data at runtime — CE039 can only see in-repo suites, and no static pass can read a customer's JSONL. The artifact half is a property of a written file observed after a real run, not of the source. Prevents: The src/coder_eval/orchestration/task_loader.py:437 finding (silent full-dataset run under every split name — the failure that makes tutorial-08's two arms identical while being reported as "confirmed on held-out data") and the src/coder_eval/orchestration/config.py:89 finding (selector unrecoverable from suite.json, so a run made inert this way leaves no trace distinguishing it from a full run).
• Denominator-integrity assertion on the dogfood classification suites. After the nightly/activation rollup, assert rows_excluded == 0 (equivalently completion_rate == 1.0) for any suite gating recall.*/precision.*, and add completion_rate: 1.0 to the shipped template's suite_thresholds so a copied suite inherits the gate rather than the hazard. Copy the in-tree suite's explanatory comment rather than its exact max_turns: 2, which risks false-negative recall for a skill that activates only after an exploratory tool call. Why not static: Whether rows were actually excluded is a property of a real run's rollup — CE044 can assert only that a CAP exists, never that no row hit it. The two are complementary: the lint rule bounds the exposure, this assertion detects when it materialized. Prevents: The live half of the plugins/coder-eval/reference/templates/activation.yaml:41 finding — a row cancelled by the task-timeout watchdog never reaches check_all_async, so reports.py:849-851 drops it from per_rows and the 0.7 recall/precision gates are computed over a shrunken denominator with nothing failing. The published surfaces offer no protection today: check-skill's SKILL.md contains no run_limits guidance at all, and optimize-skill only tells the user to inspect completion_rate/rows_excluded after the fact.
• Recorded boundary — the findings static analysis deliberately cannot reach, so the omission is a decision rather than a gap. (i) The dead-assertion nit at tests/test_custom_lint.py:1361/1377: assert A and B where B is logically implied by A needs semantic reasoning, not an AST shape. Its sibling half IS mechanizable if it ever recurs — "the string you test membership of must be the string you .split() on" (if TOK in line or TOK in line.replace('"','') followed by line.split(TOK, 1)[1], which raises IndexError instead of the intended assertion message) — but a rule for a one-off in test code is over-fitting; fix by hand. (ii) The SKILL.md size finding is PARTLY mechanizable: add a SKILL_BODY_BUDGET_LINES cap (~400) with a declared per-skill exemption dict to the derived skill test that already owns SKILL_LISTING_BUDGET_CHARS (tests/test_custom_lint.py:1234) — measured 626 lines / 5,647 words versus 330 for the next largest, and the plugin already has the progressive-disclosure mechanism (${CLAUDE_PLUGIN_ROOT}/reference/*.md, used at lines 39, 60, 194, 263) — but WHICH sections to demote (Step 10's gate reasoning at 437-596, Step 8's snapshot rules at 311-385) is editorial judgment a budget cannot make. (iii) This finding set contains no security findings, so the bandit / pip-audit / CodeQL lane has nothing to enforce here. Why not static: Each sub-item names its own reason: implication between two boolean conjuncts is undecidable by pattern; a size budget can gate growth but cannot choose the split point; and the security lane has no finding to gate against. Prevents: The A3 low finding (dead assertion clauses in the new reachability guard) and the A1 low finding (SKILL.md size / no use of the reference/ progressive-disclosure pattern) — recorded as reviewer-fixed and budget-gated respectively, rather than silently dropped.

Top 5 Priority Actions

1. Fail non-zero when a CLI selector (--split, --tags) eliminated every task — today src/coder_eval/cli/run_command.py:560 gates only on executed-task outcomes, so --split trian prints one yellow line and exits 0 while zero task files is fatal at src/coder_eval/cli/run_helpers.py:93, the same "nothing ran" outcome with two different exit codes.
2. Make the inert-filter case observable at src/coder_eval/orchestration/task_loader.py:437: when --split is requested and no row carries split_field, log a warning naming the field and row count (verified: split_field='spilt' or '' silently returns the whole dataset under both --split train and --split test, so both arms of an optimize-skill run are identical full-dataset runs reported as held-out confirmation).
3. Pin the --split seam in tests: add a _run_all_tasks(..., split="test") case (nulling split=split at src/coder_eval/cli/run_command.py:500 today leaves 4301 tests green), assert the mistyped-split exit-0/skipped_tasks consequence beyond the resolver-level tests/test_dataset_expansion.py:742, and pin the shipped label vocabulary at tests/test_custom_lint.py:1437 (>= 2 splits accepts any names, so renaming the template's train/test keeps all 344 lint tests green while every documented command hardcodes the old names).
4. Narrow the blanket except Exception at src/coder_eval/orchestration/task_loader.py:68 to (ValueError, FileNotFoundError, OSError, yaml.YAMLError) — it relabels AttributeError/TypeError from the four path resolvers as ValueError, defeating the narrow-catch contract documented at src/coder_eval/orchestration/experiment.py:637 and demoting a first-party regression to a SkippedTask in an exit-0 zero-task run.
5. Fix the two published plugin surfaces users copy: add a run_limits block (plus a completion_rate threshold) to plugins/coder-eval/reference/templates/activation.yaml:41, since generated suites inherit the packaged task_timeout: 600 and a timed-out row is excluded from the confusion matrix rather than scored (src/coder_eval/reports.py:849), and rename the undefined M_tune/M_holdout symbols in the pre-spend cost table at plugins/coder-eval/skills/optimize-skill/SKILL.md:444.

---

Stats: 0 🔴 · 2 🟠 · 8 🟡 · 9 🔵 across 8 axes reviewed.