feat(plugin): ship coder_eval as a Claude Code plugin + marketplace

[uipreliga]

What

A new way to use Coder Eval: from inside Claude Code, alongside the CLI and the GitHub
Action. This repo becomes a Claude Code plugin marketplace hosting one coder-eval
plugin, so the whole loop — scaffold a suite, author a task, check whether a skill
triggers, audit the tasks you already have, read the results, wire it into CI — runs in
the agent.

/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval

That adds six slash commands: /coder-eval:init, /coder-eval:check-skill,
/coder-eval:task, /coder-eval:lint-tasks, /coder-eval:analyze, /coder-eval:ci.
They drive the coder-eval CLI, which stays the prerequisite — the plugin ships prompts
and reference material, not a second implementation.

Standing cost is the skill listing: 1,576 characters of description across all six,
always-on for as long as the plugin is installed. Bodies load only on invoke. That budget
is shared with every skill the user has installed, so a test caps the total at 1,600 —
raising the ceiling is allowed in a commit that says why, which is exactly what silent
drift would not be. Current headroom is 24 characters, so the next skill added here is a
deliberate decision rather than an accident.

What a reviewer should check

The src/ diff is three files, +46/−15, and is behavior-preserving. It is not empty —
an earlier revision of this description said it was, and that stopped being true at
ca64bc4:

• models/tasks.py — NORMALIZED_CRITERION_ALIASES and REMOVED_CRITERION_TYPES hoisted
  out of check_removed_criteria_types to module scope, and exported from
  coder_eval.models. The generated criteria reference renders its accepted-alias table
  from the same declaration the loader enforces; a hand-written table there would be a
  second declaration free to drift. The overlay is still applied as {**item, **overlay},
  same order, same semantics.
• models/criteria.py — one Field(description=...) added to LLMJudgeCriterion.temperature.

No criterion type, agent, CLI flag or merge layer moved, so no existing evaluation result
can change. Everything else new is a distribution surface, one test-harness generator, new
lint clauses, one CI job, and docs.

Three invariants carry the weight:

1. The bundled criteria reference is generated, not written (CE033,
   tests/lint/plugin_reference.py). An installed plugin is copied to
   ~/.claude/plugins/cache/ without its parent directories, so a skill cannot read
   docs/TASK_DEFINITION_GUIDE.md at runtime — the criterion vocabulary has to ship
   inside plugins/coder-eval/, where a hand-maintained copy would drift on the next
   criterion change. make plugin-reference renders it from the SuccessCriterion union;
   CE033 re-renders and diffs. Inherited base fields and the discriminator name are
   computed, never listed — the stop_early refactor is exactly the change a hardcoded
   list would have leaked into all 15 criterion sections. The "Accepted aliases" table is
   rendered from NORMALIZED_CRITERION_ALIASES for the same reason.

2. The frontmatter guards are load-bearing, not belt-and-braces. Verified by spike:
   claude plugin validate --strict does not inspect skill frontmatter at all — a
   SKILL.md carrying both an unsupported name: and an invented key passes with exit 0
   and zero warnings. TestPluginArtifacts is the only thing standing between a typo'd
   key and a skill that silently never triggers. It also enforces path containment (no
   skill may name a repo path that won't exist post-install) across every SKILL.md.

3. The CI assert expands, it does not just validate. coder-eval plan exits 0 even
   when dataset.paths names a file that was never copied — verified both ways — so a
   plan-only check would have passed vacuously. The Claude Code Plugin job copies the
   activation template into $RUNNER_TEMP, outside the source tree, and asserts
   expand_dataset yields 6 row-tasks with both label polarities. No secrets, no agent
   run, no per-PR cost.

Also in here

• CE026 gains two things: its doc scan now covers plugins/**/*.md (the ci skill
  emits a workflow users copy verbatim, so it is held to the same agent-runtime
  prerequisite standard), and a fourth clause — every with: key on a snippet's
  action step must be a real action.yml input. GitHub ignores unknown inputs rather than
  failing, so a rename would silently degrade every copied workflow; simulating a model
  rename fires on all four surfaces. Consequence to know: renaming an action input, or
  changing the action's runtime prerequisites, now means updating the ci skill too.
  Documented in CLAUDE.md.
• CE034 is new, and the plugin found the bug that motivated it. Running
  /coder-eval:lint-tasks against this repository's own tasks/ surfaced that an armed,
  live-passable command_executed without require_success lets a crashed invocation
  latch a live PASS, fire on_pass: stop, and — because gating is FIRED-ONLY — report
  SUCCESS over the armed subset without ever consulting the unarmed file_exists. Three
  early_stop_* fixtures were affected. Negative assertions are fail-only and are exempt.
• Two repo-local command twins were retired. .claude/commands/coder-eval-run-analysis.md
  and coder-eval-task-create.md duplicated what /coder-eval:analyze and
  /coder-eval:task now ship — the same run-record vocabulary and task rubric in a second
  place to drift. RUN_RECORD_CONSUMERS collapses to a 1-tuple. The rest of
  .claude/commands/ stays repo-local contributor tooling.
• plugin.json carries a version pinned to pyproject.toml's, because
  --strict rejects a manifest without one. release.yml's existing bump step seds it
  alongside action.yml; tests/test_action_version_pin.py — already the owner of that
  invariant — guards both the value and the sed's line shape at rest.

Evidence the prompts were exercised, not just linted

Mechanical guards prove the plugin loads and is self-contained; they cannot prove the six
prompts are any good. So they were run against a real scratch repo (two planted skills +
a stub CLI) and, for analyze, a real 947-task two-variant experiment run — tools
allowlisted so no paid coder-eval run could fire.

They behaved: check-skill designed 18 rows (10 positive / 8 distractor, none naming the
skill, distractors genuinely adjacent), validated, and stopped to ask before spending;
init wrote .env.example and never .env; ci chose pull_request over
pull_request_target with explicit reasoning about fork secrets; lint-tasks found the
real fixture bug now guarded by CE034.

Prompt-quality defects found and fixed this way:

• analyze got a 947-task run's headline exactly right (180 rows lost to a 120s timeout)
  but drifted on two secondary figures — it split them 146/34 where the run holds 144/36.
  Its principles now require every count to come out of the extraction command.
• analyze also taught a task.json field vocabulary the schema does not have — six of
  thirteen keys were wrong. It now reads the run's actual schema.
• init scaffolded a task whose prompt dictated pypdf while a criterion grepped for it —
  a criterion that cannot fail but reads as coverage.
• task gave weight 1.5 to a criterion matching a literal (--json) the prompt has to
  name.
• ci emitted a scheduled skill-drift workflow that never passed SKILL_SOURCE_PATH, so
  the activation suite it pairs with could never reach the skill under test: every
  positive row scores 0 and the job goes red every week for a reason unrelated to drift.

Out of scope

/coder-eval:compare and /coder-eval:triage (v2, once usage is observed); a second
coder-eval-dev plugin; a duplicate example task under tasks/ — the plugin template is
the single canonical copy.

Verification

make format / make check / make lint (332) / make test (4,052 passed, 4 skipped,
coverage gate clean) all green. Both manifests pass claude plugin validate --strict with
zero warnings.

Rebased onto main at 0.9.6. That release bumped pyproject.toml and action.yml but
could not touch plugin.json, which does not exist on main yet, so the pin was advanced
by hand in 17491f0 — caught by test_action_version_pin.py, which is what that test is
for. After merge the release sed owns the file.

CI is green on the rebased head, including Claude Code Plugin (manifests + offline scaffold) — the job an earlier revision of this description flagged as never having run
on a runner. It has now, along with Action Dogfood, the live integration suites and the
Windows smoke test.

One risk remains unrehearsable: merging triggers a release, which runs the new
plugin.json sed for the first time. It fails loudly on a miss (grep -q guard) and
the at-rest test catches drift afterwards, but the release path cannot be exercised
beforehand.

---

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 33s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Get full PR diff from origin/main
• Review all 27 changed files with full context
• Verify the three claimed invariants
• Check cross-file consistency
• Perform "what's missing" analysis
• Format and post final review

Starting review now...

[bai-uipath]

Approve — no runtime behavior change, so the blast radius on existing CLI and Action users is effectively nil.

Blast-radius check	Result
src/ behavior	Unchanged. The only diff is a criterion field description.
Release path	The new plugin.json sed runs before the tag push, wheel build and PyPI job, so a miss fails the run with nothing published.
Repo-root marketplace	Marketplace registration is always explicit, so contributors working in this repo don't silently load the six skills.

Two of the six don't earn a permanent slot in the skill listing.

Skill	Audience	Frequency	Verdict
skill-check	anyone authoring Claude Code skills	every description edit, plus scheduled	Keep. The one capability with no CLI or docs equivalent.
lint-tasks	anyone with an existing suite	periodic audit	Keep. Found the early_stop_* fixture bug.
analyze	anyone running suites	every run	Keep, dedupe below.
task	suite authors	heavy during a build-out, then rare	Keep, dedupe below.
init	first-time users	once per repository	Drop. Punts to skill-check for the highest-value case, and can't be model-invoked, so it only gets typed by someone who already knows it exists.
ci	first-time users	once per repository	Drop. Restates the CI docs page, and the workflow it emits never carries the skill-source variable, so its headline scheduled-drift case measures nothing.

Unlike a slash command, a skill's description occupies the listing on every turn for as long as the plugin is installed, on a budget shared with the user's own skills. Two once-per-repository entries is a poor trade for that.

Deduplicate the two skills that already have repo-local twins.

Plugin skill	Repo-local twin	State
analyze	coder-eval-run-analysis	Rubric and jq field vocabulary are single-declared and test-guarded; the analysis dimensions and scope-marker detection are independent prose in both.
task	coder-eval-task-create	Rubric shared, the rest restated.

Scope-marker detection now lives in four places. Fix: extend what the rubric already does here. The plugin can't read .claude/ from its install cache, so the direction has to be repo-local pointing at the bundled copy.

Minor: the three early_stop_* fixtures got stricter, so confirm no gate suite selects that tag; tutorial 07 and the plugin doc disagree on how many skills drive the CLI; CLAUDE.md's list of non-BaseRule lint classes omits CE034.

[uipreliga]

Thanks — the ci finding was the sharp one and it was a real bug, not a doc nit.

Scope note first: this review was generated against f8c9691, which is now 11 commits back. Two of its items were already closed by commits it could not see; those are marked below. Everything newly fixed is in 6d9d434.

Net: 3 fixed, 2 already fixed, 1 verified clean, 1 declined.

---

ci's scheduled drift case measures nothing — ✅ Fixed

Reproduced exactly as filed, and it is worse than "measures nothing".

The skill headlines the scheduled trigger as "the skill-drift case … the one they forget", but the workflow it emits passes only ANTHROPIC_API_KEY. The activation template it pairs with loads the skill under test through agent.plugins, whose path is $SKILL_SOURCE_PATH so the committed suite stays portable across machines. Unset, the skill is never offered to the sandboxed agent → every positive row scores 0 → the job fails its recall threshold.

So the failure mode is not a silent zero, it is a permanent red every Monday that looks exactly like total drift — which means the one signal the schedule exists to produce is the signal it destroys. Real drift arrives into an already-red job and goes unnoticed.

Step 3 now names the variable, instructs the skill to grep the discovered tasks for the general case (any $ in an agent.plugins path, not just this one name), and shows it resolved against the checkout:

env: |
  ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
  SKILL_SOURCE_PATH=${{ github.workspace }}/.claude/skills

Step 2 marks it non-optional for that trigger rather than leaving it to the generic "pass your env vars" paragraph, which is what it fell through before.

Tutorial 07 vs. the plugin doc on CLI-driving skills — ✅ Fixed

You were right that they disagree, and the tutorial was the wrong one. It claimed five of six drive the CLI with lint-tasks as the exception; three do — init, check-skill, task (the ones that shell out to coder-eval --version / plan / run). analyze reads a finished run directory and ci writes a workflow; neither invokes the CLI. PLUGIN.md already said three.

CLAUDE.md omits CE034 — ✅ Fixed

Correct. CE032 is a genuine BaseRule in the runner so its absence from that list is right, but CE034 is a dedicated whole-tree class over tasks/. Added, with a line on what it enforces rather than just the ID.

early_stop_* fixtures vs. gate suites — ✅ Verified clean, no change

Worth checking and it holds: those three carry early-stop, weighted-early-stop, decision-budget; pr-checks.yml selects only smoke-pass, smoke-fail, smoke-variants. Nothing else in the repo selects those tags.

Deduplicate analyze / task against their repo-local twins — ✅ Already fixed (a40848b)

.claude/commands/coder-eval-run-analysis.md and coder-eval-task-create.md are deleted. RUN_RECORD_CONSUMERS collapses to a 1-tuple and README/USER_GUIDE point at the plugin. Scope-marker detection is back to one place, not four.

---

Drop init and ci — ❌ Declined

Your premise checks out: I confirmed disable-model-invocation does not remove a skill from the listing (the Claude Code changelog carries an entry about improving the refusal when Claude tries to invoke one — so the model sees it and is turned away at the door). Both do occupy the shared budget on every turn. The cost is real.

Declining on three grounds:

1. The budget guard already exists and is the intended control. Descriptions total 1,576 against a 1,600 ceiling that a test enforces, with a comment saying raising it is allowed in a commit that says why. That mechanism was built for exactly this tension. Dropping skills to service a limit that is not yet binding spends the wrong currency.

2. Discovery is the docs' job, not the model's. "Can't be model-invoked, so it only gets typed by someone who already knows it exists" applies equally to ci, which you keep-listed on other grounds — and both are the first command tutorial 07 and PLUGIN.md tell a new user to type. If a once-per-repository skill is only reachable by someone who read the docs, that is the intended path, not a defect.

3. The ci case rested on the drift bug, which is now fixed. "Restates the CI docs page" also undersells it — the docs page does not do repository discovery, does not resolve a version pin, and does not emit per-depth globs, all of which exist because the action's input expansion has globstar and nullglob off.

Happy to revisit when the budget actually binds — at that point the honest move is probably trimming the four long descriptions rather than removing capability, since lint-tasks (307) and check-skill (289) cost more than either candidate for removal (219 / 218).

---

make format / check / lint (332) / test (4,052 passed, 4 skipped) clean.