[Cedric Vidal (cedricvidal)]

Summary

Add Taxonome, a standalone toolkit for evaluating agent trajectories against YAML-authored taxonomies:

• Reusable engine and catalog: taxonomy-engine provides loading, validation and execution; ax-taxonomy supplies the agent-behavior catalog and deterministic handlers; the taxonome CLI composes both. Custom deterministic handlers can be registered through explicit plugins.
• Typed, grounded assessments: distinguish observed, negative, indeterminate and inapplicable results; validate occurrence grouping, evidence roles, cardinality, JSONPaths, types and ordering against the input trajectory.
• Strict ATIF ingestion: validate and upcast ATIF v1.6–v1.8, retain original-source provenance separately from canonical hashes, and safely ingest existing Scope run archives.
• Ax evaluation and optimization: run prompt-based elements through the Copilot SDK with one bounded Ax correction; evaluate curated datasets, optimize prompts and replay compatible artifacts without optimizer repair loops.
• CLI and readouts: support single inputs, newline/NUL file lists, mixed plain/archive batches, bounded concurrency, resumable results, cancellation and terminal/plain progress. Generate deterministic JSON/Markdown readouts with validated evidence and explicit coverage.
• Website guide and canonical references: one practical workflow/authoring guide and four Reference pages cover commands, formats, elements and handlers. The website generates its catalog, filenames and complete YAML disclosures from the committed taxonomy; allowlisted implementation panels read source text without importing the engine, executing plugins or initializing a model. Keyboard-accessible, high-contrast panels and focused browser tests accompany the docs.

The shipped catalog includes prompt-based Skipped Invocation, Wrong Discovery and Content Misused, plus deterministic Discovered, Malformed Invocation and Timeout. Taxonome runs without Scope API, Judge, database, queue or token-manager services; deterministic evaluation needs no model credentials. Production processing/report integration is not included.

Demos

Before

N/A: new standalone Taxonome CLI.

After

Earlier recording from #1529 — not newly recorded or re-executed for this PR:

terminal.mp4

21-beat walkthrough covering deterministic contracts, plugins, plain/archive provenance, prompt evaluation/optimization/replay, and readouts. Paid-call accounting and command versions belong to that earlier recording, not this PR's validation. The documentation publication below does not add or replace media and does not claim new live-model acceptance.

Testing

November 4 documentation publication: commit cbca7d010077e72484e1c818944bd73ebcaf6371 contains exactly the approved 21 documentation/catalog paths, preserving the refreshed parent's contributor tests and navigation. No runtime taxonomy identity, media or integration changes are included.

Command / check	Result
pnpm --dir website test	Passed: 30 tests, including canonical catalog/source safety, keyboard plugin and existing contributor regressions.
pnpm exec vitest run packages/ax-taxonomy/src packages/taxonomy-engine/src/plugins.test.ts --reporter=default --testTimeout=15000	Passed: 41 tests / 6 files, including catalog definitions, deterministic handlers and plugin contracts; no model calls.
SITE=https://microsoft.github.io BASE_PATH=/scope pnpm --dir website build	Passed: 232 static pages and Pagefind search index.
pnpm --dir website exec playwright test taxonome.spec.ts --config "$HOME/.copilot/session-state/a451d4e9-4a8d-4a6c-b948-215c12a6a7a8/files/docs-publish-20261104/playwright.config.ts"	Passed: 12/12, using a session-only config with installed headless Chrome and the existing light/dark × 1280/320 matrix; checks all five pages, source fidelity, keyboard access, axe and overflow.
pnpm --dir website install --frozen-lockfile --ignore-scripts	Passed; no lockfile changes.
pnpm headers:check	Passed: 1,090 in-scope files.
pnpm notice and pnpm notice:check	Passed; generated notices unchanged.
git diff HEAD --check and git diff --cached --check before commit	Passed.
pnpm --filter taxonome exec tsc --noEmit -p ../../website/tsconfig.json	Baseline limitation: 8 diagnostics, reproduced with the same codes/messages against the unchanged committed website at 07a0eae1: one existing implicit-any parameter in astro.config.mjs and seven upstream starlight-openapi diagnostics. No new docs diagnostics. The website has no standalone typecheck script or local tsc; this uses the already-installed workspace compiler.

Fresh checks on cbca7d010077e72484e1c818944bd73ebcaf6371: 17 passed, 9 skipped, none failed or pending. CI run passed application/Windows builds, Node 22 unit tests, lint, headers/NOTICE, CLI bundle integration, both ACP worker integrations and queue recovery. Pages build passed all 30 site unit tests, built 232 pages and passed the full 104/104 website accessibility/privacy browser tests. PR deployment, non-applicable LLM/Gateway jobs and Daily Test Improver jobs were skipped. These fresh CI results are separate from the local baseline-reproduced TypeScript limitation above; they do not imply that standalone website typechecking passes.

Earlier foundation validation, retained as historical evidence:

Command / check	Result
pnpm exec vitest run packages/taxonomy-engine/src packages/ax-taxonomy/src apps/taxonome/src --reporter=default --testTimeout=15000	Passed: 438 tests / 42 files, including ATIF/archive conformance, hashes, typed evidence, file-list/process contracts and mocked Ax correction behavior.
pnpm --filter shared --filter telemetry --filter taxonomy-engine --filter ax-taxonomy --filter taxonome --filter taxonome-custom-handler-example build	Passed: all six packages compile.
pnpm --filter taxonomy-engine lint && pnpm --filter ax-taxonomy lint && pnpm --filter taxonome lint	Passed: strict type checks.
pnpm --filter taxonome-custom-handler-example test:handler	Passed: 2 tests.
pnpm --filter static-prompt-evals test:ts	Passed: 37 tests / 7 files, including production prompt-composition contracts.
pnpm --filter static-prompt-evals validate-data	Passed: 222 approved cases, 15 quality targets and 3 standalone SDK targets.
pnpm --filter static-prompt-evals typecheck	Passed.
pnpm install --frozen-lockfile --ignore-scripts	Passed.
pnpm notice and pnpm notice:check	Passed; notices were current.
Built-CLI file-list/TTY/archive/resume/readout acceptance harness	Passed 30/30 scenarios with 10 copied playbook trajectories, 30 primary deterministic assessments and zero model calls. Covers newline/NUL lists, whitespace/Unicode paths, aliases, distinct input origins, partial errors, overlapping archives, archive root input, resume, readout coverage and cursor restoration.

The earlier foundation results were not rerun wholesale for this documentation publication. Committed model outputs and optimization evidence retain their historical provenance and are not new accuracy claims. No inference, optimization or media publication was performed for this update.

Documentation and compatibility

The new practical website guide links four references: CLI, taxonomy/result format, generated default elements, and handlers/plugins. The catalog is generated during website builds rather than maintained as a second element list. Sidebar links, source-triggered Pages filters, website-only YAML dependency/lockfile and maintenance documentation are included. These pages are PR source, not a claim that the public Pages deployment already contains them.

Taxonome architecture and usage, the normative trajectory taxonomy format, package READMEs and a runnable custom-handler example document authoring, APIs, commands and evidence contracts. Package architecture and the static-prompt inventory are updated.

This is an experimental authored-YAML interface. run, eval and optimize require a taxonomy; v2 optimization artifacts explicitly reject historical v1 artifacts. Plugins are opt-in and run with process permissions, not in a sandbox. Archive validation reuses the existing run-state schema through the narrow shared/schemas/request export. No database migration or existing Scope CLI behavior change is required. The documentation commit does not rename the shipped taxonomy or change runtime contracts.

Checklist

• If Portal features changed, keep CLI capabilities in sync. N/A: no Portal changes.
• If Portal components changed, update their Storybook stories. N/A.
• If database changes require a migration, include up() / down() and keep it CosmosDB-compatible. N/A: no database changes.
• If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed.
• Video showing the behavior before the suggested change. N/A: new standalone CLI.
• Video showing the behavior after the suggested change. Earlier recording linked and attributed above.

[github-actions]

Test Results (Node.js 22)

test: Run #429

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3727	3727	0	0	0	0	0	1m30s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #511

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3754	3754	0	0	0	0	0	1m52s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #520

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3754	3754	0	0	0	0	0	1m32s

🎉 All tests passed!

Github Test Reporter