Repository navigation
fix(contract): make the contract lanes report what they ran - #41
Merged
offendingcommit merged 11 commits intoOct 24, 2026
Merged
Conversation
The drift lane fed the classifier only the last 25 lines of the suite log. When both contract modules failed to import the host (No module named 'ruamel'), the ERROR headers were cut, no failure block parsed, and the classifier printed "No contract failures against upstream." and exited 0. These tests use the real captured shape of that log, tail-only and full, plus Ran 0, a missing Ran line, and an unexplained non-zero exit. Five fail on today's classify_drift.py; the full-log case already passes because its ERROR headers survive. The two baseline tests gain a realistic Ran/EXIT trailer, since a log without one is no longer evidence that anything ran. Refs: #40 Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
classify_drift.py only knew about parsed FAIL/ERROR blocks, so a log whose blocks were missing read as clean. It now also files as new drift: - any "ImportError: Failed to import test module: X", naming X; - no "Ran N tests" line, or Ran 0; - no EXIT= line, or a non-zero EXIT with no failure block to explain it. A run whose only failure is baselined still exits non-zero and still classifies as known. _run-contract-suite now prints the whole log instead of tail -25, since the drift workflow classifies that output. The workflow's step summary keeps its own tail for people. Refs: #41 Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
DEPLOYED_HERMES_REVISION moves from f80f453 to 939e45c9 (v2026.9.11), the revision minilab deploys today. test.yml's hermes-deployed-contract job carries two literals for it (env and checkout ref), because actions/checkout reads the ref before any Python exists. Both the constant's docstring and test.yml said a release-contract test kept those literals equal to the constant. No such test existed: moving the constant alone left `just test` green. A harness test now asserts it, and fails on a constant that moves without the workflow. release.yml reads the constant at run time and compares it with the pinned checkout's HEAD, so it has no literal to move. Refs: #41 Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
The lanes ran the contract modules in the kit's venv, which lacks the host's dependencies, so both modules died importing hermes_cli (No module named 'ruamel'). The host's pyproject.toml alone does not resolve on v0.22.5 either, so its uv.lock is the faithful source (KTD4). provision-hermes-host now runs `uv sync --frozen` from the host checkout into a venv kept in this repository (the checkout is only read), with the first system interpreter satisfying the host's own requires-python, then installs the kit into it with --no-deps. Both lanes call it after the checkout is resolved, so CI's HERMES_AGENT_PATH path is covered too, and _run-contract-suite runs on that venv's python. The deployed pin 939e45c9 requires <3.14, so mise.toml adds Python 3.13.14 (the deployed image's 3.13 line) beside 3.14.9, and mise.lock is regenerated for linux-x64, linux-arm64 and macos-arm64. A scoped UV_PYTHON_DOWNLOADS=automatic was the alternative; it would bypass the locked toolchain the [env] block exists to enforce. Drop 3.13 when the pin moves to v0.22.5 (KTD7). Verified locally: pinned lane at 939e45c9 ran 17 tests, all passing, on 3.13.14. The upstream lane at v0.22.5 now imports the host and fails only on SCHEMA_VERSION (U5), which the classifier reports as new drift. Refs: #41 Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
test-context-engine-contract still ran in the kit venv, so pointed at a real host it hit the same import failure the full lanes just shed. It now provisions the named host through provision-hermes-host first. Against 939e45c9 it ran 3 tests, OK. Refs: #41 Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
Hermes v0.22.5 removed the PLUGIN-COMPAT pointer layer (upstream #126164), so hermes_state no longer re-exports SCHEMA_VERSION and the contract module failed to import against bf0203a7. Import it from hermes_state_common first and fall back to hermes_state for older hosts. SessionDB still comes from hermes_state. Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
2 tasks done
The lanes installed the kit into the host env with --no-deps, on the theory that the env should stay exactly the host's lock. The runtime image does not install the kit that way: its Dockerfile and plugin-refresh install it with its dependencies. The pinned host (939e45c9) happened to lock pyyaml, so the gap was invisible there. v0.22.5 dropped PyYAML for ruamel, so plugin_skill raised "No module named 'yaml'" and the upstream lane reported one error. Install the kit with its deps. Packages the host lock already satisfies are left alone; on v0.22.5 the only addition is pyyaml 6.0.3. Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
…comments provision-hermes-host read requires-python with whatever python3 was first on PATH; it now goes through the same uv indirection as every other recipe. Comments that narrated the change, restated code, or carried a plan label from another repository are gone; the ones recording an external constraint stay. Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
A lane whose unittest summary reads OK (skipped=N) exits 0 inside the suite; the lane recipe then fails on its own skip guard, but the drift workflow never appends the outer EXIT line, so the classifier saw a clean run. Cross-model code review (adversarial lens) found the path; it is not reachable today because the contract modules skip only at class level under HERMES_CONTRACT_REQUIRED. Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
Without a version input the action installs the newest mise at least a day old, so two runs of one commit, release jobs holding PRIVATE_ARTIFACTS_TOKEN included, could use different installers. The action keeps verifying the download against mise's signed checksums; a sha256 input would replace that signature check with a plain hash, so only the version is pinned. min_version follows the pin. Claude-Session: https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb
fix(contract): make the contract lanes pass on Hermes v0.22.5
offendingcommit
marked this pull request as ready for review
October 24, 2026 22:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
U4 of the Hermes v0.22.5 upgrade plan: the kit's contract lanes report what they actually ran, against what minilab actually runs. Stacked on #40 (mise toolchain).
tests/test_drift_classification.py::RunThatDidNotExecuteTests): with a real captured log in which both contract modules failed to import the host, the classifier printed "No contract failures against upstream." and exited 0. Five of the new tests failed on the oldclassify_drift.py. The full-log case already passed, because itsERROR:headers survive.scripts/classify_drift.py::_did_not_execute): files as new drift anyImportError: Failed to import test module: X(naming X), a missingRan Nline orRan 0, a missingEXIT=line, or a non-zeroEXIT=that no failure block explains. A run whose only failure is in the baseline still classifies as known. Each rule was removed in turn, and in each case its own test failed._run-contract-suitenow prints the full log instead oftail -25.DEPLOYED_HERMES_REVISION=939e45c9(v2026.9.11) intesting.pyand in both literals intest.yml. The docstring said a test kept those literals equal to the constant, but no such test existed.AbsentHostTests.test_the_deployed_contract_job_checks_out_the_same_revisionnow checks it.release.ymlreads the constant at run time, so it has no literal to change.justfileprovision-hermes-host): the host env comes from the host's ownuv.lockviauv sync --frozen, built into a venv in the kit repo, then the kit is installed with--no-deps. The interpreter is the first mise interpreter that satisfies the host'srequires-python.mise.tomladds Python 3.13.14 because 939e45c9 needs a version below 3.14. Remove it when the pin moves to v0.22.5 (KTD7).Decision: Python 3.13 comes from mise. The alternative was a scoped
UV_PYTHON_DOWNLOADS=automatic, rejected because it would bypass the locked toolchain thatmise.toml's[env]enforces.Local evidence:
just test: 267 tests, OK (17 skipped)just test-contract-pinnedagainst 939e45c9:contract receipt: pinned 939e45c91d751fadd94dcd1b873ac3cb44846213; executed=17on Python 3.13.14just test-contract-upstreamagainst v0.22.5 (bf0203a7): fails on theSCHEMA_VERSIONimport (fixed in U5). The classifier reports 2 new drift entries and exits 1 (R4).https://claude.ai/code/session_01VDtQ1pDTJksqAcSdrGePVb