Repository navigation
test(omp-compat): ignore skill-description pre-pass requests - #176
Merged
Merged
Conversation
OMP 18.13 compresses skill descriptions into routing hints at start-up and calls the configured model once per skill before the first agent turn. That adds a second POST /provider/v1/chat/completions, which the mock counted as an agent request, so `assert.equal(requestCount, 1)` fails against an unmodified main as soon as the runner installs @oh-my-pi/pi-coding-agent@latest (18.13.2). Count those pre-passes separately. They are recognised by body shape — user messages only, since an agent turn always carries a system prompt — rather than by matching OMP's prompt wording. They also keep the ordinary mock response while the `403 upgrade_required` fallback scenario is active, so the fallback stays the only path under test.
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
Memory benchmarkCompared base
Interpretation: Differences larger than their measured MAD: Stable RSS -1.0 MiB ± 0.4 MiB, Physical peak -0.6 MiB ± 0.5 MiB. Extension overhead above pi baseline
Values are medians of 6 alternating, paired runs. The value after Environment: pi
|
This was referenced Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mainis red since Oh My Pi 18.13: theOh My Pi compatibilityjob failsassert.equal(requestCount, 1)intest-omp-compat.mjs:338with2 !== 1. This is environmental, not a regression in this repository — the same assertion fails on an unmodifiedmainwhen the runner installs@oh-my-pi/pi-coding-agent@latest(dispatch run 40657231956).Root cause. OMP 18.13.0 (published 2026-10-26) compresses skill descriptions into one-line routing hints at start-up (
dist/extensibility/skill-descriptions.ts): for every skill it finds, it calls the configured model once — with a single user message, a 1024-token cap and no system prompt — before the first agent turn. The repository ships a skill (.agents/skills/refresh-model-catalog), so everyomp -prun now issues one extraPOST /provider/v1/chat/completionsthat the mock counted as an agent request. Confirmed locally:agent=1 prepass=0on OMP 18.12.6,agent=1 prepass=1on 18.13.2.Fix. Count pre-passes separately (
skillDescriptionRequestCount) and stop counting them as agent requests or storing them aslastRequest*. Detection is by body shape — every message is ausermessage, because an agent turn always carries a system prompt — instead of matching OMP's prompt wording, so a wording change upstream cannot silently break it. Pre-passes also keep the ordinary mock response while the403 upgrade_requiredfallback scenario is active; otherwise a 403 on the compression call would abort the pre-pass rather than exercise the transport fallback. The request-count assertion now reports the pre-pass count, so a future host change is diagnosable from the failure message.No behaviour change in the extension. The test still asserts exactly one agent request, its
Authorizationheader, its model, and its message array for every scenario (print mode, stored OAuth, stored API key,--api-key, env key, no credential, generate fallback with advisory).Verification
OMP_BIN=omp@18.12.6 node tests/test-omp-compat.mjs: PASS (prepass=0) — no regression on the last version that made CI green.OMP_BIN=omp@18.13.2 node tests/test-omp-compat.mjs: PASS (prepass=1) — the version that currently breaksmain.OMP_BIN=omp@18.13.2from the local bun install: PASS.npm test: PASS (full suite, 21 suites, 0 fail — run with OMP 18.13.2 installed locally, so[pi-local]and[omp-compat]both exercised the new host).npm run typecheck,npm run format:check,git diff --check: clean.Notes
src/or release metadata touched.