Skip to content

test(scripts): the CLI-version gate's rules, which have passed while wrong twice - #144

Merged
viztor merged 1 commit into
mainfrom
test/vendor-version
Oct 31, 2026
Merged

viztor merged 1 commit into
mainfrom
test/vendor-version

Conversation

@viztor

@viztor viztor commented Oct 31, 2026

Copy link
Copy Markdown
Owner

The gate compares the version the plugin claims against the one the vendor publishes, and every copy in the tree against the constant. It runs on a schedule, and its failure mode is the worst kind: it passes while reading nothing.

It has already done so twice:

  • the first version only scanned for literals and could not see the constant at all;
  • the second only scanned opencode/<version> and could not see a bare copy — three e2e comments were two patches stale while it reported OK.

Both were found by a control run, not by a test. So the rules are now scripts/vendor-version.ts — a function of its input — and the test holds them to:

  • the constant is found, and its absence is undefined rather than a default, because "read nothing" and "passed" have to be distinguishable;
  • a bare mention matches 1.19.5 and v1.19.5, and does not match opencode/1.19.5 — the look-behind whose absence made the second version double-count every prefixed copy;
  • a stale patch of the pinned series is caught while another major.minor is the literal scan's business;
  • the disagreement list names versions and files, sorted, and is empty when everything agrees.

Planted the look-behind's removal — the exact historical bug — and the case fails.

A detail worth keeping

The test had to build its stale versions at run time. Writing 1.19.3 in the file makes the gate it describes fail, because the scan reads test/ too. It caught that on the first run; the digits are now assembled from the series, and the file says why in its first line.

gate result
suite 1224 passed / 82 files (+1)
vp check 0 errors
check-vendor-version ok — 10 copies, 55 bare mentions agree

…wrong twice

The gate compares the version the plugin claims against the one the vendor
publishes, and every copy in the tree against the constant. It runs on a
schedule, and its failure mode is the worst kind: it PASSES while reading
nothing.

It has already done so twice. The first version only scanned for literals and
could not see the constant at all; the second only scanned `opencode/<version>`
and could not see a bare copy (three e2e comments were two patches stale while
it reported OK). Both were found by a control run, not by a test.

So the rules are now `scripts/vendor-version.ts` — a function of its input — and
`test/vendor-version.test.ts` holds them:

- the constant is FOUND, and its absence is `undefined` rather than a default,
  because "read nothing" and "passed" have to be distinguishable;
- a bare mention matches `1.19.5` and `v1.19.5`, and does NOT match
  `opencode/1.19.5` — the look-behind whose absence made the second version
  double-count every prefixed copy;
- a stale patch of the pinned series is caught while another major.minor is the
  literal scan's business;
- the disagreement list names versions and files, sorted, and is empty when
  everything agrees.

Planted the look-behind's removal — the exact historical bug — and the case
fails.

**The test had to build its stale versions at run time.** Writing `1.19.3` in
the file makes the gate it describes fail, because the scan reads `test/` too;
the digits are assembled from the series instead, and the file says in the first
line why.
@viztor
viztor merged commit 3391b35 into main Oct 31, 2026
3 checks passed
@viztor
viztor deleted the test/vendor-version branch October 31, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant