fix(ci): surface test failures in logs and de-flake perf tests - #62
Merged
Conversation
Collaborator
|
Ouch... that was really poor earlier. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two separate issues on
main:1. Test failures are invisible in CI logs.
scripts/test-summary.jsran each package withstdio: "pipe"and printed only scraped counts, discarding all failure output. The failing run on main showed just❌ @codemcp/knowledge-mcp-server: 21/22 passed.2. Worse — the script could report red as green. Pass/fail was derived from scraped numbers, and the regexes didn't match vitest output containing skipped tests (
Tests 1 failed | 22 passed | 2 skipped (25)). Unmatched packages were silently dropped from the report and the totals. Verified against the old script with a deliberately failing test: it printed🎉 All tests completed successfully!and exited 0.3. Flaky perf assertions in
packages/mcp-server/src/__tests__/performance.test.ts:expect(time).toBeLessThan(10)on a single cold sample, including module init and JIT warm-up — 2.29ms locally, under 4x headroom on a shared CI runner.expect(memoryDiff).toBeLessThan(10MB)on a singleheapUseddelta — reported-0.36MBlocally, i.e. GC noise dominated the signal entirely.Changes
testnow runsturbo run buildfirst.packages/cliintegration tests execdist/index.js, but the root script bypassed turbo so its existingdependsOn: ["build"]never applied — a staledistproduced failures unrelated to the current source.dist/to.prettierignore(root + per package). Prettier was checking build artifacts; CI only passes today becauseformat:checkhappens to run beforebuild.Verification
22/25 passed, 1 failed, 2 skipped).format:check:all,lint:all,build,test— 265 tests, 4 consecutive runs.Not verified: behaviour on a genuinely contended CI runner. The headroom argument is inference from local numbers, not a measurement.