[not-matthias]

TLDR: Rounds within one process are correlated (JIT tier-up, heap sizing, allocator state), so walltime results depend on the process they ran in. Each walltime benchmark now runs in 3 fresh processes (forks) by default, and their rounds are pooled into one result (tinybench and Vitest 5).

Fork count, highest priority first:

1. CODSPEED_WALLTIME_FORKS=<n>
2. Vitest 5 per-benchmark: bench(name, { forks }, fn) or bench.compare(..., { forks })
3. Integration option: withCodSpeed(bench, { forks }), codspeedPlugin({ forks })
4. 3

CODSPEED_PROCESS_ISOLATION=false runs everything in one process.

How it works

• The parent re-runs the benchmark command (execPath + execArgv + argv, so the runner's V8 flags survive) once per fork, with CODSPEED_ISOLATED_BENCHMARK=<uri>. The fork measures only that benchmark and writes its samples to a temp file. The parent pools them and writes the walltime result.
• Vitest workers spawn the main process's command, and each fork narrows benchmark discovery to the benchmark's file.

Cost (examples/with-typescript-esm tinybench, 9 benchmarks): 77 s with isolation off, 140 s with 1 fork, 412 s with 3 forks. Each fork pays process startup.

Review notes

• Simulation/memory runs, benchmark.js and Vitest 3/4 don't fork.
• Vitest 5: its own report shows zeroed statistics for forked benchmarks; the CodSpeed results hold the pooled rounds. The test timeout is multiplied by the env or plugin fork count. A per-benchmark forks above that count can still time out.
• Vitest's BenchOptions type doesn't include forks, so the per-benchmark option needs a cast in TypeScript.

Closes COD-3960

[codspeed]

Merging this PR will regress 4 benchmarks

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 14 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 8 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

ℹ️ 16 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 23 improved benchmarks
❌ 4 regressed benchmarks
✅ 243 untouched benchmarks
⏩ 56 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	test_recursive_fibo_10	339.7 µs	690.8 µs	-50.83%
❌	CPU Simulation	wait 500ms	29.7 ms	53 ms	-43.9%
❌	CPU Simulation	wait 1sec	59.1 ms	97.6 ms	-39.48%
❌	Memory	one	1.5 KB	2 KB	-26.19%
⚡	CPU Simulation	switch 1	651.7 µs	305.2 µs	×2.1
⚡	CPU Simulation	recursive fibo 10	677.4 µs	333.1 µs	×2
⚡	Walltime	switch 2	120 ns	72 ns	+66.67%
⚡	Walltime	test_iterative_fibo_10	120 ns	84 ns	+42.86%
⚡	Walltime	test_iterative_fibo_10	120 ns	84 ns	+42.86%
⚡	Walltime	test sync baz 10	120 ns	84 ns	+42.86%
⚡	CPU Simulation	switch 1	586.1 µs	418.7 µs	+39.97%
⚡	Walltime	switch 2	96 ns	72 ns	+33.33%
⚡	Memory	wait 1sec	28.7 KB	22.7 KB	+26.61%
⚡	Walltime	short body	2 µs	1.6 µs	+25.95%
⚡	Walltime	short body	2 µs	1.6 µs	+25.95%
⚡	Walltime	test sync baz 10	120 ns	96 ns	+25%
⚡	Walltime	short body	2 µs	1.6 µs	+23.31%
⚡	Walltime	short body	1.9 µs	1.6 µs	+23.08%
⚡	CPU Simulation	recursive fibo 10	676.4 µs	557.1 µs	+21.41%
⚡	Walltime	short body	1.9 µs	1.6 µs	+18.66%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-3960-implement-process-isolation-in-codspeed-node (c376bfd) with main (9c93085)}

Footnotes

1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The stack appears safe to merge, with this assessment depending on PR #111’s changes to walltime execution.

Summary

Adds fresh-process walltime measurements for Tinybench and Vitest 5, with three forks by default. The parent pools their samples into one CodSpeed result.

• Tinybench walltime benchmarks run in fresh processes and pool their rounds.
• Vitest 5 benchmarks run in fresh processes and pool their rounds.

Diagram

sequenceDiagram
  participant Parent as Benchmark parent
  participant Child as Fresh process
  participant File as Temporary samples
  participant Results as CodSpeed results
  loop Each benchmark and fork
    Parent->>Child: Restart command with selected URI
    Child->>Child: Measure selected benchmark
    Child->>File: Write round samples
    Child-->>Parent: Exit
    Parent->>File: Read samples
  end
  Parent->>Results: Write pooled statistics

Loading

_{Reviews (2) · Last reviewed commit: "feat(vitest): run walltime benchmarks in..." · Reviewed by Greptile}

[GuillaumeLagrange]

olgtm