feat!: ship one binary — bundle exec-harness and memtrack, drop the LD_PRELOAD hack by moha-bekh · Pull Request #531 · CodSpeedHQ/codspeed

moha-bekh · 2026-09-07T15:37:43Z

exec-harness and memtrack are compiled into codspeed and reached as hidden subcommands, and
exec-harness no longer injects libcodspeed_preload.so. The two are one change: a statically
linked musl binary cannot be preloaded into a glibc process, so the single binary was blocked on
removing the preload.

One release artifact instead of three, 11.0 MB compressed against 13.5 MB today. The download
machinery and both installer pins are gone.

Three things worth a reviewer's attention:

The measurement baseline shifts. The measured region now starts in exec-harness before the
fork, so it also covers fork/exec/wait and the child's pre-main startup. It is a constant
~940k Ir per exec-harness benchmark, not a percentage — stable to 0.22% across a 1000× range of
benchmark size. Shipping as is: no forced baseline, no history surgery.
Attribution rests entirely on the spawn-chain walk. Children no longer self-label with the
benchmark URI, since LD_PRELOAD is what used to be inherited by every descendant.
setcap now lands on codspeed itself, so memtrack's five capabilities sit on the CLI.
They are +ep with no inheritable set, so a spawned benchmark does not receive them.

Also removes the user-facing "CPU Simulation mode does not support statically linked binaries"
error: nothing is injected into the benchmarked executable any more.

Closes COD-3218
Closes COD-3440

codspeed · 2026-09-07T15:44:50Z

Merging this PR will degrade performance by 65.7%

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran on different hardware

Its base and head results were measured on machines with a different CPU, core count or memory, so the change might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

❌ 1 (👁 1) regressed benchmark
✅ 40 untouched benchmarks
⏩ 6 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
👁	CPU Simulation	`sleep 1`	1.6 ms	4.6 ms	-65.7%

_{Comparing spike/cod-3440-memtrack-musl (2673cb3) with main (ae2799e)}

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

greptile-apps · 2026-09-17T14:41:21Z

Confidence Score: 5/5

[High impact] The reviewed changes appear safe to merge; no new blocking issue was established.

Summary

The PR bundles exec-harness and memtrack into codspeed, removes their downloads, and replaces benchmark preloading with parent-controlled measurement.

Exec benchmarks run through the bundled harness without a preload library.
Memory runs use bundled memtrack from a cached, capability-bearing runner copy.
CodSpeed ships one runner executable with both internal tools built in.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI["codspeed"] --> Harness["Hidden exec-harness command"]
  Harness --> Start["Start measurement"]
  Start --> Child["Spawn and wait for benchmark"]
  Child --> Stop["Stop measurement"]
  Stop --> Declare["Declare child PID and benchmark URI"]
  CLI --> Copy["Cached copy with file capabilities"]
  Copy --> Memtrack["Hidden memtrack command"]
  Memtrack --> Track["Track benchmark memory"]

_{Reviews (14) · Last reviewed commit: "feat(instrument-hooks): declare benchmar..." · Reviewed by Greptile}

`self_exe()` resolves the binary that internal subcommands are re-invoked through, and the memory executor hands that same path to `sudo setcap <caps>+ep` so the capabilities land on the binary that is actually exec'd. Reading an environment variable there means anyone able to set one variable chooses which file receives CAP_SYS_ADMIN and CAP_BPF. The override exists for the tests, where `current_exe()` is the test harness and cannot dispatch a subcommand. Nothing in production sets it -- the doc comment justified it with a launcher scenario that has no caller. Putting it behind `cfg(test)`, constant included, removes the escalation path outright while keeping the tests working; a release build now always resolves `current_exe()`. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`codspeed memtrack` and `codspeed exec-harness` are a re-exec of this binary and share nothing with the runner, but they were dispatched at the bottom of `run()` -- after the profile config is loaded, after the API client is built, and after `DiscoveredProjectConfig::discover_and_load` walks the filesystem. That last one is the problem: the re-exec runs in the benchmark's working directory, which is the user's project. A malformed `codspeed.yaml` there aborts the subcommand, so a measurement fails for a reason that has nothing to do with the measurement, and a `--config` given to the outer run is not forwarded to the inner one to override it. Move them into `run_internal`, called right after `Cli::parse()`. The logger match loses its internal arms for the same reason it had them. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The released Linux artifacts are `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and nothing in CI built either: a break in the argp stub, in the kernel-header paths or in the aarch64 `-lgcc` link flag would have surfaced for the first time during a tag-triggered release. The throwaway spike workflow used to cover this and was deleted with the spike. Both legs build on a native runner, with no environment variables, which is also what keeps `.cargo/config.toml` honest -- it has to carry the whole recipe on its own. The assertions are `readelf`-based rather than a `file` string, since rustc emits a static-PIE for x86_64 musl and spells it differently from aarch64, and `codspeed exec-harness --version` / `codspeed memtrack --version` answer only if both CLIs really are linked in. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`self_exe()` resolves the binary that internal subcommands are re-invoked through, and the memory executor hands that same path to `sudo setcap <caps>+ep` so the capabilities land on the binary that is actually exec'd. Reading an environment variable there means anyone able to set one variable chooses which file receives CAP_SYS_ADMIN and CAP_BPF. The override exists for the tests, where `current_exe()` is the test harness and cannot dispatch a subcommand. Nothing in production sets it -- the doc comment justified it with a launcher scenario that has no caller. Putting it behind `cfg(test)`, constant included, removes the escalation path outright while keeping the tests working; a release build now always resolves `current_exe()`. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`codspeed memtrack` and `codspeed exec-harness` are a re-exec of this binary and share nothing with the runner, but they were dispatched at the bottom of `run()` -- after the profile config is loaded, after the API client is built, and after `DiscoveredProjectConfig::discover_and_load` walks the filesystem. That last one is the problem: the re-exec runs in the benchmark's working directory, which is the user's project. A malformed `codspeed.yaml` there aborts the subcommand, so a measurement fails for a reason that has nothing to do with the measurement, and a `--config` given to the outer run is not forwarded to the inner one to override it. Move them into `run_internal`, called right after `Cli::parse()`. The logger match loses its internal arms for the same reason it had them. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The released Linux artifacts are `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and nothing in CI built either: a break in the argp stub, in the kernel-header paths or in the aarch64 `-lgcc` link flag would have surfaced for the first time during a tag-triggered release. The throwaway spike workflow used to cover this and was deleted with the spike. Both legs build on a native runner, with no environment variables, which is also what keeps `.cargo/config.toml` honest -- it has to carry the whole recipe on its own. The assertions are `readelf`-based rather than a `file` string, since rustc emits a static-PIE for x86_64 musl and spells it differently from aarch64, and `codspeed exec-harness --version` / `codspeed memtrack --version` answer only if both CLIs really are linked in. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

GuillaumeLagrange

@not-matthias can you do a first round of review 🙏 ? Let's ignore the LD_PRELOAD removal overhead for the exec-harness simulation, I'll spec this in a dedicated ticket and we'll do it before we merge and release this.

not-matthias

Overall pretty good, just a few comments on how to better structure the code/comments

And a few notes on how to best structure the PRs for reviewers:

Try to keep the changes as minimal as possible
Do not modify unrelated comments/code -> should be done in separate PRs for easier review
Remove obvious/LLM-written/bloated comments (using the deslop skill)
you should also review the PR before putting it into review

GuillaumeLagrange

See comments for details.

I also see that there are unresolved comments.

When you handle a review, all comments should either be

resolved and handled properly (very important to not silently resolve comments)
answered with either counter arguments or questions so the reviewer can either accept your arguments or open a discussion

Finally, there are 35 commits in the PR, you may want to re-do commits before the next round of reviews

GuillaumeLagrange

See comments for details.

I also see that there are unresolved comments.

When you handle a review, all comments should either be

resolved and handled properly (very important to not silently resolve comments)
answered with either counter arguments or questions so the reviewer can either accept your arguments or open a discussion

Finally, there are 35 commits in the PR, you may want to re-do commits before the next round of reviews

moha-bekh · 2026-09-21T09:51:24Z

Yeah I didn't see them since I needed to click on the see more button, I will check If they are already resolved, and try to reduce the amount of commits by squashing where it's coherent

not-matthias

we're getting there!

not-matthias

we're getting there!

GuillaumeLagrange

Still quite a few things to change but we're getting there. The main point is how we are handling the setcap copy for memtrack, which seems quite feeble right now.

GuillaumeLagrange

lgtm!

musl has no `close_range` wrapper, so call the syscall directly. libbpf-sys vendors elfutils, whose `configure` requires `argp`, `obstack` and `fts`, none of which musl provides even though a libelf-only build never calls them: `.cargo/config.toml` seeds the autoconf cache to skip those checks, adds an `argp.h` stub through `CPATH`, and points Debian's `musl-gcc` back at the kernel UAPI headers. aarch64 also links libgcc for the outline-atomic helpers. The libc symbol resolution test now reads the Ubuntu libc path and only runs in CI, since a static musl test binary maps no libc of its own. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

Move each binary's logger setup, argument parsing and dispatch into a `cli::run_cli` library function that takes its argv, leaving `main` a thin wrapper. This lets the runner call either CLI in-process with the same logging. No behavior change. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

In simulation mode, exec-harness ran each target with an `LD_PRELOAD` library that toggled instrumentation from inside it. That needed a C build step and failed for executables that ignore `LD_PRELOAD`, such as static binaries. Simulation now runs targets like memory mode does, and the runner enables subprocess tracking for exec-harness runs so valgrind follows into the target. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

exec-harness was released on its own and downloaded at a pinned version the first time a run had exec targets. Run it as a hidden `codspeed exec-harness` subcommand instead, so it always matches the runner and needs no download. The exec-harness crate is no longer released. Internal subcommands now dispatch before any runner setup: they run in the benchmark's working directory, where loading a stray `codspeed.yaml` could abort the measurement. `codspeed samply` is hidden from the help like the others. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

memtrack was released on its own as `codspeed-memtrack` and installed at a pinned version by the memory executor's setup. Link it into the runner with its eBPF feature and run it as a hidden `codspeed memtrack` subcommand, so it always matches the runner and needs no download. With exec-harness already bundled, the binary installer has no users left and is removed. memtrack is no longer released. The eBPF capabilities are granted to a copy of the runner executable rather than the runner itself, so the runner never runs in glibc's secure-execution mode, which would strip `LD_*` variables from the benchmark environment. The copy lives under `~/.cache/codspeed/memtrack`, keyed by the executable's hash, and is created before any sudo run can leave a root-owned `~/.cache`. Copies of other builds unused for a day are removed. `codspeed setup memtrack-path` prints the copy's path. Because that copy gets `setcap`, the `CODSPEED_SELF_EXE` override of the re-executed binary is now test-only. `codspeed setup status` reports no separate memtrack tool any more, and shows privilege status for executors without one. Building the runner now builds memtrack's vendored libbpf, so CI installs the bpf dependencies for it, the release gets memtrack's build dependencies, and a new CI job builds the musl targets the runner ships. Closes COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

With memtrack linked into the runner, every memory run hashes the runner binary to locate its capability-holding copy and runs memtrack from it. In the debug build both are unoptimized: hashing the 376 MB binary alone takes about 5.6s per run, which took the memory shards from ~10s to ~42s. Build and set up the executor tests with the release profile; the memory shards are back to 8-10s. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

Add set_executed_benchmark_for_pid, which passes an explicit pid to instrument_hooks_set_executed_benchmark instead of the calling process' own. set_executed_benchmark keeps its behavior and delegates to it. Bump instrument-hooks, whose valgrind instrument now writes a "Benchmark pid: <pid>" desc line in the dump part when that pid is not the calling process'. This also brings thread-safe C API exports and the callgrind_toggle_collect helper. Refs COD-3722 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

moha-bekh changed the title ~~COD-3218 / COD-3440: remove the LD_PRELOAD hack and port exec-harness to musl~~ feat!: ship one binary — bundle exec-harness and memtrack, drop the LD_PRELOAD hack Sep 17, 2026

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch from ad8eca1 to 2d8d983 Compare September 17, 2026 14:33

moha-bekh marked this pull request as ready for review September 17, 2026 14:33

greptile-apps Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread src/executor/memory/setup.rs Resolved

Comment thread src/cli/mod.rs Resolved

Comment thread crates/exec-harness/build.rs Resolved

Comment thread .github/workflows/ci.yml Resolved

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch from 3294b83 to 8e2d7e3 Compare September 17, 2026 15:34

moha-bekh requested a review from GuillaumeLagrange September 17, 2026 15:49

GuillaumeLagrange requested a review from not-matthias September 17, 2026 16:07

GuillaumeLagrange reviewed Sep 17, 2026

View reviewed changes

not-matthias requested changes Sep 17, 2026

View reviewed changes

moha-bekh requested a review from not-matthias September 18, 2026 09:58

art049 requested a review from GuillaumeLagrange September 21, 2026 08:24

GuillaumeLagrange requested changes Sep 21, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Resolved

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch from 96d9261 to ac9a23c Compare September 21, 2026 13:05

moha-bekh requested a review from GuillaumeLagrange September 21, 2026 13:10

not-matthias reviewed Sep 21, 2026

View reviewed changes

Comment thread src/executor/memory/executor.rs Resolved

Comment thread src/executor/memory/setup.rs Resolved

Comment thread src/executor/memory/setup.rs Resolved

not-matthias reviewed Sep 21, 2026

View reviewed changes

moha-bekh requested a review from not-matthias September 22, 2026 10:22

lvaroqui mentioned this pull request Oct 1, 2026

feat(exec-harness): declare the benchmarked command's pid #559

Open

greptile-apps Bot reviewed Oct 4, 2026

View reviewed changes

Comment thread src/executor/config.rs Resolved

GuillaumeLagrange requested changes Oct 5, 2026

View reviewed changes

lvaroqui force-pushed the spike/cod-3440-memtrack-musl branch from b933455 to 3a766f1 Compare October 7, 2026 09:41

GuillaumeLagrange requested changes Oct 8, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Resolved

moha-bekh requested a review from GuillaumeLagrange October 8, 2026 12:11

lvaroqui force-pushed the spike/cod-3440-memtrack-musl branch from 2f56c2f to ecbfb5b Compare October 8, 2026 13:34

GuillaumeLagrange requested changes Oct 8, 2026

View reviewed changes

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch 2 times, most recently from 69769df to b774221 Compare October 9, 2026 09:47

moha-bekh requested a review from GuillaumeLagrange October 9, 2026 11:44

GuillaumeLagrange approved these changes Oct 9, 2026

View reviewed changes

moha-bekh requested a review from not-matthias October 9, 2026 12:58

GuillaumeLagrange mentioned this pull request Oct 12, 2026

perf: use mimalloc as the runner's global allocator #581

Merged

lvaroqui force-pushed the spike/cod-3440-memtrack-musl branch 3 times, most recently from 0ce29e7 to 56ed550 Compare October 14, 2026 13:28

greptile-apps Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread src/executor/memory/setup.rs Resolved

Comment thread src/cli/mod.rs Resolved

Comment thread src/cli/setup.rs Resolved

GuillaumeLagrange mentioned this pull request Oct 28, 2026

fix(exec-harness): compile node analysis workloads eagerly with sparkplug #595

Open

GuillaumeLagrange force-pushed the spike/cod-3440-memtrack-musl branch 3 times, most recently from d5b22ed to 5840369 Compare November 2, 2026 17:21

not-matthias mentioned this pull request Nov 2, 2026

fix(memtrack): quiet expected libbpf warnings in a user namespace #602

Merged

GuillaumeLagrange removed the request for review from not-matthias November 2, 2026 23:02

not-matthias approved these changes Nov 3, 2026

View reviewed changes

GuillaumeLagrange and others added 7 commits November 3, 2026 11:57

GuillaumeLagrange force-pushed the spike/cod-3440-memtrack-musl branch from f2907d4 to 2673cb3 Compare November 3, 2026 10:59

Conversation

moha-bekh commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 65.7%

Performance Changes

Footnotes

Uh oh!

greptile-apps Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

not-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

moha-bekh commented Sep 21, 2026

Uh oh!

not-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

not-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

moha-bekh commented Sep 7, 2026 •

edited

Loading

codspeed Bot commented Sep 7, 2026 •

edited

Loading

greptile-apps Bot commented Sep 17, 2026 •

edited

Loading