diff --git a/Cargo.lock b/Cargo.lock index fb9bd943a..ea80396be 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1206,6 +1206,7 @@ name = "fbuild-core" version = "2.5.47" dependencies = [ "async-trait", + "flate2", "fs2", "interprocess", "libc", diff --git a/README.md b/README.md index e08eff13e..0ba8a80a9 100644 --- a/README.md +++ b/README.md @@ -290,12 +290,85 @@ points: - URL-based package and library management, including GitHub `lib_deps` - Build, deploy, serial monitor, and emulator test workflows from one CLI - Cross-platform support on Windows, macOS, and Linux +- Host tools as Rust-built APE native executables: one binary per tool that + runs everywhere fbuild does, even on NixOS + ([details](#host-tools-ape-native-executables)) - Transparent architecture with Rust workspace internals documented under [`docs/architecture/`](docs/architecture/README.md) See [`docs/WHY.md`](docs/WHY.md) for the full rationale, benefits, and performance notes. +## Host Tools: APE Native Executables + +The tools fbuild runs on the host for each platform are moving to FastLED-built +Rust executables shipped in a single format: the cosmocc +[Actually Portable Executable](https://justine.lol/ape.html) (APE). That covers +upload/flash utilities, deployers and image/format helpers that would otherwise +come as Python scripts, per-OS vendor binaries, or packages that behave +differently on each host. Each tool is built once and runs as a native +executable on every host fbuild supports: Linux, macOS and Windows, x86_64 and +aarch64. There is one artifact per tool instead of one per OS and CPU, and one +behavior to test instead of a matrix. + +The goal per platform is to replace its third-party host tools with Rust tools +whose bugs FastLED fixes once and then ships as APE, so the fix reaches every +host at the same time. + +**Reference platform: NXP LPC8xx.** Its flasher is FastLED's fork of the Rust +[`probe-rs`](https://probe.rs). The fork carries the fixes that make the +LPC845-BRK's LPC-Link2 CMSIS-DAP firmware work: spec-default fallbacks when +the firmware doesn't answer `DAP_Info`, an explicit `DAP_Connect(Swd)` with +retry, and a CMSIS-DAP v1 HID transport over `nusb` (FastLED/fbuild#935, +FastLED/fbuild#936). The source is on the `tools` branch of +[`FastLED/framework-arduino-lpc8xx`](https://github.com/FastLED/framework-arduino-lpc8xx), +and `fastled-release-cross.yml` cross-builds it today as **six** per-host +release assets: Windows, Linux and macOS, each on x86_64 and aarch64. fbuild +pins and checksums all six in `crates/fbuild-deploy/src/probe_rs.rs`. The APE +target is one asset, built and pinned once, that runs on all of those hosts. + +**Next: WCH CH32V.** Its flashers, [`wlink`](https://github.com/ch32-rs/wlink) +(WCH-LinkE) and [`wchisp`](https://github.com/ch32-rs/wchisp) (USB ISP), are +already Rust. fbuild pins them per host in +`crates/fbuild-deploy/src/wlink.rs` and `wchisp.rs`, and `wlink` has assets for +only three hosts. Every other host needs `FBUILD_WLINK_PATH`. An APE build +removes that gap. + +### Guaranteed launch, nothing to install + +An APE tool launches anywhere fbuild does. Nothing has to be installed on the +host: no `ape` loader, no `binfmt_misc` registration, no shell support. That +guarantee is needed because most Unix hosts can't exec an APE file directly. +`posix_spawn` returns `ENOEXEC` ("Exec format error") unless an APE handler is +registered, and some distros, NixOS among them, ship none. Every fbuild spawn +detects the APE magic (`MZqFpD='`, `jartsr='`, `APEDBG='`) and runs the tool +through a loader: + +| Host | How an APE tool runs | +|---|---| +| Windows | Natively; an APE file is a valid PE. | +| Linux | fbuild extracts the loader embedded in the tool itself, checks it is a 64-bit ELF for the host CPU, and caches it in an owner-only, exec-capable directory. If no such directory exists (read-only `HOME`, `noexec` `/tmp`), it runs the loader from a sealed in-memory file instead. The tool's environment needs no `PATH`, `sh`, coreutils, `HOME` or `TMPDIR`. | +| macOS | x86_64: fbuild extracts the tool's embedded loader and turns it into a Mach-O (the prologue's header move). Apple Silicon: fbuild compiles the tool's embedded `ape-m1.c` once with `/usr/bin/cc` (Xcode Command Line Tools). Both are cached like on Linux. | + +Loader precedence is `FBUILD_APE_LOADER` (explicit override), then the loader +embedded in the tool, then `ape` on `PATH`, `/usr/bin/ape` and +`/usr/local/bin/ape`, then `/bin/sh`. Set `FBUILD_APE_CACHE_DIR` to choose +where extracted loaders are cached. Parallel builds and deploys are safe: fbuild +holds a fork lock while it writes a loader, so a concurrent first spawn can't +fail with `ETXTBSY` ("Text file busy"). + +### Adding an APE tool + +Build the Rust tool with cosmocc so the output is an APE, provision it like any +other package, and spawn it through `fbuild_core::subprocess::run_command*`, or +build the command with `fbuild_core::platform::process::command` / +`tokio_command` instead of `Command::new`. The tool then works like any other +executable, with no per-host special casing. A hello-world fixture and its build +script live in +[`crates/fbuild-core/data/ape-hello/`](crates/fbuild-core/data/ape-hello/README.md). +Implementation notes are in +[`crates/fbuild-core/src/platform/README.md`](crates/fbuild-core/src/platform/README.md). + ## CLI Usage The user-facing command reference is diff --git a/ci/platform_boundary_research.tsv b/ci/platform_boundary_research.tsv index 5d77da2ce..a49498662 100644 --- a/ci/platform_boundary_research.tsv +++ b/ci/platform_boundary_research.tsv @@ -1,11 +1,25 @@ path line kind normalized capability classification -crates/fbuild-core/Cargo.toml 49 native_dependency interprocess ipc host_mechanic -crates/fbuild-core/Cargo.toml 50 native_dependency socket2 ipc host_mechanic -crates/fbuild-core/Cargo.toml 56 target_dependency_table [target.'cfg(unix)'.dependencies] fs host_mechanic -crates/fbuild-core/Cargo.toml 58 native_dependency libc fs host_mechanic -crates/fbuild-core/Cargo.toml 60 target_dependency_table [target.'cfg(windows)'.dependencies] fs host_mechanic -crates/fbuild-core/Cargo.toml 68 native_dependency windows-sys fs host_mechanic -crates/fbuild-core/src/platform/executable.rs 56 native_path std::env::current_exe host_executable host_mechanic +crates/fbuild-core/Cargo.toml 52 native_dependency interprocess ipc host_mechanic +crates/fbuild-core/Cargo.toml 53 native_dependency socket2 ipc host_mechanic +crates/fbuild-core/Cargo.toml 59 target_dependency_table [target.'cfg(unix)'.dependencies] fs host_mechanic +crates/fbuild-core/Cargo.toml 61 native_dependency libc fs host_mechanic +crates/fbuild-core/Cargo.toml 63 target_dependency_table [target.'cfg(windows)'.dependencies] fs host_mechanic +crates/fbuild-core/Cargo.toml 71 native_dependency windows-sys fs host_mechanic +crates/fbuild-core/src/platform/executable.rs 167 native_path std::env::current_exe host_executable host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 11 native_path std::os::unix::fs::PermissionsExt fs host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 69 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 69 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 72 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 76 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 76 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 77 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 84 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 84 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 84 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 84 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 86 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 86 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/ape.rs 165 native_path std::os::unix::fs::symlink fs host_mechanic crates/fbuild-core/src/platform/linux/device.rs 33 native_path std::os::unix::fs::OpenOptionsExt fs host_mechanic crates/fbuild-core/src/platform/linux/device.rs 41 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/linux/device.rs 41 native_path libc:: process host_mechanic @@ -15,19 +29,25 @@ crates/fbuild-core/src/platform/linux/fs.rs 51 native_path libc:: process host_m crates/fbuild-core/src/platform/linux/fs.rs 51 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/linux/fs.rs 54 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/linux/fs.rs 68 native_path std::os::unix::fs::MetadataExt fs host_mechanic -crates/fbuild-core/src/platform/linux/fs.rs 112 native_path std::os::unix::fs::symlink fs host_mechanic -crates/fbuild-core/src/platform/linux/fs.rs 116 native_path std::os::unix::fs::symlink fs host_mechanic -crates/fbuild-core/src/platform/linux/fs.rs 158 native_path std::os::unix::ffi::OsStrExt process host_mechanic -crates/fbuild-core/src/platform/linux/fs.rs 163 native_path libc:: process host_mechanic -crates/fbuild-core/src/platform/linux/fs.rs 165 native_path libc:: process host_mechanic -crates/fbuild-core/src/platform/linux/fs.rs 175 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 104 native_path std::os::unix::fs fs host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 110 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 117 native_path std::os::unix::ffi::OsStrExt process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 121 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 123 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 128 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 144 native_path std::os::unix::fs::symlink fs host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 148 native_path std::os::unix::fs::symlink fs host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 190 native_path std::os::unix::ffi::OsStrExt process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 195 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 197 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/linux/fs.rs 207 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/linux/ipc.rs 1 native_path interprocess::local_socket ipc host_mechanic crates/fbuild-core/src/platform/linux/ipc.rs 2 native_path interprocess::local_socket ipc host_mechanic crates/fbuild-core/src/platform/linux/ipc.rs 3 native_path interprocess::os::unix ipc host_mechanic crates/fbuild-core/src/platform/linux/ipc.rs 4 native_path socket2:: ipc host_mechanic crates/fbuild-core/src/platform/linux/ipc.rs 66 native_path std::os::unix::fs::PermissionsExt ipc host_mechanic crates/fbuild-core/src/platform/linux/ipc.rs 86 native_path socket2:: ipc host_mechanic -crates/fbuild-core/src/platform/linux/mod.rs 14 compile_host_fact std::env::consts::ARCH host host_mechanic +crates/fbuild-core/src/platform/linux/mod.rs 15 compile_host_fact std::env::consts::ARCH host host_mechanic crates/fbuild-core/src/platform/linux/process.rs 1 native_path std::os::unix::process::ExitStatusExt process host_mechanic crates/fbuild-core/src/platform/macos/fs.rs 2 native_path std::os::unix::fs::PermissionsExt fs host_mechanic crates/fbuild-core/src/platform/macos/fs.rs 31 native_path std::os::unix::fs fs host_mechanic @@ -35,18 +55,24 @@ crates/fbuild-core/src/platform/macos/fs.rs 51 native_path libc:: process host_m crates/fbuild-core/src/platform/macos/fs.rs 51 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/macos/fs.rs 54 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/macos/fs.rs 68 native_path std::os::unix::fs::MetadataExt fs host_mechanic -crates/fbuild-core/src/platform/macos/fs.rs 112 native_path std::os::unix::fs::symlink fs host_mechanic -crates/fbuild-core/src/platform/macos/fs.rs 116 native_path std::os::unix::fs::symlink fs host_mechanic -crates/fbuild-core/src/platform/macos/fs.rs 158 native_path std::os::unix::ffi::OsStrExt process host_mechanic -crates/fbuild-core/src/platform/macos/fs.rs 163 native_path libc:: process host_mechanic -crates/fbuild-core/src/platform/macos/fs.rs 165 native_path libc:: process host_mechanic -crates/fbuild-core/src/platform/macos/fs.rs 175 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 104 native_path std::os::unix::fs fs host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 110 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 117 native_path std::os::unix::ffi::OsStrExt process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 121 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 123 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 128 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 144 native_path std::os::unix::fs::symlink fs host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 148 native_path std::os::unix::fs::symlink fs host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 190 native_path std::os::unix::ffi::OsStrExt process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 195 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 197 native_path libc:: process host_mechanic +crates/fbuild-core/src/platform/macos/fs.rs 207 native_path libc:: process host_mechanic crates/fbuild-core/src/platform/macos/ipc.rs 1 native_path interprocess::local_socket ipc host_mechanic crates/fbuild-core/src/platform/macos/ipc.rs 2 native_path interprocess::local_socket ipc host_mechanic crates/fbuild-core/src/platform/macos/ipc.rs 3 native_path socket2:: ipc host_mechanic crates/fbuild-core/src/platform/macos/ipc.rs 19 native_path std::os::unix::fs::PermissionsExt ipc host_mechanic crates/fbuild-core/src/platform/macos/ipc.rs 72 native_path std::os::unix::fs::PermissionsExt ipc host_mechanic -crates/fbuild-core/src/platform/macos/mod.rs 13 compile_host_fact std::env::consts::ARCH host host_mechanic +crates/fbuild-core/src/platform/macos/mod.rs 14 compile_host_fact std::env::consts::ARCH host host_mechanic crates/fbuild-core/src/platform/macos/process.rs 1 native_path std::os::unix::process::ExitStatusExt process host_mechanic crates/fbuild-core/src/platform/windows/device.rs 18 native_path windows_sys:: process host_mechanic crates/fbuild-core/src/platform/windows/device.rs 27 native_path windows_sys:: process host_mechanic @@ -64,24 +90,24 @@ crates/fbuild-core/src/platform/windows/fs.rs 81 native_path windows_sys:: proce crates/fbuild-core/src/platform/windows/fs.rs 82 native_path windows_sys:: process host_mechanic crates/fbuild-core/src/platform/windows/fs.rs 116 native_path windows_sys:: process host_mechanic crates/fbuild-core/src/platform/windows/fs.rs 156 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 260 native_path std::os::windows::fs::symlink_dir fs host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 264 native_path std::os::windows::fs::symlink_file fs host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 292 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 352 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 394 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 405 attr_cfg #[cfg(windows)] host host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 417 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 418 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 438 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 439 native_path windows_sys:: process host_mechanic -crates/fbuild-core/src/platform/windows/fs.rs 458 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 273 native_path std::os::windows::fs::symlink_dir fs host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 277 native_path std::os::windows::fs::symlink_file fs host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 305 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 365 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 407 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 418 attr_cfg #[cfg(windows)] host host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 430 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 431 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 451 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 452 native_path windows_sys:: process host_mechanic +crates/fbuild-core/src/platform/windows/fs.rs 471 native_path windows_sys:: process host_mechanic crates/fbuild-core/src/platform/windows/ipc.rs 1 native_path interprocess::local_socket ipc host_mechanic crates/fbuild-core/src/platform/windows/ipc.rs 2 native_path interprocess::local_socket ipc host_mechanic crates/fbuild-core/src/platform/windows/ipc.rs 3 native_path interprocess::os::windows ipc host_mechanic crates/fbuild-core/src/platform/windows/ipc.rs 4 native_path interprocess::os::windows ipc host_mechanic crates/fbuild-core/src/platform/windows/ipc.rs 5 native_path socket2:: ipc host_mechanic crates/fbuild-core/src/platform/windows/ipc.rs 6 native_path std::os::windows::io::AsRawSocket ipc host_mechanic -crates/fbuild-core/src/platform/windows/mod.rs 14 compile_host_fact std::env::consts::ARCH host host_mechanic +crates/fbuild-core/src/platform/windows/mod.rs 15 compile_host_fact std::env::consts::ARCH host host_mechanic crates/fbuild-core/src/platform/windows/private_directory.rs 3 native_path std::os::windows::ffi::OsStrExt process host_mechanic crates/fbuild-core/src/platform/windows/private_directory.rs 8 native_path windows_sys:: process host_mechanic crates/fbuild-core/src/platform/windows/private_directory.rs 13 native_path windows_sys:: process host_mechanic @@ -162,12 +188,12 @@ crates/fbuild-paths/src/executable_hash.rs 71 attr_cfg #[cfg(unix)] host host_me crates/fbuild-paths/src/executable_hash.rs 76 attr_cfg #[cfg(unix)] host host_mechanic crates/fbuild-paths/src/executable_hash.rs 85 attr_cfg #[cfg(all(test,unix))] host host_mechanic crates/fbuild-python/tests/python_facades/main.rs 318 native_path std::env::current_exe host_executable host_mechanic -crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 528 attr_cfg #[cfg(windows)] host_executable host_mechanic -crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 534 attr_cfg #[cfg(windows)] host_executable host_mechanic -crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 565 attr_cfg #[cfg(not(windows))] host_executable host_mechanic -crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 570 attr_cfg #[cfg(not(windows))] host_executable host_mechanic -crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 728 attr_cfg #[cfg(windows)] host_executable host_mechanic -crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 755 attr_cfg #[cfg(windows)] host_executable host_mechanic +crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 539 attr_cfg #[cfg(windows)] host_executable host_mechanic +crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 545 attr_cfg #[cfg(windows)] host_executable host_mechanic +crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 576 attr_cfg #[cfg(not(windows))] host_executable host_mechanic +crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 581 attr_cfg #[cfg(not(windows))] host_executable host_mechanic +crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 739 attr_cfg #[cfg(windows)] host_executable host_mechanic +crates/fbuild-toolchain/src/toolchain/esp_qemu.rs 766 attr_cfg #[cfg(windows)] host_executable host_mechanic crates/fbuild-toolchain/src/toolchain/esp_qemu_runtime.rs 253 attr_cfg #[cfg_attr(not(target_os=),allow(dead_code))] host_executable host_artifact_policy crates/fbuild-toolchain/src/toolchain/esp_qemu_runtime.rs 295 attr_cfg #[cfg(not(target_os=))] host_executable host_artifact_policy crates/fbuild-toolchain/src/toolchain/esp_qemu_runtime.rs 301 attr_cfg #[cfg(target_os=)] host_executable host_artifact_policy diff --git a/crates/fbuild-build-engine/src/build_info.rs b/crates/fbuild-build-engine/src/build_info.rs index 5f219ee02..efab0615a 100644 --- a/crates/fbuild-build-engine/src/build_info.rs +++ b/crates/fbuild-build-engine/src/build_info.rs @@ -707,6 +707,15 @@ mod tests { ); } + #[test] + fn derive_gcc_tool_path_preserves_ape_com_suffix() { + let size = PathBuf::from("/toolchain/bin/xtensa-esp32s3-elf-size.com"); + assert_eq!( + derive_gcc_tool_path(&size, "nm"), + PathBuf::from("/toolchain/bin").join("xtensa-esp32s3-elf-nm.com") + ); + } + #[test] fn derive_gcc_tool_path_bare_size_falls_back_to_bare_target() { // PATH-resolved `size` with no prefix. diff --git a/crates/fbuild-build-engine/src/compile_database/cache_wrapper.rs b/crates/fbuild-build-engine/src/compile_database/cache_wrapper.rs index 3cf0db777..1a7ef3a69 100644 --- a/crates/fbuild-build-engine/src/compile_database/cache_wrapper.rs +++ b/crates/fbuild-build-engine/src/compile_database/cache_wrapper.rs @@ -14,10 +14,8 @@ pub fn strip_cache_wrapper(args: &[String]) -> Vec { // `Path::file_stem` only splits on the platform's native separator, so // `C:\...\sccache.exe` is treated as one component on Linux/macOS. let filename = args[0].rsplit(['/', '\\']).next().unwrap_or(&args[0]); - let stem = filename - .strip_suffix(".exe") - .or_else(|| filename.strip_suffix(".EXE")) - .unwrap_or(filename) + let stem = fbuild_core::platform::executable::split_tool_suffix(filename) + .0 .to_lowercase(); if stem == "sccache" || stem == "ccache" || stem == "zccache" { diff --git a/crates/fbuild-build-engine/src/compile_database/clang.rs b/crates/fbuild-build-engine/src/compile_database/clang.rs index 8de0eff84..0c0a2b60d 100644 --- a/crates/fbuild-build-engine/src/compile_database/clang.rs +++ b/crates/fbuild-build-engine/src/compile_database/clang.rs @@ -55,7 +55,10 @@ pub fn translate_flags_for_clang(args: &[String], arch: TargetArchitecture) -> V let compiler_path = fbuild_core::path::NormalizedPath::from(args[0].as_str()) .display_slash() .to_lowercase(); - let clang_name = if compiler_path.ends_with("g++") || compiler_path.ends_with("g++.exe") { + let clang_name = if compiler_path.ends_with("g++") + || compiler_path.ends_with("g++.exe") + || compiler_path.ends_with("g++.com") + { "clang++" } else { "clang" diff --git a/crates/fbuild-build-engine/src/linker.rs b/crates/fbuild-build-engine/src/linker.rs index b57d41516..559780c57 100644 --- a/crates/fbuild-build-engine/src/linker.rs +++ b/crates/fbuild-build-engine/src/linker.rs @@ -913,4 +913,11 @@ mod tests { let nm = LinkerBase::nm_path_from_size_path(&size); assert_eq!(nm, PathBuf::from("C:/toolchain/bin/arm-none-eabi-nm.exe")); } + + #[test] + fn test_nm_path_from_size_path_with_ape_com() { + let size = PathBuf::from("/toolchain/bin/arm-none-eabi-size.com"); + let nm = LinkerBase::nm_path_from_size_path(&size); + assert_eq!(nm, PathBuf::from("/toolchain/bin/arm-none-eabi-nm.com")); + } } diff --git a/crates/fbuild-build-engine/src/symbol_analyzer/tests.rs b/crates/fbuild-build-engine/src/symbol_analyzer/tests.rs index aedd35164..82bfdf413 100644 --- a/crates/fbuild-build-engine/src/symbol_analyzer/tests.rs +++ b/crates/fbuild-build-engine/src/symbol_analyzer/tests.rs @@ -17,6 +17,15 @@ fn derive_cppfilt_prefix() { ); } +#[test] +fn derive_cppfilt_preserves_ape_com_suffix() { + let nm = PathBuf::from("/tools/xtensa-esp32s3-elf-nm.com"); + assert_eq!( + derive_cppfilt_path(&nm), + PathBuf::from("/tools/xtensa-esp32s3-elf-c++filt.com") + ); +} + #[test] fn derive_cppfilt_no_prefix() { let nm = PathBuf::from("/usr/bin/nm"); diff --git a/crates/fbuild-build-engine/src/zccache_embedded.rs b/crates/fbuild-build-engine/src/zccache_embedded.rs index 073051b7b..7eb32b8fc 100644 --- a/crates/fbuild-build-engine/src/zccache_embedded.rs +++ b/crates/fbuild-build-engine/src/zccache_embedded.rs @@ -298,9 +298,15 @@ impl FbuildZccacheService { cwd: PathBuf, env: Vec<(String, String)>, ) -> Result { + // zccache spawns the compiler itself, so an APE (cosmocc) compiler + // can't be routed through fbuild's loader; hand it a host-native + // stand-in with the same file name instead. + let compiler = fbuild_core::platform::ape::native_executable(compiler) + .map(fbuild_core::path::NormalizedPath::into_path_buf) + .unwrap_or_else(|| compiler.to_path_buf()); let req = ZccacheCompileRequest { audit: default_audit_context(), - compiler: compiler.to_path_buf().into(), + compiler: compiler.into(), args, cwd: cwd.into(), env, diff --git a/crates/fbuild-build/tests/it/ape_toolchain.rs b/crates/fbuild-build/tests/it/ape_toolchain.rs new file mode 100644 index 000000000..a9db8897f --- /dev/null +++ b/crates/fbuild-build/tests/it/ape_toolchain.rs @@ -0,0 +1,183 @@ +//! A toolchain made entirely of APE (cosmocc) executables, driven the two +//! ways fbuild runs compilers: `run_command` (fbuild's own spawn path) and the +//! embedded zccache service (which spawns the compiler itself). +//! +//! cosmocc's gcc, cc1, as and ld are all APE images, and gcc spawns cc1/as +//! itself, so this exercises nested APE spawns too. Both paths give the +//! compiler a hostile environment (no usable PATH/TMPDIR/HOME; zccache passes +//! an empty env), so only fbuild's own loader support can make it work. +//! +//! Opt-in: point `FBUILD_TEST_COSMOCC` at an unpacked +//! (the directory holding `bin/`). + +use std::path::Path; + +use fbuild_build::zccache_embedded::FbuildZccacheService; +use fbuild_core::path::NormalizedPath; +use zccache::embedded::ShutdownMode; + +fn cosmo_gcc() -> NormalizedPath { + let root = std::env::var_os("FBUILD_TEST_COSMOCC").expect("FBUILD_TEST_COSMOCC"); + let arch = fbuild_core::platform::host::arch_name(); + let gcc = Path::new(&root) + .join("bin") + .join(format!("{arch}-linux-cosmo-gcc")); + assert!( + fbuild_core::platform::ape::is_ape_file(&gcc), + "{} must be an APE image", + gcc.display() + ); + NormalizedPath::from(gcc) +} + +/// A packaged-toolchain layout of the APE gcc. Raw cosmocc gcc has no +/// built-in system include dirs (the `cosmocc` driver script passes them), and +/// zccache bypasses its cache when include discovery finds none. A platform +/// package ships its gcc configured, so mirror that: a tree of symlinks into +/// the unpacked cosmocc with the gcc APE image copied in and a GCC `specs` file +/// adding the toolchain's include root. +fn packaged_cosmo_gcc(into: &Path) -> NormalizedPath { + let src = cosmo_gcc(); + let root = src.parent().unwrap().parent().unwrap(); + let triple = format!("{}-linux-cosmo", fbuild_core::platform::host::arch_name()); + for entry in std::fs::read_dir(root).unwrap() { + let name = entry.unwrap().file_name(); + if name != "bin" && name != "lib" { + link(&root.join(&name), &into.join(&name)).unwrap(); + } + } + std::fs::create_dir_all(into.join("bin")).unwrap(); + let gcc = into.join("bin").join(src.file_name().unwrap()); + std::fs::copy(&src, &gcc).unwrap(); + // GCC reads `specs` from lib/gcc/// under its prefix. + let version = std::fs::read_dir(root.join("libexec/gcc").join(&triple)) + .unwrap() + .next() + .unwrap() + .unwrap() + .file_name(); + let dest = into.join("lib/gcc").join(&triple).join(&version); + std::fs::create_dir_all(&dest).unwrap(); + for entry in std::fs::read_dir(root.join("lib/gcc")).unwrap() { + let entry = entry.unwrap(); + if entry.file_type().unwrap().is_file() { + link(&entry.path(), &into.join("lib/gcc").join(entry.file_name())).unwrap(); + } + } + std::fs::write( + dest.join("specs"), + format!("*cpp:\n+ -isystem {}\n\n", root.join("include").display()), + ) + .unwrap(); + NormalizedPath::from(gcc) +} + +fn link(original: &Path, link: &Path) -> std::io::Result<()> { + if original.is_dir() { + fbuild_core::platform::fs::symlink_dir(original, link) + } else { + fbuild_core::platform::fs::symlink_file(original, link) + } +} + +fn source(dir: &Path) -> NormalizedPath { + let src = dir.join("add.c"); + std::fs::write(&src, "int add(int a, int b) { return a + b; }\n").unwrap(); + NormalizedPath::from(src) +} + +fn compile_args(src: &Path, obj: &Path) -> Vec { + vec![ + "-nostdinc".into(), + "-c".into(), + src.display().to_string(), + "-o".into(), + obj.display().to_string(), + ] +} + +fn zccache_args(src: &Path, obj: &Path) -> Vec { + vec![ + "-c".into(), + src.display().to_string(), + "-o".into(), + obj.display().to_string(), + ] +} + +#[test] +#[ignore = "requires the cosmocc toolchain (~440 MB) via FBUILD_TEST_COSMOCC"] +fn ape_gcc_compiles_through_run_command_in_hostile_env() { + let gcc = cosmo_gcc(); + let dir = tempfile::tempdir().unwrap(); + let src = source(dir.path()); + let obj = dir.path().join("add.o"); + let mut argv = vec![gcc.display().to_string()]; + argv.extend(compile_args(&src, &obj)); + let argv: Vec<&str> = argv.iter().map(String::as_str).collect(); + let out = fbuild_core::subprocess::run_command_blocking( + &argv, + Some(dir.path()), + Some(&[ + ("PATH", "/nonexistent"), + ("TMPDIR", "/nonexistent"), + ("HOME", "/nonexistent"), + ]), + None, + ) + .expect("APE gcc must spawn"); + assert!( + out.success(), + "stdout: {}\nstderr: {}", + out.stdout, + out.stderr + ); + assert!(std::fs::read(&obj).unwrap().starts_with(b"\x7fELF")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires the cosmocc toolchain (~440 MB) via FBUILD_TEST_COSMOCC"] +async fn ape_gcc_compiles_through_embedded_zccache_and_hits_cache() { + let tmp = tempfile::tempdir().unwrap(); + let toolchain = tmp.path().join("toolchain"); + std::fs::create_dir(&toolchain).unwrap(); + let gcc = packaged_cosmo_gcc(&toolchain); + let svc = FbuildZccacheService::start_in(tmp.path().join("zccache")) + .await + .expect("zccache starts"); + let work = tmp.path().join("work"); + std::fs::create_dir(&work).unwrap(); + let src = source(&work); + let obj = work.join("add.o"); + // The same hermetic env production compiles get (no inherited PATH). + let mut env = fbuild_core::subprocess::compile_env_for_build(&work).unwrap(); + // A daemon started with a minimal PATH (systemd service, launchd): no + // coreutils for the APE prologue's own fallback to lean on. + env.retain(|(k, _)| k != "PATH"); + env.push(("PATH".to_string(), "/nonexistent".to_string())); + env.push(( + "ZCCACHE_WORKTREE_ROOT".to_string(), + work.display().to_string(), + )); + for round in 0..2 { + let _ = std::fs::remove_file(&obj); + let out = svc + .compile(&gcc, zccache_args(&src, &obj), work.clone(), env.clone()) + .await + .expect("zccache compile"); + assert_eq!( + out.exit_code, + 0, + "round {round} stderr: {}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(std::fs::read(&obj).unwrap().starts_with(b"\x7fELF")); + assert_eq!( + out.cached, + round == 1, + "round {round}: cold miss, then warm hit" + ); + svc.flush().await.expect("flush into the cache"); + } + svc.shutdown(ShutdownMode::Graceful).await.unwrap(); +} diff --git a/crates/fbuild-build/tests/it/main.rs b/crates/fbuild-build/tests/it/main.rs index c94b44c50..127e23b53 100644 --- a/crates/fbuild-build/tests/it/main.rs +++ b/crates/fbuild-build/tests/it/main.rs @@ -2,6 +2,7 @@ //! each top-level `tests/*.rs` file links the crate's whole dependency //! graph separately, so these modules share one link instead. +mod ape_toolchain; mod cache_survives_tar_extract; mod clangd_check_parity; mod compile_many_stage2_perf; diff --git a/crates/fbuild-cli/src/cli/clang_tools.rs b/crates/fbuild-cli/src/cli/clang_tools.rs index b44417919..4303f7ba1 100644 --- a/crates/fbuild-cli/src/cli/clang_tools.rs +++ b/crates/fbuild-cli/src/cli/clang_tools.rs @@ -288,7 +288,7 @@ pub async fn run_iwyu( // Cache miss — run IWYU. Parallel async fan-out inside the CLI binary // (no daemon containment group in this process). // allow-direct-spawn: parallel async fan-out in CLI; no containment group here. - let mut cmd = tokio::process::Command::new(tool.as_ref()); + let mut cmd = fbuild_core::platform::process::tokio_command(tool.as_ref()); cmd.arg("-p").arg(p_dir.as_ref()); cmd.arg("-Xiwyu").arg("--no_comments"); cmd.arg("-Xiwyu").arg("--quoted_includes_first"); @@ -299,8 +299,21 @@ pub async fn run_iwyu( cmd.arg(&file); // FastLED/fbuild#810: cap each IWYU invocation at 120s so a wedged // subprocess can't hang the whole fan-out forever. - let output = - match tokio::time::timeout(std::time::Duration::from_secs(120), cmd.output()).await + // Fork under the shared fork lock (released before the await), with + // the stdio `output()` would set. + cmd.stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()); + let child = { + let _fork = fbuild_core::platform::process::shared_fork_guard(); + cmd.spawn() + }; + let output = match child { + Ok(child) => match tokio::time::timeout( + std::time::Duration::from_secs(120), + child.wait_with_output(), + ) + .await { Ok(res) => res, Err(_) => { @@ -311,7 +324,9 @@ pub async fn run_iwyu( src_path, ); } - }; + }, + Err(e) => Err(e), + }; match output { Ok(out) => { @@ -575,7 +590,7 @@ pub async fn run_clang_tool( .await .expect("fbuild-cli: clang-tool semaphore is never closed before all tasks finish"); // allow-direct-spawn: parallel async fan-out (clang-tidy) in CLI binary. - let mut cmd = tokio::process::Command::new(tool.as_ref()); + let mut cmd = fbuild_core::platform::process::tokio_command(tool.as_ref()); cmd.arg("-p").arg(pd.as_ref()); for arg in &extra { cmd.arg(arg); @@ -586,15 +601,30 @@ pub async fn run_clang_tool( } // FastLED/fbuild#810: cap each clang-tidy invocation at 120s so a // wedged subprocess can't hang the whole fan-out forever. - let output = - match tokio::time::timeout(std::time::Duration::from_secs(120), cmd.output()).await + // Fork under the shared fork lock (released before the await), with + // the stdio `output()` would set. + cmd.stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()); + let child = { + let _fork = fbuild_core::platform::process::shared_fork_guard(); + cmd.spawn() + }; + let output = match child { + Ok(child) => match tokio::time::timeout( + std::time::Duration::from_secs(120), + child.wait_with_output(), + ) + .await { Ok(res) => res, Err(_) => Err(std::io::Error::new( std::io::ErrorKind::TimedOut, "clang-tidy timed out after 120s", )), - }; + }, + Err(e) => Err(e), + }; (file, output) }); handles.push(handle); diff --git a/crates/fbuild-cli/src/cli/debug.rs b/crates/fbuild-cli/src/cli/debug.rs index 4d088566a..281228bb3 100644 --- a/crates/fbuild-cli/src/cli/debug.rs +++ b/crates/fbuild-cli/src/cli/debug.rs @@ -81,7 +81,6 @@ //! `-ex "set serial baud " -ex "target remote " `. use std::path::Path; -use std::process::Command; use fbuild_build::build_info::{find_build_info_near, load_build_info}; use fbuild_build::symbol_analyzer::discover_elf_in_project; @@ -246,27 +245,25 @@ pub(crate) fn gdb_candidate_names(toolchain_prefix: &str) -> Vec { names } +#[cfg(test)] fn exe_name(name: &str) -> String { fbuild_core::platform::executable::native_name(name) } /// Search a list of directories (in order) for the first existing -/// candidate binary name. Pure function of its inputs — I/O-free beyond -/// `Path::exists`, so tests can point it at a `tempdir` fixture instead of -/// the real `PATH`/toolchain cache. +/// candidate binary (native spelling, or a `.com`/`.exe` APE; see +/// `find_tool_in`). Pure function of its inputs — file I/O only, so tests +/// can point it at a `tempdir` fixture instead of the real +/// `PATH`/toolchain cache. pub(crate) fn find_gdb_in_dirs( dirs: &[NormalizedPath], candidate_names: &[String], ) -> Option { - for dir in dirs { - for name in candidate_names { - let candidate = dir.join(exe_name(name)); - if candidate.exists() { - return Some(candidate); - } - } - } - None + dirs.iter().find_map(|dir| { + candidate_names + .iter() + .find_map(|name| fbuild_core::platform::executable::find_tool_in(dir, name)) + }) } fn path_dirs() -> Vec { @@ -545,10 +542,17 @@ pub async fn run_debug( // interactively in the user's terminal exactly as if they'd typed the // command themselves. // allow-direct-spawn: interactive gdb must inherit the user's terminal stdio; the capturing subprocess helpers would break the session. - let status = Command::new(&gdb_path) - .args(&argv) - .status() - .map_err(|e| FbuildError::Other(format!("failed to launch {}: {e}", gdb_path.display())))?; + let mut gdb = fbuild_core::platform::process::command(&gdb_path); + gdb.args(&argv); + let launch_err = |e: std::io::Error| { + FbuildError::Other(format!("failed to launch {}: {e}", gdb_path.display())) + }; + // Fork under the shared fork lock; release it before the interactive wait. + let mut child = { + let _fork = fbuild_core::platform::process::shared_fork_guard(); + gdb.spawn().map_err(launch_err)? + }; + let status = child.wait().map_err(launch_err)?; if !status.success() { return Err(FbuildError::CommandFailed { diff --git a/crates/fbuild-cli/src/cli/ide.rs b/crates/fbuild-cli/src/cli/ide.rs index a898a79fc..f2b3014d8 100644 --- a/crates/fbuild-cli/src/cli/ide.rs +++ b/crates/fbuild-cli/src/cli/ide.rs @@ -315,7 +315,7 @@ fn print_zed_install_guidance() { /// wait on the editor process). fn launch_zed(zed_path: &Path, project_dir: &str) -> fbuild_core::Result<()> { // allow-direct-spawn: editor is launched detached and must outlive the CLI; deliberately not in a containment group. - let mut cmd = std::process::Command::new(zed_path); + let mut cmd = fbuild_core::platform::process::command(zed_path); cmd.arg(project_dir); fbuild_core::platform::process::spawn_detached( &mut cmd, diff --git a/crates/fbuild-cli/src/cli/symbols_cmd.rs b/crates/fbuild-cli/src/cli/symbols_cmd.rs index 59ed8f7de..ede17b638 100644 --- a/crates/fbuild-cli/src/cli/symbols_cmd.rs +++ b/crates/fbuild-cli/src/cli/symbols_cmd.rs @@ -227,11 +227,10 @@ fn resolve_elf(input: &Path) -> Result { fn find_nm_on_path() -> Result { let exe_name = fbuild_core::platform::executable::native_name("nm"); let path = std::env::var_os("PATH").ok_or_else(|| FbuildError::Other("PATH not set".into()))?; - for dir in std::env::split_paths(&path) { - let candidate = dir.join(&exe_name); - if candidate.exists() { - return Ok(candidate); - } + if let Some(found) = + fbuild_core::platform::executable::find_tool_on_paths(std::env::split_paths(&path), "nm") + { + return Ok(found.into_path_buf()); } Err(FbuildError::BuildFailed(format!( "{exe_name} not found on PATH; pass --nm to point at a cross toolchain nm" diff --git a/crates/fbuild-cli/src/daemon_client.rs b/crates/fbuild-cli/src/daemon_client.rs index 50b89dfcb..aed5eb7f7 100644 --- a/crates/fbuild-cli/src/daemon_client.rs +++ b/crates/fbuild-cli/src/daemon_client.rs @@ -1383,7 +1383,7 @@ async fn spawn_daemon_process() -> fbuild_core::Result { // name (and thus PATH) when no sibling is found. let daemon_exe = daemon_executable_hint(); // allow-direct-spawn: daemon must outlive the CLI; see INTENTIONALLY DETACHED comment below. - let mut cmd = std::process::Command::new(&daemon_exe); + let mut cmd = fbuild_core::platform::process::command(&daemon_exe); let baseline = running_process::environment::user_baseline_environment().map_err(|error| { fbuild_core::FbuildError::DaemonError(format!( "failed to construct daemon user environment: {error}" diff --git a/crates/fbuild-cli/src/daemon_client/tests.rs b/crates/fbuild-cli/src/daemon_client/tests.rs index 1f39fcb75..475ed24d8 100644 --- a/crates/fbuild-cli/src/daemon_client/tests.rs +++ b/crates/fbuild-cli/src/daemon_client/tests.rs @@ -236,13 +236,17 @@ fn foreign_root_owner_message_names_the_owner() { cache_root_key: "/c".to_string(), port: 60903, }; + // NormalizedPath renders host-native separators, so assert against the + // same path the message will carry on any platform (Windows would turn + // the unix literal below into backslashes and fail the match). + let exe = claim.exe.display().to_string(); let message = foreign_root_owner_message(&claim, 19142); for needle in [ "98258", "2.5.44", "60903", "19142", - "/venv/bin/fbuild-daemon", + exe.as_str(), "No compilation", ] { assert!(message.contains(needle), "{needle} missing from: {message}"); diff --git a/crates/fbuild-cli/src/main.rs b/crates/fbuild-cli/src/main.rs index 0d90b6de4..b359cec90 100644 --- a/crates/fbuild-cli/src/main.rs +++ b/crates/fbuild-cli/src/main.rs @@ -22,6 +22,8 @@ fn main() { } fn async_main_entry() { + // APE (cosmocc) host tools: keep extracted loaders under fbuild's own cache. + fbuild_core::platform::ape::set_default_cache_root(fbuild_paths::get_cache_root().join("ape")); // FastLED/fbuild#1285: derive the dev daemon-identity stamp once, at the // top level, and export the value so every child — including the spawned // daemon — inherits it instead of re-hashing per invocation. Official diff --git a/crates/fbuild-core/Cargo.toml b/crates/fbuild-core/Cargo.toml index f7ce13338..91a6e8949 100644 --- a/crates/fbuild-core/Cargo.toml +++ b/crates/fbuild-core/Cargo.toml @@ -19,6 +19,9 @@ sha2 = { workspace = true } semver = { workspace = true } # Runtime decompression for the downloaded FastLED/boards cache. zstd = { workspace = true } +# Inflates the gzip'd loader embedded in APE (cosmocc) images so +# `platform::ape` can run them without a host APE loader. +flate2 = { workspace = true } prost = { workspace = true } # Cross-platform containment and detached-spawn implementation delegated by # `platform::process`. Its source and revision intentionally match zccache's diff --git a/crates/fbuild-core/data/README.md b/crates/fbuild-core/data/README.md index b507cad16..fe55b443b 100644 --- a/crates/fbuild-core/data/README.md +++ b/crates/fbuild-core/data/README.md @@ -8,6 +8,7 @@ the published FastLED/boards artifacts during the build/cache phase. | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `usb-vendors.tar.zst` | Frozen test-only USB Vendor-ID fixture used by `crate::usb::embedded` under `cfg(test)`. It is never a production fallback. | | `usb-vids.proto.zstd` | Test fixture for the compact VID:PID overlay produced by **FastLED/boards**. It must never be used as a production built-in catalogue. Production ingestion fetches/consumes the published boards artifact. | +| `ape-hello/` | Test-only cosmocc hello-world APE (with source and build script) used by `crate::platform::ape` tests. See `ape-hello/README.md`. | ## How to refresh the VID:PID overlay (`usb-vids.proto.zstd`) diff --git a/crates/fbuild-core/data/ape-hello/README.md b/crates/fbuild-core/data/ape-hello/README.md new file mode 100644 index 000000000..b812f8abb --- /dev/null +++ b/crates/fbuild-core/data/ape-hello/README.md @@ -0,0 +1,17 @@ +# ape-hello — APE test fixture + +A minimal [Actually Portable Executable](https://justine.lol/ape.html) +(cosmocc) used by `crate::platform::ape` tests to prove fbuild can spawn APE +binaries on hosts with no APE `binfmt_misc` handler (e.g. NixOS), where a direct +`posix_spawn` fails with `ENOEXEC` ("Exec format error"). + +| File | Purpose | +| ---------- | ------- | +| `hello.c` | Source: prints `hello world` followed by its arguments. | +| `hello.com`| Fat x86_64 + aarch64 APE built from `hello.c` (`-Os -mtiny -s`, ~295 KB). Test-only. | +| `build.sh` | Rebuilds `hello.com`; set `COSMOCC` to the `cosmocc` driver. | + +```console +$ ./hello.com a b # works from bash (it retries ENOEXEC via sh) +hello world a b +``` diff --git a/crates/fbuild-core/data/ape-hello/build.sh b/crates/fbuild-core/data/ape-hello/build.sh new file mode 100755 index 000000000..6e36021ce --- /dev/null +++ b/crates/fbuild-core/data/ape-hello/build.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# Rebuild hello.com with cosmocc (https://cosmo.zip/pub/cosmocc/cosmocc.zip). +# Usage: COSMOCC=/path/to/cosmocc/bin/cosmocc ./build.sh +set -euo pipefail +cd "$(dirname "$0")" +"${COSMOCC:-cosmocc}" -Os -mtiny -s -o hello.com hello.c +# cosmocc also leaves per-arch debug images next to the output. +rm -f hello.com.dbg hello.aarch64.elf diff --git a/crates/fbuild-core/data/ape-hello/hello.c b/crates/fbuild-core/data/ape-hello/hello.c new file mode 100644 index 000000000..fa59efea0 --- /dev/null +++ b/crates/fbuild-core/data/ape-hello/hello.c @@ -0,0 +1,8 @@ +#include + +int main(int argc, char **argv) { + printf("hello world"); + for (int i = 1; i < argc; ++i) printf(" %s", argv[i]); + printf("\n"); + return 0; +} diff --git a/crates/fbuild-core/data/ape-hello/hello.com b/crates/fbuild-core/data/ape-hello/hello.com new file mode 100755 index 000000000..47bff35a1 Binary files /dev/null and b/crates/fbuild-core/data/ape-hello/hello.com differ diff --git a/crates/fbuild-core/src/platform/README.md b/crates/fbuild-core/src/platform/README.md index 272a03b1f..5ec4b61d9 100644 --- a/crates/fbuild-core/src/platform/README.md +++ b/crates/fbuild-core/src/platform/README.md @@ -38,3 +38,31 @@ Unix endpoint creation, TCP listener socket policy, and endpoint readiness probing. Broker framing/routing, daemon retry/yield policy, and HTTP/protobuf compatibility remain with the daemon. Additional native shutdown notifications route through `process` into the daemon's neutral shutdown channel. + +`ape` detects Actually Portable Executable (cosmocc) images by their magic +(`MZqFpD='`, `jartsr='`, `APEDBG='`) and, on Unix hosts, launches them as +` `, because `posix_spawn` cannot exec an APE without a +`binfmt_misc` handler (NixOS ships none). Loader order: `FBUILD_APE_LOADER`, then +the loader embedded in the image itself (Linux ELF, macOS x86_64 Mach-O, or +Apple Silicon `ape-m1.c` compiled once with `cc`), then `ape` on PATH, then +`/usr/bin/ape` / `/usr/local/bin/ape`, then `/bin/sh` (which runs the image's +self-extracting prologue). Windows runs APE natively. Every +`subprocess::run_command*` spawn applies this; direct spawns build their command +with `process::command` / `process::tokio_command` instead of `Command::new`. + +The embedded loader is located from the prologue's `dd skip=N count=M | gzip -dc` +line for the host CPU, inflated in-process, validated (64-bit ELF or Mach-O for +that CPU), and installed content-addressed into the first owner-only (0700, not +group/world-writable, not a symlink), exec-capable directory among +`FBUILD_APE_CACHE_DIR`, fbuild's cache root (registered by the daemon and CLI via +`ape::set_default_cache_root`), `$XDG_CACHE_HOME`/`~/.cache` `fbuild/ape` (macOS: +`~/Library/Caches/fbuild/ape`) and `$XDG_RUNTIME_DIR/fbuild/ape`; on Linux, failing +all of those, it lives in a sealed `memfd` exec'd via `/proc/self/fd/N`. The child +needs no PATH, `sh`, coreutils, `gzip`, `HOME` or `TMPDIR`, and the loader is put +first on its PATH as `ape` so APE programs it spawns itself resolve too. Installs +are atomic (temp file + `rename`), tampered or truncated copies are rewritten, +and the writes hold `process::exclusive_fork_guard` — a Go-style fork lock every +spawn helper (and the few allowlisted direct spawns, via +`process::shared_fork_guard`) holds shared across fork→exec — so no concurrently +forked child can inherit the writable descriptor. Spawners fbuild doesn't +control (zccache) can still race it, so APE launches retry `ETXTBSY` briefly. diff --git a/crates/fbuild-core/src/platform/ape/README.md b/crates/fbuild-core/src/platform/ape/README.md new file mode 100644 index 000000000..1ff876ae2 --- /dev/null +++ b/crates/fbuild-core/src/platform/ape/README.md @@ -0,0 +1,18 @@ +# `platform::ape` — APE (cosmocc) host-tool support + +Runs Actually Portable Executable tools on every host fbuild supports without +any host APE support installed. + +- `mod.rs` — APE detection, launch planning (` `), and + per-host loader extraction: Linux ELF loader, macOS x86_64 Mach-O loader, + macOS arm64 loader compiled from the image's `ape-m1.c`. +- `prologue.rs` — parses the image's shell prologue (per-CPU loader blobs, + the macOS x86_64 header-patched blob, the Apple Silicon loader source). +- `install.rs` — owner-only, exec-capable, atomic, content-addressed installs + under the fork lock (no `ETXTBSY`). +- `native.rs` — a `/bin/sh` shim on every Unix host for spawners fbuild + doesn't control (zccache): same file name, the image's identity baked in, + `ape` first on PATH, `exec `. + +Host specifics (cache dirs, Linux `memfd` fallback) live in +`../{linux,macos,windows}/ape.rs`. diff --git a/crates/fbuild-core/src/platform/ape/install.rs b/crates/fbuild-core/src/platform/ape/install.rs new file mode 100644 index 000000000..62b62f01f --- /dev/null +++ b/crates/fbuild-core/src/platform/ape/install.rs @@ -0,0 +1,91 @@ +//! Owner-only, atomic installation of executables fbuild derives from APE +//! images (extracted loaders, `ape` PATH entries, launch shims). +//! +//! A candidate directory is used only when it is a real directory owned by +//! the effective user with no group/other write access (so no other account +//! can plant or swap a file in it) and sits on a mount that allows exec. + +use std::fs::OpenOptions; +use std::io::Write; +use std::path::Path; + +use crate::path::NormalizedPath; +use std::sync::atomic::{AtomicU64, Ordering}; + +use super::super::{fs, process}; + +/// Install `bytes` as executable `//` in the first usable +/// candidate `dir`, reusing an identical existing file. `None` when no +/// candidate is usable. +pub(crate) fn install( + dirs: &[NormalizedPath], + subdir: Option<&str>, + name: &str, + bytes: &[u8], +) -> Option { + dirs.iter().find_map(|dir| { + let dir = match subdir { + Some(sub) => { + // The parent must pass the same checks before we nest in it. + if !usable_dir(dir) { + return None; + } + dir.join(sub) + } + None => dir.clone(), + }; + install_in(&dir, bytes, name) + }) +} + +fn usable_dir(dir: &Path) -> bool { + fs::ensure_private_dir(dir) && fs::mount_allows_exec(dir) +} + +pub(crate) fn install_in(dir: &Path, bytes: &[u8], name: &str) -> Option { + if !usable_dir(dir) { + return None; + } + let target = NormalizedPath::new(dir.join(name)); + if is_installed(&target, bytes) { + return Some(target); + } + // Unique per process and attempt inside an owner-only directory, so a + // plain create+truncate can't collide with another writer. + static SEQ: AtomicU64 = AtomicU64::new(0); + let tmp = dir.join(format!( + ".{name}.{}.{}", + std::process::id(), + SEQ.fetch_add(1, Ordering::Relaxed) + )); + // No child may inherit the writable fd, or exec of the file would hit + // ETXTBSY until that child execs (see `process::exclusive_fork_guard`). + let written = { + let _fork = process::exclusive_fork_guard(); + OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .open(&tmp) + .and_then(|mut file| { + file.write_all(bytes)?; + file.sync_all() + }) + } + .and_then(|()| fs::set_executable(&tmp)) + // rename(2) is atomic: concurrent installers race benignly to identical + // content, and readers never see a partial file. + .and_then(|()| std::fs::rename(&tmp, &target)); + if written.is_err() { + let _ = std::fs::remove_file(&tmp); + return None; + } + is_installed(&target, bytes).then_some(target) +} + +/// A regular (not symlinked), executable file whose content is exactly `bytes`. +pub(crate) fn is_installed(path: &Path, bytes: &[u8]) -> bool { + std::fs::symlink_metadata(path).is_ok_and(|meta| { + meta.file_type().is_file() && fs::is_executable(&meta) && meta.len() == bytes.len() as u64 + }) && std::fs::read(path).is_ok_and(|content| content == bytes) +} diff --git a/crates/fbuild-core/src/platform/ape/mod.rs b/crates/fbuild-core/src/platform/ape/mod.rs new file mode 100644 index 000000000..e975fcacf --- /dev/null +++ b/crates/fbuild-core/src/platform/ape/mod.rs @@ -0,0 +1,984 @@ +//! Actually Portable Executable (APE / cosmocc) launch support. +//! +//! An APE image is simultaneously a Windows PE, a Bourne-shell script, and +//! (via its embedded loader) an ELF/Mach-O program. Windows runs it natively. +//! Unix kernels do not: `execve` returns `ENOEXEC` unless the host registered +//! an APE `binfmt_misc` handler, and `posix_spawn` — what Rust's `Command` +//! uses — never retries through `/bin/sh` the way an interactive shell does. +//! NixOS ships no APE handler, so a bare spawn fails with "Exec format error". +//! +//! fbuild therefore detects the APE magic and launches the image through a +//! loader explicitly: ` `. fbuild provides the loader +//! itself from the image (see [`prologue`]): +//! +//! * Linux x86_64/aarch64: the gzip'd static ELF loader for the host CPU. +//! * macOS x86_64: the same blob with its embedded Mach-O header moved to +//! offset 0. +//! * macOS arm64: the image's `ape-m1.c`, compiled once with the host `cc`. +//! +//! Each is validated and installed content-addressed into an owner-only, +//! exec-capable cache directory ([`install`]); on Linux a sealed `memfd` is the +//! last resort. None of this needs `sh`, coreutils, `gzip`, PATH, or a +//! writable temp dir in the child environment. +//! +//! Spawners fbuild does not control (zccache runs the compiler itself) get a +//! host-native path instead, from [`native_executable`]. +//! +//! Loader precedence: `FBUILD_APE_LOADER` → loader embedded in the image → +//! `ape` on PATH → well-known `ape` install locations → `/bin/sh` → `sh` on +//! PATH. + +use std::collections::HashMap; +use std::ffi::{OsStr, OsString}; +use std::io::Read; +use std::path::Path; + +use crate::path::NormalizedPath; +use std::sync::Mutex; +use std::time::SystemTime; + +use sha2::{Digest, Sha256}; + +use super::host::{self, HostArch, HostPlatform}; + +pub(crate) mod install; +mod native; +mod prologue; + +pub use native::native_executable; + +/// Environment variable naming an explicit APE loader (an `ape` binary or a +/// POSIX shell). Read from the child env overlay first, then the process env. +pub const LOADER_ENV: &str = "FBUILD_APE_LOADER"; + +/// Environment variable naming the preferred directory for loaders extracted +/// from APE images. Consulted before the host defaults; it must be owned by +/// the current user, not group/world-writable, and on an exec-capable mount. +pub const CACHE_DIR_ENV: &str = "FBUILD_APE_CACHE_DIR"; + +/// Bytes of an image scanned for the shell prologue. Real prologues end +/// within the first ~8 KiB; the cap bounds work on hostile input. +const PROLOGUE_SCAN_BYTES: u64 = 64 * 1024; + +/// Upper bound on a compressed or inflated embedded loader (real ones are +/// ~4–5 KiB compressed, ~10 KiB inflated). +const MAX_LOADER_BYTES: u64 = 4 * 1024 * 1024; + +/// Leading bytes that identify an APE image. `MZqFpD='` is the standard +/// header (also a valid DOS/PE `MZ` stub), `jartsr='` is the non-Windows +/// variant, and `APEDBG='` is the debug build variant. +const APE_MAGICS: [&[u8; 8]; 3] = [b"MZqFpD='", b"jartsr='", b"APEDBG='"]; + +/// Well-known install locations for the Cosmopolitan loader, consulted when +/// `ape` is not on PATH (e.g. NixOS service environments with a minimal PATH). +const WELL_KNOWN_LOADERS: [&str; 2] = ["/usr/bin/ape", "/usr/local/bin/ape"]; + +/// Whether `head` starts with an APE magic. +pub fn is_ape_header(head: &[u8]) -> bool { + APE_MAGICS.iter().any(|magic| head.starts_with(*magic)) +} + +/// Whether the file at `path` is an APE image. Unreadable files are not APE. +pub fn is_ape_file(path: &Path) -> bool { + let mut head = [0u8; 8]; + std::fs::File::open(path) + .and_then(|mut file| file.read_exact(&mut head)) + .is_ok_and(|()| is_ape_header(&head)) +} + +/// How to launch an APE image: run `loader` with `image` as its first argument, +/// followed by the caller's original arguments. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ApeLaunch { + pub loader: NormalizedPath, + pub image: NormalizedPath, + /// A private directory holding this loader as `ape`. Spawners prepend it + /// to the child's `PATH` so APE programs the tool itself spawns (gcc → + /// `cc1`) find a loader via the prologue's `type ape` — without it those + /// nested spawns need `sh` + coreutils + a writable temp dir. + pub ape_path_dir: Option, +} + +impl ApeLaunch { + /// The child's `PATH` with [`Self::ape_path_dir`] prepended, given the + /// `PATH` it would otherwise get. `None` when there is nothing to add. + pub fn child_path(&self, inherited: Option<&OsStr>) -> Option { + let dir = self.ape_path_dir.as_ref()?; + let rest = inherited.filter(|p| !p.is_empty()); + std::env::join_paths( + std::iter::once(dir.as_path().to_path_buf()) + .chain(rest.into_iter().flat_map(std::env::split_paths)), + ) + .ok() + } +} + +/// Plan an APE launch for `program` on the current host. +/// +/// `cwd` is the child's working directory (used to resolve a relative +/// program path) and `env_overlay` the child's env overlay (consulted for +/// `PATH`, [`LOADER_ENV`] and [`CACHE_DIR_ENV`] before the process +/// environment). Returns `None` when `program` is not an APE image or the +/// host runs APE natively. +pub fn plan_launch( + program: &OsStr, + cwd: Option<&Path>, + env_overlay: Option<&[(&str, &str)]>, +) -> Option { + let host = host::current(); + if host.is_windows() { + return None; + } + let overlay_var = |key: &str| { + env_overlay + .and_then(|vars| vars.iter().rev().find(|(k, _)| *k == key)) + .map(|(_, v)| OsString::from(*v)) + .or_else(|| std::env::var_os(key)) + .filter(|v| !v.is_empty()) + }; + let path_var = overlay_var("PATH"); + let loader_override = overlay_var(LOADER_ENV); + let cache_dirs = cache_dirs(overlay_var(CACHE_DIR_ENV)); + plan_launch_for( + host, + program, + cwd, + path_var.as_deref(), + loader_override.as_deref(), + &cache_dirs, + ) +} + +static DEFAULT_CACHE_ROOT: std::sync::OnceLock = std::sync::OnceLock::new(); + +/// Register fbuild's own cache directory for APE-derived executables +/// (loaders, `ape` PATH entries, shims). Binaries call this at startup with a +/// directory under `fbuild_paths::get_cache_root()`, which `fbuild-core` can't +/// depend on. The first registration wins. +pub fn set_default_cache_root(dir: impl AsRef) { + let _ = DEFAULT_CACHE_ROOT.set(NormalizedPath::new(dir)); +} + +/// Candidate cache directories, most preferred first: the explicit +/// [`CACHE_DIR_ENV`] value, the registered fbuild cache root, then host +/// defaults (XDG/`~/.cache`, runtime dir; `~/Library/Caches` on macOS). +fn cache_dirs(explicit: Option) -> Vec { + explicit + .map(NormalizedPath::new) + .into_iter() + .chain(DEFAULT_CACHE_ROOT.get().cloned()) + .chain(super::selected::ape::default_loader_dirs()) + .collect() +} + +/// [`plan_launch`] with every host input explicit, for deterministic tests. +/// `cache_dirs` are the candidate directories for an extracted loader. +pub fn plan_launch_for( + host: HostPlatform, + program: &OsStr, + cwd: Option<&Path>, + path_var: Option<&OsStr>, + loader_override: Option<&OsStr>, + cache_dirs: &[NormalizedPath], +) -> Option { + if host.is_windows() { + return None; + } + let image = resolve_program(program, cwd, path_var)?; + if !is_ape_file(&image) { + return None; + } + let loader = match loader_override.filter(|v| !v.is_empty()) { + Some(explicit) => resolve_explicit_loader(explicit, path_var)?, + None => { + embedded_loader(&image, host, cache_dirs).or_else(|| resolve_host_loader(path_var))? + } + }; + let ape_path_dir = ape_path_dir(&loader, cache_dirs); + Some(ApeLaunch { + loader, + image, + ape_path_dir, + }) +} + +/// Resolve `program` to the file the kernel would execute: a path with a +/// separator is taken as-is (relative to `cwd` when given), a bare name is +/// searched on `path_var`. +fn resolve_program( + program: &OsStr, + cwd: Option<&Path>, + path_var: Option<&OsStr>, +) -> Option { + let path = Path::new(program); + if path.components().count() > 1 || path.is_absolute() { + let resolved = match cwd { + Some(dir) if path.is_relative() => dir.join(path), + _ => path.to_path_buf(), + }; + return std::path::absolute(resolved).ok().map(NormalizedPath::new); + } + find_executable_on_path(program, path_var) +} + +fn resolve_explicit_loader(explicit: &OsStr, path_var: Option<&OsStr>) -> Option { + let explicit = Path::new(explicit); + if explicit.components().count() > 1 || explicit.is_absolute() { + return Some(NormalizedPath::new(explicit)); + } + find_executable_on_path(explicit.as_os_str(), path_var) +} + +fn resolve_host_loader(path_var: Option<&OsStr>) -> Option { + find_executable_on_path(OsStr::new("ape"), path_var) + .or_else(|| first_executable(WELL_KNOWN_LOADERS.iter().map(NormalizedPath::new))) + .or_else(|| first_executable([NormalizedPath::from("/bin/sh")])) + .or_else(|| find_executable_on_path(OsStr::new("sh"), path_var)) +} + +fn find_executable_on_path(name: &OsStr, path_var: Option<&OsStr>) -> Option { + let dirs = std::env::split_paths(path_var?).filter(|dir| !dir.as_os_str().is_empty()); + first_executable(dirs.map(|dir| NormalizedPath::new(dir.join(name)))) +} + +fn first_executable( + candidates: impl IntoIterator, +) -> Option { + candidates + .into_iter() + .find(|candidate| is_executable_file(candidate)) +} + +fn is_executable_file(path: &Path) -> bool { + std::fs::metadata(path).is_ok_and(|meta| meta.is_file() && super::fs::is_executable(&meta)) +} + +/// Identity of an image (and where its derived files may live) for the +/// memos: a rebuilt image at the same path changes length or mtime and is +/// re-derived. +type ImageKey = ( + NormalizedPath, + u64, + Option, + HostPlatform, + Vec, +); + +fn image_key(image: &Path, host: HostPlatform, dirs: &[NormalizedPath]) -> Option { + let meta = std::fs::metadata(image).ok()?; + Some(( + NormalizedPath::new(image), + meta.len(), + meta.modified().ok(), + host, + dirs.to_vec(), + )) +} + +/// Memo of derived executables (loaders, native copies) per image. +struct Memo(Mutex>>); + +impl Memo { + const fn new() -> Self { + Self(Mutex::new(None)) + } + + fn get(&self, key: &ImageKey) -> Option { + let guard = self.0.lock().unwrap_or_else(|e| e.into_inner()); + let hit = guard.as_ref()?.get(key)?.clone(); + // A cache cleaner may have removed it since; re-derive then. + hit.exists().then_some(hit) + } + + fn put(&self, key: ImageKey, path: NormalizedPath) { + let mut guard = self.0.lock().unwrap_or_else(|e| e.into_inner()); + guard.get_or_insert_with(HashMap::new).insert(key, path); + } +} + +static LOADER_MEMO: Memo = Memo::new(); + +/// The host-runnable loader embedded in `image`, installed into the first +/// usable `cache_dirs` entry (or, on Linux, a sealed memfd). `None` when the +/// image carries no valid loader for the host or nothing could be +/// materialized. +fn embedded_loader( + image: &Path, + host: HostPlatform, + cache_dirs: &[NormalizedPath], +) -> Option { + let key = image_key(image, host, cache_dirs)?; + if let Some(hit) = LOADER_MEMO.get(&key) { + return Some(hit); + } + let arch = arch_name(host.arch())?; + let loader = if host.is_linux() { + let bytes = extract_loader(image, host.arch())?; + materialize( + &bytes, + &format!("ape-loader-linux-{arch}-{}", short_hash(&bytes)), + cache_dirs, + )? + } else if host.is_macos() && host.arch() == HostArch::X86_64 { + let bytes = extract_macos_x86_64_loader(image)?; + materialize( + &bytes, + &format!("ape-loader-macos-{arch}-{}", short_hash(&bytes)), + cache_dirs, + )? + } else if host.is_macos() && host.arch() == HostArch::Aarch64 { + compile_macos_aarch64_loader(image, cache_dirs)? + } else { + return None; + }; + LOADER_MEMO.put(key, loader.clone()); + Some(loader) +} + +/// A private directory containing `loader` under the name `ape`, for +/// [`ApeLaunch::ape_path_dir`]. Shells are never exposed as `ape`. +fn ape_path_dir(loader: &Path, cache_dirs: &[NormalizedPath]) -> Option { + if loader.file_name().is_some_and(|n| n == "sh") || super::selected::ape::is_anonymous(loader) { + return None; + } + if loader.file_name().is_some_and(|n| n == "ape") && !super::selected::ape::is_anonymous(loader) + { + return loader.parent().map(NormalizedPath::new); + } + let bytes = std::fs::read(loader).ok()?; + if bytes.len() as u64 > MAX_LOADER_BYTES { + return None; + } + let sub = format!("bin-{}", short_hash(&bytes)); + install::install(cache_dirs, Some(&sub), "ape", &bytes)? + .parent() + .map(NormalizedPath::new) +} + +/// Install `bytes` as an executable named `name` in the first usable cache +/// dir, else as a host anonymous executable (Linux memfd). +pub(crate) fn materialize( + bytes: &[u8], + name: &str, + dirs: &[NormalizedPath], +) -> Option { + install::install(dirs, None, name, bytes) + .or_else(|| super::selected::ape::anonymous_executable(bytes, name)) +} + +fn short_hash(bytes: &[u8]) -> String { + Sha256::digest(bytes)[..8] + .iter() + .map(|b| format!("{b:02x}")) + .collect() +} + +fn arch_name(arch: HostArch) -> Option<&'static str> { + match arch { + HostArch::X86_64 => Some("x86_64"), + HostArch::Aarch64 => Some("aarch64"), + _ => None, + } +} + +/// ELF `e_machine` for `arch`. +fn elf_machine(arch: HostArch) -> Option { + match arch { + HostArch::X86_64 => Some(62), // EM_X86_64 + HostArch::Aarch64 => Some(183), // EM_AARCH64 + _ => None, + } +} + +/// An opened image with its parsed prologue. +struct Image { + file: std::fs::File, + len: u64, + prologue: prologue::Prologue, +} + +fn open_image(image: &Path) -> Option { + let mut file = std::fs::File::open(image).ok()?; + let len = file.metadata().ok()?.len(); + let mut head = Vec::new(); + (&mut file) + .take(PROLOGUE_SCAN_BYTES) + .read_to_end(&mut head) + .ok()?; + if !is_ape_header(&head) { + return None; + } + Some(Image { + file, + len, + prologue: prologue::parse(&head), + }) +} + +impl Image { + /// Read `range`, bounded by the file length and [`MAX_LOADER_BYTES`]. + fn read(&mut self, (skip, count): prologue::Range) -> Option> { + use std::io::{Seek, SeekFrom}; + let end = skip.checked_add(count)?; + if count == 0 || count > MAX_LOADER_BYTES || end > self.len { + return None; + } + self.file.seek(SeekFrom::Start(skip)).ok()?; + let mut out = Vec::with_capacity(count as usize); + (&mut self.file).take(count).read_to_end(&mut out).ok()?; + (out.len() as u64 == count).then_some(out) + } + + fn inflate(&mut self, range: prologue::Range) -> Option> { + let gz = self.read(range)?; + let mut out = Vec::new(); + flate2::read::GzDecoder::new(gz.as_slice()) + .take(MAX_LOADER_BYTES + 1) + .read_to_end(&mut out) + .ok()?; + (out.len() as u64 <= MAX_LOADER_BYTES).then_some(out) + } +} + +/// Inflate and validate the Linux ELF loader `image` embeds for `arch`. +pub fn extract_loader(image: &Path, arch: HostArch) -> Option> { + let machine = elf_machine(arch)?; + let mut img = open_image(image)?; + let range = img.prologue.linux_loader(arch)?; + let elf = img.inflate(range)?; + is_static_elf_for(&elf, machine).then_some(elf) +} + +/// The macOS x86_64 loader: the same gzip'd blob as Linux, with the Mach-O +/// header it carries at offset 320 (8 × 64 bytes) moved to offset 0 — what the +/// prologue's `dd if="$t.$$" of="$t.$$" skip=5 count=8 bs=64` does. +pub fn extract_macos_x86_64_loader(image: &Path) -> Option> { + let mut img = open_image(image)?; + let range = img.prologue.macos_loader_x86_64?; + let mut bin = img.inflate(range)?; + if bin.len() < 320 + 512 { + return None; + } + bin.copy_within(320..320 + 512, 0); + is_macho_x86_64(&bin).then_some(bin) +} + +/// The Apple Silicon loader source (`ape-m1.c`) embedded in `image`. +pub fn extract_macos_aarch64_loader_source(image: &Path) -> Option> { + let mut img = open_image(image)?; + let range = img.prologue.macos_loader_source_aarch64?; + let src = img.inflate(range)?; + (std::str::from_utf8(&src).is_ok() && src.windows(4).any(|w| w == b"main")).then_some(src) +} + +/// Compile the image's `ape-m1.c` with the host C compiler (Xcode Command +/// Line Tools) into a content-addressed cache entry, as the prologue would, +/// but without depending on `sh`, `dd`, `gzip` or `$TMPDIR`. +fn compile_macos_aarch64_loader(image: &Path, dirs: &[NormalizedPath]) -> Option { + let src = extract_macos_aarch64_loader_source(image)?; + let name = format!("ape-loader-macos-aarch64-{}", short_hash(&src)); + let dir = dirs + .iter() + .find(|dir| super::fs::ensure_private_dir(dir) && super::fs::mount_allows_exec(dir))?; + let target = dir.join(&name); + if is_executable_file(&target) { + return Some(target); + } + // One compile at a time in this process: concurrent first spawns would + // otherwise all run `cc` (seconds each). Re-check once we hold the lock. + static COMPILE: Mutex<()> = Mutex::new(()); + let _compiling = COMPILE.lock().unwrap_or_else(|e| e.into_inner()); + if is_executable_file(&target) { + return Some(target); + } + // Unique per attempt, so racing processes never share temp files. + static SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + let tag = format!( + "{}.{}", + std::process::id(), + SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + ); + let src_path = dir.join(format!(".{name}.{tag}.c")); + let out_path = dir.join(format!(".{name}.{tag}")); + std::fs::write(&src_path, &src).ok()?; + let cc = first_executable([NormalizedPath::from("/usr/bin/cc")]) + .unwrap_or_else(|| NormalizedPath::from("cc")); + let compiled = crate::subprocess::run_command_blocking( + &[ + cc.to_str()?, + "-w", + "-O", + "-o", + out_path.to_str()?, + src_path.to_str()?, + ], + None, + None, + Some(std::time::Duration::from_secs(300)), + ); + let _ = std::fs::remove_file(&src_path); + let ok = compiled.as_ref().is_ok_and(|out| out.success()) + && std::fs::rename(&out_path, &target).is_ok(); + if !ok { + let _ = std::fs::remove_file(&out_path); + if let Ok(out) = compiled { + tracing::warn!( + "could not build the Apple Silicon APE loader with {}: {}", + cc.display(), + out.stderr.trim() + ); + } + return None; + } + Some(target) +} + +/// A 64-bit little-endian ELF executable for `machine`. +fn is_static_elf_for(elf: &[u8], machine: u16) -> bool { + elf.len() >= 64 + && elf.len() as u64 <= MAX_LOADER_BYTES + && elf.starts_with(b"\x7fELF") + && elf[4] == 2 // ELFCLASS64 + && elf[5] == 1 // ELFDATA2LSB + // ET_EXEC (x86_64 loader) or ET_DYN (position-independent aarch64 loader) + && matches!(u16::from_le_bytes([elf[16], elf[17]]), 2 | 3) + && u16::from_le_bytes([elf[18], elf[19]]) == machine +} + +/// A 64-bit Mach-O for x86_64 (`MH_MAGIC_64`, `CPU_TYPE_X86_64`). +fn is_macho_x86_64(bin: &[u8]) -> bool { + bin.len() >= 32 && bin.starts_with(&[0xcf, 0xfa, 0xed, 0xfe]) && bin[4..8] == [7, 0, 0, 1] +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::platform::host::{HostArch, HostOs}; + + const LINUX: HostPlatform = HostPlatform::new(HostOs::Linux, HostArch::X86_64); + const WINDOWS: HostPlatform = HostPlatform::new(HostOs::Windows, HostArch::X86_64); + + /// Minimal APE-shaped script: the `MZqFpD='...'` header is a shell + /// assignment, exactly like a real cosmocc image's prologue. + const FAKE_APE: &str = "MZqFpD='\n'\necho fake-ape \"$@\"\n"; + + fn write_exe(dir: &Path, name: &str, body: &str) -> NormalizedPath { + let path = dir.join(name); + std::fs::write(&path, body).unwrap(); + crate::platform::fs::set_executable(&path).unwrap(); + NormalizedPath::new(path) + } + + #[test] + fn recognizes_every_ape_magic_and_rejects_other_formats() { + assert!(is_ape_header(b"MZqFpD='\n\n\0")); + assert!(is_ape_header(b"jartsr='\n")); + assert!(is_ape_header(b"APEDBG='\n")); + assert!(!is_ape_header(b"MZ\x90\0\x03\0\0\0")); // plain PE + assert!(!is_ape_header(b"\x7fELF\x02\x01\x01\0")); // ELF + assert!(!is_ape_header(b"#!/bin/sh\n")); + assert!(!is_ape_header(b"MZqF")); + } + + #[test] + fn windows_runs_ape_natively() { + let dir = tempfile::tempdir().unwrap(); + let ape = write_exe(dir.path(), "tool", FAKE_APE); + let plan = plan_launch_for( + WINDOWS, + ape.as_os_str(), + None, + None, + Some(OsStr::new("/bin/sh")), + &[], + ); + assert_eq!(plan, None); + } + + #[test] + fn non_ape_program_is_left_alone() { + let dir = tempfile::tempdir().unwrap(); + let script = write_exe(dir.path(), "tool", "#!/bin/sh\necho hi\n"); + let plan = plan_launch_for( + LINUX, + script.as_os_str(), + None, + None, + Some(OsStr::new("/bin/sh")), + &[], + ); + assert_eq!(plan, None); + } + + #[test] + fn explicit_loader_override_wins() { + let dir = tempfile::tempdir().unwrap(); + let ape = write_exe(dir.path(), "tool", FAKE_APE); + let plan = plan_launch_for( + LINUX, + ape.as_os_str(), + None, + None, + Some(OsStr::new("/opt/cosmo/ape")), + &[], + ) + .expect("APE must be planned"); + assert_eq!(plan.loader, NormalizedPath::from("/opt/cosmo/ape")); + assert_eq!(plan.image, ape); + } + + #[test] + fn bare_name_resolves_on_path_and_prefers_ape_loader_on_path() { + let bin = tempfile::tempdir().unwrap(); + let ape = write_exe(bin.path(), "tool", FAKE_APE); + let loader = write_exe(bin.path(), "ape", "#!/bin/sh\n"); + let plan = plan_launch_for( + LINUX, + OsStr::new("tool"), + None, + Some(bin.path().as_os_str()), + None, + &[], + ) + .expect("APE on PATH must be planned"); + assert_eq!(plan.image, ape); + assert_eq!(plan.loader, loader); + } + + #[test] + fn relative_program_resolves_against_child_cwd() { + let dir = tempfile::tempdir().unwrap(); + std::fs::create_dir(dir.path().join("bin")).unwrap(); + let ape = write_exe(&dir.path().join("bin"), "tool", FAKE_APE); + let plan = plan_launch_for( + LINUX, + OsStr::new("bin/tool"), + Some(dir.path()), + None, + Some(OsStr::new("/bin/sh")), + &[], + ) + .expect("relative APE must be planned"); + assert_eq!(plan.image, ape); + } + + /// End-to-end through the real spawn path: without loader routing a Unix + /// `posix_spawn` of an APE fails with ENOEXEC ("Exec format error"). + /// Windows runs an APE image directly (it is a valid PE) under every + /// spelling fbuild accepts for a package tool. + #[test] + fn windows_spawns_real_ape_under_exe_and_com_names() { + if !host::current().is_windows() { + return; + } + let dir = tempfile::tempdir().unwrap(); + for name in ["tool.exe", "tool.com"] { + let image = copy_fixture(dir.path(), name); + assert_eq!(plan_launch(image.as_os_str(), None, None), None); + let out = crate::subprocess::run_command_blocking( + &[image.to_str().unwrap(), "on", "windows"], + None, + None, + None, + ) + .expect("APE must spawn natively on Windows"); + assert!(out.success(), "{name} stderr: {}", out.stderr); + assert_eq!(out.stdout, "hello world on windows\n", "{name}"); + } + } + + #[test] + fn run_command_launches_ape_through_loader() { + if host::current().is_windows() { + return; + } + let dir = tempfile::tempdir().unwrap(); + let ape = write_exe(dir.path(), "tool", FAKE_APE); + let program = ape.to_str().unwrap(); + let out = crate::subprocess::run_command_blocking( + &[program, "a", "b c"], + None, + Some(&[(LOADER_ENV, "/bin/sh")]), + None, + ) + .expect("APE spawn must succeed through the loader"); + assert!(out.success(), "stderr: {}", out.stderr); + assert_eq!(out.stdout, "fake-ape a b c\n"); + } + + /// The checked-in cosmocc hello-world (`data/ape-hello`). + fn hello_fixture() -> NormalizedPath { + NormalizedPath::new(Path::new(env!("CARGO_MANIFEST_DIR")).join("data/ape-hello/hello.com")) + } + + #[test] + fn hello_fixture_is_detected_as_ape() { + assert!(is_ape_file(&hello_fixture())); + } + + /// A real cosmocc image through the default loader chain (no override). + /// Every Unix host: Linux uses the embedded ELF loader, macOS x86_64 the + /// embedded Mach-O loader, Apple Silicon the image's `ape-m1.c` built with `cc`. + #[test] + fn real_ape_hello_runs_via_run_command() { + if host::current().is_windows() { + return; + } + let ape = hello_fixture(); + let out = crate::subprocess::run_command_blocking( + &[ape.to_str().unwrap(), "from", "fbuild"], + None, + None, + None, + ) + .expect("real APE must spawn"); + assert!(out.success(), "stderr: {}", out.stderr); + assert_eq!(out.stdout, "hello world from fbuild\n"); + } + + #[test] + fn real_ape_hello_runs_via_process_command() { + if host::current().is_windows() { + return; + } + let mut command = crate::platform::process::command(hello_fixture()); + command.arg("std"); + let mut child = crate::platform::process::spawn_contained( + &mut command, + crate::platform::process::ContainedStdio { + stdout: crate::platform::process::StdioSource::Pipe, + ..Default::default() + }, + ) + .expect("real APE must spawn"); + let mut stdout = String::new(); + child + .take_stdout() + .unwrap() + .read_to_string(&mut stdout) + .unwrap(); + assert_eq!(child.wait().unwrap(), 0); + assert_eq!(stdout, "hello world std\n"); + } + + fn copy_fixture(dir: &Path, name: &str) -> NormalizedPath { + let path = dir.join(name); + std::fs::copy(hello_fixture(), &path).unwrap(); + NormalizedPath::new(path) + } + + /// Hostile/corrupt images never panic and never yield a loader. + #[test] + fn corrupt_images_yield_no_embedded_loader() { + let dir = tempfile::tempdir().unwrap(); + let real = std::fs::read(hello_fixture()).unwrap(); + let arch = HostArch::X86_64; + + let truncated = dir.path().join("truncated"); + std::fs::write(&truncated, &real[..20_000]).unwrap(); + assert_eq!(extract_loader(&truncated, arch), None); + + let mut garbage = real.clone(); + garbage[275392..275392 + 4180].fill(0xA5); + let garbled = dir.path().join("garbled"); + std::fs::write(&garbled, &garbage).unwrap(); + assert_eq!(extract_loader(&garbled, arch), None); + + let huge = dir.path().join("huge-range"); + std::fs::write( + &huge, + "MZqFpD='\n'\nif [ \"$m\" = x86_64 ]; then\ndd if=\"$o\" skip=18446744073709551615 count=99 bs=1 2>/dev/null | gzip -dc >\"$t.$$\" ||exit\nfi\n", + ) + .unwrap(); + assert_eq!(extract_loader(&huge, arch), None); + + // A valid gzip of a non-ELF payload is rejected too. + let mut fake = b"MZqFpD='\n'\nif [ \"$m\" = x86_64 ]; then\ndd if=\"$o\" skip=200 count=COUNT bs=1 2>/dev/null | gzip -dc >\"$t.$$\" ||exit\nfi\n".to_vec(); + let mut gz = Vec::new(); + { + use std::io::Write; + let mut enc = flate2::write::GzEncoder::new(&mut gz, flate2::Compression::fast()); + enc.write_all(b"#!/bin/sh\necho pwned\n").unwrap(); + enc.finish().unwrap(); + } + let text = String::from_utf8(fake) + .unwrap() + .replace("COUNT", &gz.len().to_string()); + fake = text.into_bytes(); + fake.resize(200, b'\n'); + fake.extend_from_slice(&gz); + let not_elf = dir.path().join("not-elf"); + std::fs::write(¬_elf, &fake).unwrap(); + assert_eq!(extract_loader(¬_elf, arch), None); + } + + /// Spawn the fixture in a child environment with no PATH, TMPDIR or HOME, + /// so neither `ape`, `sh` utilities, nor the prologue's `$TMPDIR` cache can + /// help: only fbuild's own extracted loader can make this work. + fn run_hostile(program: &str, cache: &Path, args: &[&str]) -> crate::subprocess::ToolOutput { + let mut argv = vec![program]; + argv.extend_from_slice(args); + crate::subprocess::run_command_blocking( + &argv, + None, + Some(&[ + ("PATH", "/nonexistent"), + ("TMPDIR", "/nonexistent"), + ("HOME", "/nonexistent"), + (CACHE_DIR_ENV, cache.to_str().unwrap()), + ]), + None, + ) + .expect("APE must spawn") + } + + #[test] + fn real_ape_runs_with_hostile_child_env_using_own_loader() { + if host::current().is_windows() { + return; + } + let cache = tempfile::tempdir().unwrap(); + let out = run_hostile( + hello_fixture().to_str().unwrap(), + cache.path(), + &["hostile"], + ); + assert!(out.success(), "stderr: {}", out.stderr); + assert_eq!(out.stdout, "hello world hostile\n"); + let mut installed: Vec<_> = std::fs::read_dir(cache.path()) + .unwrap() + .map(|e| e.unwrap().file_name().into_string().unwrap()) + .collect(); + installed.sort(); + // The loader, plus the same loader exposed as `bin-/ape` for + // the child's PATH (nested APE spawns). + assert_eq!(installed.len(), 2, "{installed:?}"); + assert!(installed[0].starts_with("ape-loader-"), "{installed:?}"); + assert!(installed[1].starts_with("bin-"), "{installed:?}"); + let exposed = cache.path().join(&installed[1]).join("ape"); + assert_eq!( + std::fs::read(exposed).unwrap(), + std::fs::read(cache.path().join(&installed[0])).unwrap() + ); + } + + #[test] + fn broken_ape_on_path_does_not_hijack_launch() { + if host::current().is_windows() { + return; + } + let bin = tempfile::tempdir().unwrap(); + write_exe(bin.path(), "ape", "#!/bin/sh\necho hijacked; exit 99\n"); + let image = copy_fixture(bin.path(), "hello"); + let cache = tempfile::tempdir().unwrap(); + let out = crate::subprocess::run_command_blocking( + &["hello", "ok"], + None, + Some(&[ + ("PATH", bin.path().to_str().unwrap()), + (CACHE_DIR_ENV, cache.path().to_str().unwrap()), + ]), + None, + ) + .expect("bare-name APE must spawn"); + assert_eq!(out.stdout, "hello world ok\n", "stderr: {}", out.stderr); + assert!(image.exists()); + } + + #[test] + fn awkward_image_paths_and_args_survive() { + if host::current().is_windows() { + return; + } + let dir = tempfile::tempdir().unwrap(); + let sub = dir.path().join("dir with spaces – ünïcode"); + std::fs::create_dir(&sub).unwrap(); + let image = copy_fixture(&sub, "hello world.com"); + let link = dir.path().join("link-to-hello"); + crate::platform::fs::symlink_file(&image, &link).unwrap(); + let cache = tempfile::tempdir().unwrap(); + let link = NormalizedPath::new(link); + for program in [&image, &link] { + let out = run_hostile( + program.to_str().unwrap(), + cache.path(), + &["a b", "", "$HOME", "--assimilate"], + ); + assert!(out.success(), "stderr: {}", out.stderr); + assert_eq!(out.stdout, "hello world a b $HOME --assimilate\n"); + } + // `--assimilate` is just an argument: the image must be untouched. + assert_eq!( + std::fs::read(&image).unwrap(), + std::fs::read(hello_fixture()).unwrap() + ); + } + + #[test] + fn rebuilt_image_at_same_path_is_reextracted() { + if host::current().is_windows() { + return; + } + let dir = tempfile::tempdir().unwrap(); + let image = copy_fixture(dir.path(), "tool"); + let cache = tempfile::tempdir().unwrap(); + let program = image.to_str().unwrap(); + assert_eq!( + run_hostile(program, cache.path(), &["1"]).stdout, + "hello world 1\n" + ); + // Replace with a non-APE script: must now run natively, not via loader. + std::fs::write(&image, "#!/bin/sh\necho native \"$@\"\n").unwrap(); + crate::platform::fs::set_executable(&image).unwrap(); + // Just-written script: retry ETXTBSY from sibling tests forking (#1366). + let out = crate::subprocess::run_command_blocking_retrying_exec_busy( + &[program, "2"], + None, + None, + None, + ) + .unwrap(); + assert_eq!(out.stdout, "native 2\n"); + // Deleted cache between spawns is re-materialized. + std::fs::copy(hello_fixture(), &image).unwrap(); + for entry in std::fs::read_dir(cache.path()).unwrap() { + let path = entry.unwrap().path(); + if path.is_dir() { + std::fs::remove_dir_all(path).unwrap(); + } else { + std::fs::remove_file(path).unwrap(); + } + } + assert_eq!( + run_hostile(program, cache.path(), &["3"]).stdout, + "hello world 3\n" + ); + } + + #[test] + fn stress_many_concurrent_first_spawns() { + if host::current().is_windows() { + return; + } + let cache = tempfile::tempdir().unwrap(); + let cache_dir = cache.path().join("cold"); + let image = hello_fixture(); + let program = image.to_str().unwrap(); + std::thread::scope(|s| { + let handles: Vec<_> = (0..48) + .map(|i| { + let cache_dir = &cache_dir; + s.spawn(move || { + let arg = i.to_string(); + let out = run_hostile(program, cache_dir, &[&arg]); + assert!(out.success(), "spawn {i} stderr: {}", out.stderr); + assert_eq!(out.stdout, format!("hello world {i}\n")); + }) + }) + .collect(); + for h in handles { + h.join().unwrap(); + } + }); + } +} diff --git a/crates/fbuild-core/src/platform/ape/native.rs b/crates/fbuild-core/src/platform/ape/native.rs new file mode 100644 index 000000000..10cc02cbb --- /dev/null +++ b/crates/fbuild-core/src/platform/ape/native.rs @@ -0,0 +1,245 @@ +//! Host-native stand-ins for APE images, for spawners fbuild does not control. +//! +//! zccache runs the compiler itself (through `kernal-api`), so it can't be +//! routed through ` `. For those callers fbuild hands over a +//! `#!/bin/sh` shim the kernel executes directly. It keeps the image's file +//! name (zccache detects the compiler family from it) and `exec`s the loader +//! on the *original* image path, so tools that locate their own subprograms +//! relative to themselves (gcc → `../libexec/.../cc1`) still find them. The +//! shim also prepends fbuild's `ape` directory to `PATH`, so APE programs the +//! tool spawns in turn (gcc → `cc1`, `as`) resolve a loader too. +//! +//! Relocating an assimilated (native-header) copy instead was tried and +//! rejected: gcc then fails with "cannot execute 'cc1'". +//! +//! Shims live in content-addressed subdirectories of the APE cache +//! (`shim-/`). + +use std::path::Path; + +use crate::path::NormalizedPath; + +use super::super::host::{self, HostPlatform}; +use super::{ + CACHE_DIR_ENV, LOADER_ENV, Memo, image_key, install, is_ape_file, plan_launch_for, short_hash, +}; + +static NATIVE_MEMO: Memo = Memo::new(); + +/// A path the kernel can execute directly that behaves like `program`, when +/// `program` is an APE image on a host that can't exec it natively. `None` +/// for non-APE programs, on Windows (APE runs natively), or when nothing +/// could be materialized — callers then use `program` unchanged. +pub fn native_executable(program: &Path) -> Option { + let host = host::current(); + if host.is_windows() || !is_ape_file(program) { + return None; + } + let env = |key: &str| std::env::var_os(key).filter(|v| !v.is_empty()); + let dirs = super::cache_dirs(env(CACHE_DIR_ENV)); + native_executable_for( + host, + program, + &dirs, + env(LOADER_ENV).as_deref(), + env("PATH").as_deref(), + ) +} + +/// [`native_executable`] with every host input explicit, for tests. +pub fn native_executable_for( + host: HostPlatform, + program: &Path, + dirs: &[NormalizedPath], + loader_override: Option<&std::ffi::OsStr>, + path_var: Option<&std::ffi::OsStr>, +) -> Option { + if host.is_windows() || !is_ape_file(program) { + return None; + } + let image = std::path::absolute(program).ok()?; + let key = image_key(&image, host, dirs)?; + if let Some(hit) = NATIVE_MEMO.get(&key) { + return Some(hit); + } + let name = image.file_name()?.to_str()?.to_owned(); + let launch = plan_launch_for( + host, + image.as_os_str(), + None, + path_var, + loader_override, + dirs, + )?; + // A memfd path is only valid in this process's direct children; the + // shim's exec runs one level further down. + if super::super::selected::ape::is_anonymous(&launch.loader) { + return None; + } + let mut shim = String::from("#!/bin/sh\n"); + // The image's identity goes into the shim bytes (and so its cache path): + // zccache fingerprints a compiler by its path, mtime and size, so a shim + // that stayed byte-identical across an in-place compiler upgrade would let + // zccache reuse the old compiler's cache entries. + let mtime = key + .2 + .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok()) + .map_or(0, |d| d.as_nanos()); + shim.push_str(&format!( + "# fbuild-ape-image: len={} mtime={mtime}\n", + key.1 + )); + if let Some(dir) = &launch.ape_path_dir { + shim.push_str(&format!( + "PATH={}\"${{PATH:+:$PATH}}\"; export PATH\n", + sh_quote(dir)? + )); + } + shim.push_str(&format!( + "exec {} {} \"$@\"\n", + sh_quote(&launch.loader)?, + sh_quote(&launch.image)? + )); + let sub = format!("shim-{}", short_hash(shim.as_bytes())); + let native = install::install(dirs, Some(&sub), &name, shim.as_bytes())?; + NATIVE_MEMO.put(key, native.clone()); + Some(native) +} + +/// Single-quote `path` for `/bin/sh`. +fn sh_quote(path: &Path) -> Option { + let s = path.to_str()?; + Some(format!("'{}'", s.replace('\'', r"'\''"))) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::platform::host::{HostArch, HostOs}; + + const LINUX_X64: HostPlatform = HostPlatform::new(HostOs::Linux, HostArch::X86_64); + const WINDOWS: HostPlatform = HostPlatform::new(HostOs::Windows, HostArch::X86_64); + + fn hello() -> NormalizedPath { + NormalizedPath::new(Path::new(env!("CARGO_MANIFEST_DIR")).join("data/ape-hello/hello.com")) + } + + #[test] + fn shim_keeps_name_runs_original_image_and_exposes_ape_on_path() { + if host::current().is_windows() { + return; + } + let cache = tempfile::tempdir().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let image = dir.path().join("it's x86_64-linux-cosmo-gcc"); + std::fs::copy(hello(), &image).unwrap(); + let shim = native_executable_for( + host::current(), + &image, + &[NormalizedPath::new(cache.path())], + None, + None, + ) + .expect("shim"); + assert_eq!(shim.file_name(), image.file_name()); + assert!( + shim.relative_to(&NormalizedPath::from(cache.path())) + .is_some() + ); + assert!(!is_ape_file(&shim), "the shim itself is a plain script"); + let text = std::fs::read_to_string(&shim).unwrap(); + assert!( + text.starts_with("#!/bin/sh\n# fbuild-ape-image: len="), + "{text}" + ); + assert!(text.contains("\nPATH="), "{text}"); + // Spawned directly — no fbuild loader routing involved. + let out = crate::subprocess::run_command_blocking_retrying_exec_busy( + &[shim.to_str().unwrap(), "via", "shim"], + None, + Some(&[("PATH", "/nonexistent")]), + None, + ) + .unwrap(); + assert_eq!( + out.stdout, "hello world via shim\n", + "stderr: {}", + out.stderr + ); + // Memoized and stable. + assert_eq!( + native_executable_for( + host::current(), + &image, + &[NormalizedPath::new(cache.path())], + None, + None + ), + Some(shim) + ); + } + + /// An in-place compiler upgrade must change the shim zccache sees, or + /// zccache's path/mtime/size fingerprint would reuse the old compiler. + #[test] + fn upgrading_the_image_in_place_changes_the_shim() { + if host::current().is_windows() { + return; + } + let cache = tempfile::tempdir().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let image = dir.path().join("x86_64-linux-cosmo-gcc"); + std::fs::copy(hello(), &image).unwrap(); + let dirs = [NormalizedPath::new(cache.path())]; + let before = native_executable_for(host::current(), &image, &dirs, None, None).unwrap(); + let mut grown = std::fs::read(hello()).unwrap(); + grown.extend_from_slice(b"\0upgrade"); + std::fs::write(&image, grown).unwrap(); + let after = native_executable_for(host::current(), &image, &dirs, None, None).unwrap(); + assert_ne!(before, after); + assert_ne!( + std::fs::read(&before).unwrap(), + std::fs::read(&after).unwrap() + ); + assert_eq!(before.file_name(), after.file_name()); + } + + #[test] + fn windows_non_ape_and_unusable_cache_yield_none() { + let dir = tempfile::tempdir().unwrap(); + assert_eq!( + native_executable_for( + WINDOWS, + &hello(), + &[NormalizedPath::new(dir.path())], + None, + None + ), + None + ); + let script = dir.path().join("tool"); + std::fs::write(&script, "#!/bin/sh\n").unwrap(); + assert_eq!( + native_executable_for( + LINUX_X64, + &script, + &[NormalizedPath::new(dir.path())], + None, + None + ), + None + ); + let file_parent = dir.path().join("f"); + std::fs::write(&file_parent, "").unwrap(); + assert_eq!( + native_executable_for( + LINUX_X64, + &hello(), + &[NormalizedPath::new(file_parent.join("cache"))], + None, + None + ), + None + ); + } +} diff --git a/crates/fbuild-core/src/platform/ape/prologue.rs b/crates/fbuild-core/src/platform/ape/prologue.rs new file mode 100644 index 000000000..cf23ed924 --- /dev/null +++ b/crates/fbuild-core/src/platform/ape/prologue.rs @@ -0,0 +1,101 @@ +//! Parsing of the APE shell prologue: where an image keeps its per-CPU +//! loaders and the macOS loader source. +//! +//! cosmocc emits the same line shapes for every image; only the numbers +//! change. Recognized lines, keyed by the enclosing `if [ "$m" = ]`: +//! +//! * `dd if="$o" skip=N count=M … | gzip -dc >"$t.$$"` — a gzip'd loader. +//! When the next line patches it (`dd if="$t.$$" of="$t.$$" skip=5 count=8 +//! bs=64`) it is the macOS x86_64 loader (ELF blob turned Mach-O), otherwise +//! the Linux loader for the CPU. +//! * `dd if="$o" skip=N count=M … | gzip -dc >"$t.c.$$"` — the gzip'd C +//! source of the Apple Silicon loader (`ape-m1.c`), compiled with `cc`. + +use super::super::host::HostArch; + +/// Byte range `(skip, count)` within the image. +pub(crate) type Range = (u64, u64); + +/// Everything fbuild needs from one image's prologue. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub(crate) struct Prologue { + pub linux_loader_x86_64: Option, + pub linux_loader_aarch64: Option, + pub macos_loader_x86_64: Option, + pub macos_loader_source_aarch64: Option, +} + +impl Prologue { + pub(crate) fn linux_loader(&self, arch: HostArch) -> Option { + match arch { + HostArch::X86_64 => self.linux_loader_x86_64, + HostArch::Aarch64 => self.linux_loader_aarch64, + _ => None, + } + } +} + +/// Parse the prologue text at the start of an APE image. +pub(crate) fn parse(prologue: &[u8]) -> Prologue { + let text = String::from_utf8_lossy(prologue); + let mut out = Prologue::default(); + let mut cpu = None; + let mut lines = text.lines().map(str::trim).peekable(); + while let Some(line) = lines.next() { + if line.contains("\"$m\" = x86_64") { + cpu = Some(HostArch::X86_64); + } else if line.contains("\"$m\" = aarch64") { + cpu = Some(HostArch::Aarch64); + } + let Some(arch) = cpu else { continue }; + + if line.starts_with("dd if=\"$o\" ") && line.contains("| gzip -dc >\"$t.c.$$\"") { + if arch == HostArch::Aarch64 { + out.macos_loader_source_aarch64 = range(line); + } + } else if line.starts_with("dd if=\"$o\" ") && line.contains("| gzip -dc >\"$t.$$\"") { + let patched = lines + .peek() + .is_some_and(|next| next.starts_with("dd if=\"$t.$$\" of=\"$t.$$\"")); + match (arch, patched) { + (HostArch::X86_64, true) => out.macos_loader_x86_64 = range(line), + (HostArch::X86_64, false) => out.linux_loader_x86_64 = range(line), + (HostArch::Aarch64, false) => out.linux_loader_aarch64 = range(line), + _ => {} + } + } + } + out +} + +fn range(line: &str) -> Option { + let field = |key: &str| { + line.split_whitespace() + .find_map(|tok| tok.strip_prefix(key)) + .and_then(|v| v.parse::().ok()) + }; + Some((field("skip=")?, field("count=")?)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn fixture_prologue_yields_every_branch() { + let image = std::fs::read( + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("data/ape-hello/hello.com"), + ) + .unwrap(); + let p = parse(&image[..64 * 1024]); + assert_eq!(p.linux_loader_x86_64, Some((275392, 4180))); + assert_eq!(p.linux_loader_aarch64, Some((279572, 4928))); + assert_eq!(p.macos_loader_x86_64, Some((275392, 4180))); + assert_eq!(p.macos_loader_source_aarch64, Some((284500, 10590))); + } + + #[test] + fn garbage_parses_to_nothing() { + assert_eq!(parse(b"MZqFpD='\n\0\xff\xfe random\n"), Prologue::default()); + } +} diff --git a/crates/fbuild-core/src/platform/executable.rs b/crates/fbuild-core/src/platform/executable.rs index 6f39ce032..2975977f6 100644 --- a/crates/fbuild-core/src/platform/executable.rs +++ b/crates/fbuild-core/src/platform/executable.rs @@ -51,6 +51,117 @@ pub fn path_candidate_names(stem: &str) -> Vec { path_candidate_names_for(host::current(), stem) } +/// Return the ordered file names probed for a package-provided host tool. +/// +/// The host-native spelling comes first, then the cosmocc default `.com` +/// suffix, then the other host's spelling. Non-native spellings are only +/// accepted by [`find_tool_in_for`] when the file is an APE image, which runs +/// on every host. +pub fn tool_candidate_names_for(host: HostPlatform, stem: &str) -> Vec { + if host.is_windows() { + vec![ + format!("{stem}.exe"), + format!("{stem}.com"), + stem.to_owned(), + ] + } else { + vec![ + stem.to_owned(), + format!("{stem}.com"), + format!("{stem}.exe"), + ] + } +} + +/// Return the ordered file names probed for a host tool on the current host. +pub fn tool_candidate_names(stem: &str) -> Vec { + tool_candidate_names_for(host::current(), stem) +} + +/// Find a package-provided tool named `stem` in `dir` for an explicit host. +/// +/// Returns the first candidate from [`tool_candidate_names_for`] that is a +/// regular file and runnable on `host`: the host-native spelling is always +/// accepted, any other spelling only when it is an APE image (so a +/// Windows-only PE `tool.exe` is not picked on Linux, and an extensionless +/// shell script is not picked on Windows). +pub fn find_tool_in_for(host: HostPlatform, dir: &Path, stem: &str) -> Option { + let native = native_name_for(host, stem); + tool_candidate_names_for(host, stem) + .into_iter() + .map(|name| { + let is_native = name == native; + (dir.join(name), is_native) + }) + .find(|(path, is_native)| path.is_file() && (*is_native || super::ape::is_ape_file(path))) + .map(|(path, _)| NormalizedPath::from(path)) +} + +/// Find a package-provided tool named `stem` in `dir` on the current host. +pub fn find_tool_in(dir: &Path, stem: &str) -> Option { + find_tool_in_for(host::current(), dir, stem) +} + +/// Find a tool named `stem` anywhere below `root` (depth-first), applying +/// [`find_tool_in`]'s acceptance rules in each directory. Used for archives +/// whose binary sits under an unknown nesting of top-level folders. +pub fn find_tool_in_tree(root: &Path, stem: &str) -> Option { + if let Some(found) = find_tool_in(root, stem) { + return Some(found); + } + std::fs::read_dir(root) + .ok()? + .flatten() + .map(|entry| entry.path()) + .filter(|path| path.is_dir()) + .find_map(|dir| find_tool_in_tree(&dir, stem)) +} + +/// Find a tool on `PATH` (or any list of search directories) by stem, using +/// [`find_tool_in`]'s per-directory candidate order and acceptance rules. +pub fn find_tool_on_paths(dirs: I, stem: &str) -> Option +where + I: IntoIterator, + I::Item: AsRef, +{ + dirs.into_iter() + .find_map(|dir| find_tool_in(dir.as_ref(), stem)) +} + +/// Resolve a package-provided tool in `dir` for an explicit host, falling back +/// to the host-native spelling when no acceptable candidate exists. +pub fn resolve_tool_in_for(host: HostPlatform, dir: &Path, stem: &str) -> NormalizedPath { + find_tool_in_for(host, dir, stem) + .unwrap_or_else(|| NormalizedPath::from(dir.join(native_name_for(host, stem)))) +} + +/// Resolve a package-provided tool in `dir` on the current host. +/// +/// Returns [`find_tool_in`]'s hit, or `dir/` when nothing usable +/// exists so callers keep reporting the conventional missing path. +pub fn resolve_tool_in(dir: &Path, stem: &str) -> NormalizedPath { + resolve_tool_in_for(host::current(), dir, stem) +} + +/// Split a host-tool file name into its stem and executable suffix. +/// +/// Recognizes `.exe` and `.com` (case-insensitive); the returned suffix keeps +/// its leading dot and original case, or is empty when there is none. +pub fn split_tool_suffix(file_name: &str) -> (&str, &str) { + for suffix in [".exe", ".com"] { + let Some(split) = file_name.len().checked_sub(suffix.len()) else { + continue; + }; + if split > 0 + && file_name.is_char_boundary(split) + && file_name[split..].eq_ignore_ascii_case(suffix) + { + return (&file_name[..split], &file_name[split..]); + } + } + (file_name, "") +} + /// Discover the path of the currently running executable image. pub fn current_image() -> io::Result { std::env::current_exe().map(NormalizedPath::from) @@ -92,6 +203,7 @@ pub fn current_image_sibling_candidates(stem: &str) -> io::Result<[NormalizedPat #[cfg(test)] mod tests { + use crate::path::NormalizedPath; use crate::platform::host::{HostArch, HostOs, HostPlatform}; #[test] @@ -112,6 +224,115 @@ mod tests { assert_eq!(super::path_candidate_names_for(linux, "pio"), ["pio"]); } + const WINDOWS: HostPlatform = HostPlatform::new(HostOs::Windows, HostArch::X86_64); + const LINUX: HostPlatform = HostPlatform::new(HostOs::Linux, HostArch::X86_64); + const MACOS_ARM: HostPlatform = HostPlatform::new(HostOs::Macos, HostArch::Aarch64); + + fn write_ape(path: &std::path::Path) { + std::fs::write(path, b"MZqFpD='\n#!/bin/sh\n").expect("write fake APE"); + } + + fn write_plain(path: &std::path::Path, bytes: &[u8]) { + std::fs::write(path, bytes).expect("write plain file"); + } + + #[test] + fn tool_candidate_names_put_the_native_spelling_first_then_com() { + assert_eq!( + super::tool_candidate_names_for(WINDOWS, "gcc"), + ["gcc.exe", "gcc.com", "gcc"] + ); + assert_eq!( + super::tool_candidate_names_for(LINUX, "gcc"), + ["gcc", "gcc.com", "gcc.exe"] + ); + assert_eq!( + super::tool_candidate_names_for(MACOS_ARM, "gcc"), + ["gcc", "gcc.com", "gcc.exe"] + ); + } + + #[test] + fn find_tool_prefers_the_native_spelling_over_an_ape() { + let dir = tempfile::tempdir().unwrap(); + write_plain(&dir.path().join("tool"), b"\x7fELF"); + write_ape(&dir.path().join("tool.com")); + assert_eq!( + super::find_tool_in_for(LINUX, dir.path(), "tool"), + Some(NormalizedPath::from(dir.path().join("tool"))) + ); + } + + #[test] + fn find_tool_accepts_ape_com_and_exe_on_unix_hosts() { + let dir = tempfile::tempdir().unwrap(); + write_ape(&dir.path().join("tool.com")); + assert_eq!( + super::find_tool_in_for(LINUX, dir.path(), "tool"), + Some(NormalizedPath::from(dir.path().join("tool.com"))) + ); + + let dir = tempfile::tempdir().unwrap(); + write_ape(&dir.path().join("tool.exe")); + assert_eq!( + super::find_tool_in_for(MACOS_ARM, dir.path(), "tool"), + Some(NormalizedPath::from(dir.path().join("tool.exe"))) + ); + } + + #[test] + fn find_tool_refuses_a_non_ape_windows_pe_on_unix_hosts() { + let dir = tempfile::tempdir().unwrap(); + write_plain(&dir.path().join("tool.exe"), b"MZ\x90\x00\x03\x00\x00\x00"); + assert_eq!(super::find_tool_in_for(LINUX, dir.path(), "tool"), None); + assert_eq!( + super::resolve_tool_in_for(LINUX, dir.path(), "tool"), + NormalizedPath::from(dir.path().join("tool")) + ); + } + + #[test] + fn find_tool_on_windows_prefers_exe_then_ape_com_and_refuses_plain_scripts() { + let dir = tempfile::tempdir().unwrap(); + write_plain(&dir.path().join("tool"), b"#!/bin/sh\n"); + assert_eq!(super::find_tool_in_for(WINDOWS, dir.path(), "tool"), None); + assert_eq!( + super::resolve_tool_in_for(WINDOWS, dir.path(), "tool"), + NormalizedPath::from(dir.path().join("tool.exe")) + ); + + write_ape(&dir.path().join("tool.com")); + assert_eq!( + super::find_tool_in_for(WINDOWS, dir.path(), "tool"), + Some(NormalizedPath::from(dir.path().join("tool.com"))) + ); + + write_plain(&dir.path().join("tool.exe"), b"MZ\x90\x00"); + assert_eq!( + super::find_tool_in_for(WINDOWS, dir.path(), "tool"), + Some(NormalizedPath::from(dir.path().join("tool.exe"))) + ); + } + + #[test] + fn find_tool_accepts_an_extensionless_ape_on_windows_and_skips_directories() { + let dir = tempfile::tempdir().unwrap(); + std::fs::create_dir(dir.path().join("tool.exe")).unwrap(); + write_ape(&dir.path().join("tool")); + assert_eq!( + super::find_tool_in_for(WINDOWS, dir.path(), "tool"), + Some(NormalizedPath::from(dir.path().join("tool"))) + ); + } + + #[test] + fn split_tool_suffix_recognizes_exe_and_com() { + assert_eq!(super::split_tool_suffix("gcc.exe"), ("gcc", ".exe")); + assert_eq!(super::split_tool_suffix("gcc.COM"), ("gcc", ".COM")); + assert_eq!(super::split_tool_suffix("gcc"), ("gcc", "")); + assert_eq!(super::split_tool_suffix(".exe"), (".exe", "")); + } + #[test] fn current_image_and_sibling_discovery_share_the_same_parent() { let image = super::current_image().expect("current test image"); diff --git a/crates/fbuild-core/src/platform/fs.rs b/crates/fbuild-core/src/platform/fs.rs index dcdc3f827..634fde009 100644 --- a/crates/fbuild-core/src/platform/fs.rs +++ b/crates/fbuild-core/src/platform/fs.rs @@ -125,6 +125,23 @@ pub fn set_executable(path: &Path) -> std::io::Result<()> { super::selected::fs::set_executable(path) } +/// Return whether metadata carries execute permission where the host represents it. +pub fn is_executable(metadata: &std::fs::Metadata) -> bool { + super::selected::fs::is_executable(metadata) +} + +/// Create `dir` (owner-only where the host has permissions) if needed and +/// report whether it is a real directory that only the current user can +/// modify — safe to hold files fbuild will later execute. +pub fn ensure_private_dir(dir: &Path) -> bool { + super::selected::fs::ensure_private_dir(dir) +} + +/// Whether files under `dir` may be executed (false on `noexec` mounts). +pub fn mount_allows_exec(dir: &Path) -> bool { + super::selected::fs::mount_allows_exec(dir) +} + /// Ensure an extracted tool is executable without changing an already-runnable file. pub fn ensure_executable(path: &Path) -> std::io::Result<()> { super::selected::fs::ensure_executable(path) diff --git a/crates/fbuild-core/src/platform/linux/ape.rs b/crates/fbuild-core/src/platform/linux/ape.rs new file mode 100644 index 000000000..1b32a4027 --- /dev/null +++ b/crates/fbuild-core/src/platform/linux/ape.rs @@ -0,0 +1,214 @@ +//! Linux pieces of APE support: default cache directories and the sealed +//! `memfd` fallback used when no cache directory is usable (read-only home, +//! `noexec` `/tmp`, no runtime dir). The memfd is held open for the life of +//! the process and exec'd via `/proc/self/fd/N`. + +use std::collections::HashMap; +use std::ffi::CString; +use std::fs::File; +use std::io::Write; +use std::os::fd::{AsRawFd, FromRawFd}; +use std::os::unix::fs::PermissionsExt; +use crate::path::NormalizedPath; +use std::sync::Mutex; + +/// Default candidate directories, most durable first. +pub(crate) fn default_loader_dirs() -> Vec { + let env_dir = |key: &str| std::env::var_os(key).filter(|v| !v.is_empty()).map(NormalizedPath::new); + let mut dirs = Vec::new(); + if let Some(cache) = env_dir("XDG_CACHE_HOME").or_else(|| env_dir("HOME").map(|h| h.join(".cache"))) { + dirs.push(cache.join("fbuild").join("ape")); + } + if let Some(runtime) = env_dir("XDG_RUNTIME_DIR") { + dirs.push(runtime.join("fbuild").join("ape")); + } + dirs +} + +/// Whether `path` is one of this process's anonymous (memfd) executables. +/// Exact identity, not a prefix test: only paths handed out here qualify. +pub(crate) fn is_anonymous(path: &std::path::Path) -> bool { + let guard = MEMFDS.lock().unwrap_or_else(|e| e.into_inner()); + guard + .as_ref() + .is_some_and(|fds| fds.values().any(|file| fd_path(file).as_path() == path)) +} + +/// Last-resort executable with no filesystem home: a sealed memfd exec'd via +/// `/proc/self/fd/N`. Valid only for direct children of this process. +pub(crate) fn anonymous_executable(bytes: &[u8], name: &str) -> Option { + memfd_loader(bytes, name) +} + +/// Sealed memfds kept open for the process lifetime, keyed by loader name. +static MEMFDS: Mutex>> = Mutex::new(None); + +fn memfd_loader(bytes: &[u8], name: &str) -> Option { + let mut guard = MEMFDS.lock().unwrap_or_else(|e| e.into_inner()); + let fds = guard.get_or_insert_with(HashMap::new); + if let Some(file) = fds.get(name) { + return Some(fd_path(file)); + } + let file = { + // The memfd is writable until sealed; keep it out of forked children. + let _fork = crate::platform::process::exclusive_fork_guard(); + create_sealed_memfd(bytes, name)? + }; + let path = fd_path(&file); + // The child resolves this path before close-on-exec runs; without /proc + // it cannot, so refuse rather than hand out a dead path. + if !path.exists() { + return None; + } + fds.insert(name.to_owned(), file); + Some(path) +} + +fn create_sealed_memfd(bytes: &[u8], name: &str) -> Option { + let cname = CString::new(name).ok()?; + let base = libc::MFD_CLOEXEC | libc::MFD_ALLOW_SEALING; + // MFD_EXEC (Linux 6.3+) keeps the memfd executable under + // `vm.memfd_noexec=1`; older kernels reject the flag with EINVAL. + let fd = [base | libc::MFD_EXEC, base].into_iter().find_map(|flags| { + // Raw syscall, not the libc wrapper: release builds link glibc 2.17, + // which predates `memfd_create()` (added in glibc 2.27). + // SAFETY: `cname` is NUL-terminated; flags are valid memfd flags. + let fd = unsafe { libc::syscall(libc::SYS_memfd_create, cname.as_ptr(), flags) }; + (fd >= 0).then_some(fd as libc::c_int) + })?; + // SAFETY: `fd` is a freshly created descriptor owned by nobody else. + let mut file = unsafe { File::from_raw_fd(fd) }; + file.write_all(bytes).ok()?; + file.set_permissions(std::fs::Permissions::from_mode(0o500)) + .ok()?; + let seals = libc::F_SEAL_SHRINK | libc::F_SEAL_GROW | libc::F_SEAL_WRITE | libc::F_SEAL_SEAL; + // SAFETY: `file` owns a valid memfd created with MFD_ALLOW_SEALING. + if unsafe { libc::fcntl(file.as_raw_fd(), libc::F_ADD_SEALS, seals) } != 0 { + return None; + } + Some(file) +} + +fn fd_path(file: &File) -> NormalizedPath { + NormalizedPath::from(format!("/proc/self/fd/{}", file.as_raw_fd())) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::platform::ape::install::install_in; + use crate::platform::host::HostArch; + use std::fs::OpenOptions; + use std::path::Path; + + fn hello() -> NormalizedPath { + NormalizedPath::new(Path::new(env!("CARGO_MANIFEST_DIR")).join("data/ape-hello/hello.com")) + } + + fn host_loader() -> Vec { + let arch = crate::platform::host::current().arch(); + crate::platform::ape::extract_loader(&hello(), arch).expect("fixture embeds host loader") + } + + fn run(loader: &Path, args: &[&str]) -> crate::subprocess::ToolOutput { + let hello = hello(); + let mut argv = vec![loader.to_str().unwrap(), hello.to_str().unwrap()]; + argv.extend_from_slice(args); + // Hostile child env: nothing on PATH, no TMPDIR, no HOME. + crate::subprocess::run_command_blocking( + &argv, + None, + Some(&[("PATH", "/nonexistent"), ("TMPDIR", "/nonexistent"), ("HOME", "/nonexistent")]), + None, + ) + .expect("loader must spawn") + } + + #[test] + fn memfd_loader_runs_real_image() { + let loader = memfd_loader(&host_loader(), "ape-loader-test-memfd").expect("memfd"); + assert!(is_anonymous(&loader)); + let out = run(&loader, &["memfd"]); + assert!(out.success(), "stderr: {}", out.stderr); + assert_eq!(out.stdout, "hello world memfd\n"); + // Sealed: the running loader can't be rewritten underneath children. + assert!(OpenOptions::new().write(true).open(&loader).and_then(|mut f| f.write_all(b"x")).is_err()); + } + + #[test] + fn unusable_dirs_fall_back_to_memfd() { + let root = tempfile::tempdir().unwrap(); + // A regular file as parent: uncreatable even for root, unlike a 0500 + // dir (root bypasses DAC permissions). + let readonly = root.path().join("not-a-dir"); + std::fs::write(&readonly, b"").unwrap(); + let shared = root.path().join("shared"); + std::fs::create_dir(&shared).unwrap(); + std::fs::set_permissions(&shared, std::fs::Permissions::from_mode(0o777)).unwrap(); + let dirs = [ + NormalizedPath::new(readonly.join("cache")), + NormalizedPath::new(&shared), + NormalizedPath::from("/proc/fbuild-nope"), + ]; + let loader = + crate::platform::ape::materialize(&host_loader(), "ape-loader-test-fallback", &dirs).unwrap(); + assert!(is_anonymous(&loader), "got {}", loader.display()); + assert!(std::fs::read_dir(&shared).unwrap().next().is_none(), "nothing planted in shared dir"); + let out = run(&loader, &["fallback"]); + assert_eq!(out.stdout, "hello world fallback\n", "stderr: {}", out.stderr); + } + + #[test] + fn symlinked_cache_dir_is_rejected() { + let root = tempfile::tempdir().unwrap(); + std::fs::create_dir(root.path().join("real")).unwrap(); + std::os::unix::fs::symlink(root.path().join("real"), root.path().join("link")).unwrap(); + assert!(install_in(&root.path().join("link"), &host_loader(), "l").is_none()); + } + + #[test] + fn tampered_or_truncated_install_is_replaced() { + let dir = tempfile::tempdir().unwrap(); + let bytes = host_loader(); + let path = install_in(dir.path(), &bytes, "ape-loader-t").expect("install"); + for bad in [&b"#!/bin/sh\nexit 66\n"[..], &bytes[..bytes.len() / 2]] { + std::fs::write(&path, bad).unwrap(); + assert_eq!(install_in(dir.path(), &bytes, "ape-loader-t"), Some(path.clone())); + assert_eq!(std::fs::read(&path).unwrap(), bytes); + let out = run(&path, &["repaired"]); + assert_eq!(out.stdout, "hello world repaired\n", "stderr: {}", out.stderr); + } + // Lost exec bit is also repaired. + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); + install_in(dir.path(), &bytes, "ape-loader-t").unwrap(); + assert_ne!(std::fs::metadata(&path).unwrap().permissions().mode() & 0o100, 0); + } + + #[test] + fn concurrent_installs_converge_without_partial_files() { + let dir = tempfile::tempdir().unwrap(); + let cache = dir.path().join("fresh/ape"); + let bytes = host_loader(); + let paths: Vec<_> = std::thread::scope(|s| { + let handles: Vec<_> = (0..32) + .map(|_| s.spawn(|| install_in(&cache, &bytes, "ape-loader-race"))) + .collect(); + handles.into_iter().map(|h| h.join().unwrap()).collect() + }); + assert!(paths.iter().all(|p| p.as_deref() == Some(cache.join("ape-loader-race").as_path()))); + let entries: Vec<_> = std::fs::read_dir(&cache).unwrap().map(|e| e.unwrap().file_name()).collect(); + assert_eq!(entries, vec![std::ffi::OsString::from("ape-loader-race")], "no stray temp files"); + assert_eq!( + std::fs::metadata(&cache).unwrap().permissions().mode() & 0o777, + 0o700 + ); + } + + #[test] + fn foreign_arch_loader_is_valid_but_not_host_runnable_choice() { + // Both CPUs are present in the fat fixture; only the host one is used. + assert!(crate::platform::ape::extract_loader(&hello(), HostArch::X86_64).is_some()); + assert!(crate::platform::ape::extract_loader(&hello(), HostArch::Aarch64).is_some()); + assert!(crate::platform::ape::extract_loader(&hello(), HostArch::Arm).is_none()); + } +} diff --git a/crates/fbuild-core/src/platform/linux/fs.rs b/crates/fbuild-core/src/platform/linux/fs.rs index 68b4269c1..c13015b6d 100644 --- a/crates/fbuild-core/src/platform/linux/fs.rs +++ b/crates/fbuild-core/src/platform/linux/fs.rs @@ -100,6 +100,38 @@ pub(crate) fn set_executable(path: &Path) -> std::io::Result<()> { std::fs::set_permissions(path, permissions) } +pub(crate) fn ensure_private_dir(dir: &Path) -> bool { + use std::os::unix::fs::{DirBuilderExt, MetadataExt}; + let _ = std::fs::DirBuilder::new() + .recursive(true) + .mode(0o700) + .create(dir); + // SAFETY: geteuid has no preconditions and cannot fail. + let uid = unsafe { libc::geteuid() }; + std::fs::symlink_metadata(dir).is_ok_and(|meta| { + meta.file_type().is_dir() && meta.uid() == uid && meta.mode() & 0o022 == 0 + }) +} + +pub(crate) fn mount_allows_exec(dir: &Path) -> bool { + use std::os::unix::ffi::OsStrExt; + let Ok(path) = std::ffi::CString::new(dir.as_os_str().as_bytes()) else { + return false; + }; + let mut stats = std::mem::MaybeUninit::::uninit(); + // SAFETY: `path` is NUL-terminated and `stats` points to writable storage. + if unsafe { libc::statvfs(path.as_ptr(), stats.as_mut_ptr()) } != 0 { + return false; + } + // SAFETY: successful statvfs initialized the complete output structure. + let stats = unsafe { stats.assume_init() }; + stats.f_flag & libc::ST_NOEXEC == 0 +} + +pub(crate) fn is_executable(metadata: &std::fs::Metadata) -> bool { + metadata.permissions().mode() & 0o111 != 0 +} + pub(crate) fn ensure_executable(path: &Path) -> std::io::Result<()> { let permissions = std::fs::metadata(path)?.permissions(); if permissions.mode() & 0o111 == 0 { diff --git a/crates/fbuild-core/src/platform/linux/mod.rs b/crates/fbuild-core/src/platform/linux/mod.rs index a45bca1f5..69496313e 100644 --- a/crates/fbuild-core/src/platform/linux/mod.rs +++ b/crates/fbuild-core/src/platform/linux/mod.rs @@ -1,5 +1,6 @@ use super::host::{HostArch, HostOs}; +pub(super) mod ape; pub(super) mod device; pub(super) mod fs; pub(super) mod host; diff --git a/crates/fbuild-core/src/platform/macos/ape.rs b/crates/fbuild-core/src/platform/macos/ape.rs new file mode 100644 index 000000000..fe2a37ab2 --- /dev/null +++ b/crates/fbuild-core/src/platform/macos/ape.rs @@ -0,0 +1,30 @@ +//! macOS pieces of APE support: default cache directories. macOS has no +//! anonymous-executable fallback (no memfd), so a usable cache directory is +//! required; `$TMPDIR` is already per-user there. + +use crate::path::NormalizedPath; + +/// Default candidate directories, most durable first. +pub(crate) fn default_loader_dirs() -> Vec { + let env_dir = |key: &str| { + std::env::var_os(key) + .filter(|v| !v.is_empty()) + .map(NormalizedPath::new) + }; + let mut dirs = Vec::new(); + if let Some(cache) = env_dir("XDG_CACHE_HOME") { + dirs.push(cache.join("fbuild").join("ape")); + } + if let Some(home) = env_dir("HOME") { + dirs.push(home.join("Library").join("Caches").join("fbuild").join("ape")); + } + dirs +} + +pub(crate) fn anonymous_executable(_bytes: &[u8], _name: &str) -> Option { + None +} + +pub(crate) fn is_anonymous(_path: &std::path::Path) -> bool { + false +} diff --git a/crates/fbuild-core/src/platform/macos/fs.rs b/crates/fbuild-core/src/platform/macos/fs.rs index f1687bd0b..16db20712 100644 --- a/crates/fbuild-core/src/platform/macos/fs.rs +++ b/crates/fbuild-core/src/platform/macos/fs.rs @@ -100,6 +100,38 @@ pub(crate) fn set_executable(path: &Path) -> std::io::Result<()> { std::fs::set_permissions(path, permissions) } +pub(crate) fn ensure_private_dir(dir: &Path) -> bool { + use std::os::unix::fs::{DirBuilderExt, MetadataExt}; + let _ = std::fs::DirBuilder::new() + .recursive(true) + .mode(0o700) + .create(dir); + // SAFETY: geteuid has no preconditions and cannot fail. + let uid = unsafe { libc::geteuid() }; + std::fs::symlink_metadata(dir).is_ok_and(|meta| { + meta.file_type().is_dir() && meta.uid() == uid && meta.mode() & 0o022 == 0 + }) +} + +pub(crate) fn mount_allows_exec(dir: &Path) -> bool { + use std::os::unix::ffi::OsStrExt; + let Ok(path) = std::ffi::CString::new(dir.as_os_str().as_bytes()) else { + return false; + }; + let mut stats = std::mem::MaybeUninit::::uninit(); + // SAFETY: `path` is NUL-terminated and `stats` points to writable storage. + if unsafe { libc::statfs(path.as_ptr(), stats.as_mut_ptr()) } != 0 { + return false; + } + // SAFETY: successful statfs initialized the complete output structure. + let stats = unsafe { stats.assume_init() }; + stats.f_flags & (libc::MNT_NOEXEC as u32) == 0 +} + +pub(crate) fn is_executable(metadata: &std::fs::Metadata) -> bool { + metadata.permissions().mode() & 0o111 != 0 +} + pub(crate) fn ensure_executable(path: &Path) -> std::io::Result<()> { let permissions = std::fs::metadata(path)?.permissions(); if permissions.mode() & 0o111 == 0 { diff --git a/crates/fbuild-core/src/platform/macos/mod.rs b/crates/fbuild-core/src/platform/macos/mod.rs index c86cf5e6e..de4eaed90 100644 --- a/crates/fbuild-core/src/platform/macos/mod.rs +++ b/crates/fbuild-core/src/platform/macos/mod.rs @@ -1,5 +1,6 @@ use super::host::{HostArch, HostOs}; +pub(super) mod ape; pub(super) mod device; pub(super) mod fs; pub(super) mod host; diff --git a/crates/fbuild-core/src/platform/mod.rs b/crates/fbuild-core/src/platform/mod.rs index cc2f0b6a9..e1ec9e0e1 100644 --- a/crates/fbuild-core/src/platform/mod.rs +++ b/crates/fbuild-core/src/platform/mod.rs @@ -4,6 +4,7 @@ //! Callers use the capability modules and cannot name the private concrete //! implementation selected below. +pub mod ape; pub mod device; pub mod executable; pub mod fs; diff --git a/crates/fbuild-core/src/platform/process.rs b/crates/fbuild-core/src/platform/process.rs index 32895fb8f..0ca65d0eb 100644 --- a/crates/fbuild-core/src/platform/process.rs +++ b/crates/fbuild-core/src/platform/process.rs @@ -1,8 +1,9 @@ //! Neutral process, containment, and exit-interpretation APIs. +use std::ffi::OsStr; use std::fs::File; use std::process::{ChildStderr, ChildStdin, ChildStdout, Command, ExitStatus}; -use std::sync::OnceLock; +use std::sync::{OnceLock, RwLock, RwLockReadGuard, RwLockWriteGuard}; use std::time::{Duration, Instant}; use crate::path::NormalizedPath; @@ -102,6 +103,77 @@ pub fn containment_is_initialized() -> bool { CONTAINMENT.get().is_some() } +/// Process-wide fork lock, as Go's `syscall.ForkLock`. +/// +/// A file this process writes and then executes fails with `ETXTBSY` if a +/// sibling thread forks while the writable descriptor is open: the forked +/// child holds it until its own `exec`. Every spawn helper here holds the +/// lock shared across fork→exec (`spawn` returns only after the child has +/// exec'd); writers of to-be-executed files hold it exclusively while their +/// descriptor is open, so no child can inherit it. +static FORK_LOCK: RwLock<()> = RwLock::new(()); + +/// Hold across a fork→exec that bypasses this module's spawn helpers (the +/// few allowlisted direct `Command` spawns), so the child can't inherit a +/// loader that [`exclusive_fork_guard`] is writing. Release it before +/// waiting on the child. +pub fn shared_fork_guard() -> RwLockReadGuard<'static, ()> { + FORK_LOCK.read().unwrap_or_else(|e| e.into_inner()) +} + +/// Hold while a file that will be executed is open for writing. +pub fn exclusive_fork_guard() -> RwLockWriteGuard<'static, ()> { + FORK_LOCK.write().unwrap_or_else(|e| e.into_inner()) +} + +/// Construct a [`Command`] for `program`, routing an APE (cosmocc) image +/// through a loader on hosts that cannot exec APE natively (see +/// [`super::ape`]). Use in place of `Command::new` for any external tool. +/// +/// A relative `program` path is resolved against the parent's working +/// directory; pass an absolute path when the child's `current_dir` differs. +pub fn command(program: impl AsRef) -> Command { + match super::ape::plan_launch(program.as_ref(), None, None) { + Some(launch) => { + // allow-direct-spawn: construction only; callers execute via spawn_contained/spawn_tokio_contained/spawn_detached. + let mut command = Command::new(&launch.loader); + command.arg(&launch.image); + if let Some(path) = launch.child_path(std::env::var_os("PATH").as_deref()) { + command.env("PATH", path); + } + command + } + // allow-direct-spawn: construction only; callers execute via spawn_contained/spawn_tokio_contained/spawn_detached. + None => Command::new(program), + } +} + +/// Tokio counterpart of [`command`]. +pub fn tokio_command(program: impl AsRef) -> tokio::process::Command { + match super::ape::plan_launch(program.as_ref(), None, None) { + Some(launch) => { + // allow-direct-spawn: construction only; callers execute via spawn_contained/spawn_tokio_contained/spawn_detached. + let mut command = tokio::process::Command::new(&launch.loader); + command.arg(&launch.image); + if let Some(path) = launch.child_path(std::env::var_os("PATH").as_deref()) { + command.env("PATH", path); + } + command + } + // allow-direct-spawn: construction only; callers execute via spawn_contained/spawn_tokio_contained/spawn_detached. + None => tokio::process::Command::new(program), + } +} + +/// APE launch plan for a `run_command`-style spawn with its child cwd and env overlay. +pub(crate) fn ape_launch( + program: &str, + cwd: Option<&std::path::Path>, + overlay: Option<&[(&str, &str)]>, +) -> Option { + super::ape::plan_launch(OsStr::new(program), cwd, overlay) +} + /// Spawn a synchronous child using running-process's contained process tree. pub fn spawn_contained( command: &mut Command, @@ -113,9 +185,12 @@ pub fn spawn_contained( stderr: into_running_stdio(stdio.stderr), ..running_process::SpawnStdio::default() }; - let inner = match CONTAINMENT.get() { - Some(group) => group.spawn(command, stdio)?, - None => running_process::spawn(command, stdio)?, + let inner = { + let _fork = shared_fork_guard(); + match CONTAINMENT.get() { + Some(group) => group.spawn(command, stdio)?, + None => running_process::spawn(command, stdio)?, + } }; Ok(ContainedChild { inner }) } @@ -138,6 +213,7 @@ pub fn spawn_detached( stderr: Option<&File>, environment: DetachedEnvironment, ) -> std::io::Result { + let _fork = shared_fork_guard(); super::selected::process::spawn_detached(command, stderr, environment) } @@ -166,8 +242,10 @@ fn spawn_tokio_contained_here( } super::selected::process::configure_tokio_owner_death(command)?; } - let child = - running_process::spawn_tokio(command, running_process::TokioSpawnOptions::default())?; + let child = { + let _fork = shared_fork_guard(); + running_process::spawn_tokio(command, running_process::TokioSpawnOptions::default())? + }; if CONTAINMENT.get().is_some() { super::selected::process::after_tokio_spawn(&child)?; } diff --git a/crates/fbuild-core/src/platform/windows/ape.rs b/crates/fbuild-core/src/platform/windows/ape.rs new file mode 100644 index 000000000..851dc29c1 --- /dev/null +++ b/crates/fbuild-core/src/platform/windows/ape.rs @@ -0,0 +1,16 @@ +//! Windows runs APE images natively (an APE file is a valid PE), so there is +//! nothing to extract or cache. + +use crate::path::NormalizedPath; + +pub(crate) fn default_loader_dirs() -> Vec { + Vec::new() +} + +pub(crate) fn anonymous_executable(_bytes: &[u8], _name: &str) -> Option { + None +} + +pub(crate) fn is_anonymous(_path: &std::path::Path) -> bool { + false +} diff --git a/crates/fbuild-core/src/platform/windows/fs.rs b/crates/fbuild-core/src/platform/windows/fs.rs index 1f120a927..34302921e 100644 --- a/crates/fbuild-core/src/platform/windows/fs.rs +++ b/crates/fbuild-core/src/platform/windows/fs.rs @@ -252,6 +252,19 @@ pub(crate) fn set_executable(_path: &Path) -> std::io::Result<()> { Ok(()) } +pub(crate) fn ensure_private_dir(dir: &Path) -> bool { + std::fs::create_dir_all(dir).is_ok() + && std::fs::symlink_metadata(dir).is_ok_and(|meta| meta.file_type().is_dir()) +} + +pub(crate) fn mount_allows_exec(_dir: &Path) -> bool { + true +} + +pub(crate) fn is_executable(_metadata: &std::fs::Metadata) -> bool { + true +} + pub(crate) fn ensure_executable(_path: &Path) -> std::io::Result<()> { Ok(()) } diff --git a/crates/fbuild-core/src/platform/windows/mod.rs b/crates/fbuild-core/src/platform/windows/mod.rs index 397e59c1a..503c81732 100644 --- a/crates/fbuild-core/src/platform/windows/mod.rs +++ b/crates/fbuild-core/src/platform/windows/mod.rs @@ -1,5 +1,6 @@ use super::host::{HostArch, HostOs}; +pub(super) mod ape; pub(super) mod device; pub(super) mod fs; pub(super) mod host; diff --git a/crates/fbuild-core/src/platformio_package.rs b/crates/fbuild-core/src/platformio_package.rs index 9e9d776eb..a502c47a1 100644 --- a/crates/fbuild-core/src/platformio_package.rs +++ b/crates/fbuild-core/src/platformio_package.rs @@ -520,6 +520,8 @@ struct RegistryVersion { #[derive(Deserialize)] struct RegistryFile { + /// Host selector; a missing or `null` field means universal, like `"*"`. + #[serde(default)] system: serde_json::Value, download_url: String, checksum: Option, @@ -559,6 +561,7 @@ fn file_matches(file: &RegistryFile, system: &str) -> Option { match &file.system { serde_json::Value::String(s) if s == system => Some(2), serde_json::Value::String(s) if s == "*" => Some(1), + serde_json::Value::Null => Some(1), serde_json::Value::Array(systems) if systems.iter().any(|s| s.as_str() == Some(system)) => { Some(2) } diff --git a/crates/fbuild-core/src/subprocess.rs b/crates/fbuild-core/src/subprocess.rs index be7a7bdc9..e3eebee7e 100644 --- a/crates/fbuild-core/src/subprocess.rs +++ b/crates/fbuild-core/src/subprocess.rs @@ -242,10 +242,7 @@ async fn run_command_inner( if args.is_empty() { return Err(FbuildError::Other("empty command".to_string())); } - let mut cmd = build_command( - args, cwd, env, /*capture=*/ true, /*stdin_piped=*/ false, - )?; - let child = process::spawn_tokio_contained(&mut cmd).map_err(|e| spawn_err(args, e))?; + let child = spawn_retrying_ape_busy(args, cwd, env, true, false).await?; wait_and_capture(child, args, timeout).await } @@ -287,10 +284,7 @@ async fn run_command_with_stdin_inner( if args.is_empty() { return Err(FbuildError::Other("empty command".to_string())); } - let mut cmd = build_command( - args, cwd, env, /*capture=*/ true, /*stdin_piped=*/ true, - )?; - let mut child = process::spawn_tokio_contained(&mut cmd).map_err(|e| spawn_err(args, e))?; + let mut child = spawn_retrying_ape_busy(args, cwd, env, true, true).await?; // Take the stdin handle and concurrently write the payload while // tokio drains stdout/stderr in the background. Dropping `stdin` @@ -374,10 +368,7 @@ async fn run_command_passthrough_inner( if args.is_empty() { return Err(FbuildError::Other("empty command".to_string())); } - let mut cmd = build_command( - args, cwd, env, /*capture=*/ false, /*stdin_piped=*/ false, - )?; - let mut child = process::spawn_tokio_contained(&mut cmd).map_err(|e| spawn_err(args, e))?; + let mut child = spawn_retrying_ape_busy(args, cwd, env, false, false).await?; let status = match wait_with_timeout(&mut child, timeout).await? { Some(status) => status, None => { @@ -598,6 +589,39 @@ fn exit_code_from(status: std::process::ExitStatus) -> i32 { process::exit_code(status) } +/// Attempts for an APE launch whose freshly installed loader is momentarily +/// busy. fbuild's own spawns hold the fork lock, but spawners it doesn't +/// control (zccache via kernal-api) can fork while a loader is being written; +/// that child holds the writable descriptor until its own exec, microseconds. +const APE_EXEC_BUSY_ATTEMPTS: u32 = 8; + +/// Build and spawn `args`, retrying `ETXTBSY` only for APE launches (whose +/// loader fbuild may have just written). Every other error returns at once. +async fn spawn_retrying_ape_busy( + args: &[&str], + cwd: Option<&Path>, + env: Option<&[(&str, &str)]>, + capture: bool, + stdin_piped: bool, +) -> Result { + let mut attempt = 1; + loop { + let mut cmd = build_command(args, cwd, env, capture, stdin_piped)?; + match process::spawn_tokio_contained(&mut cmd) { + Ok(child) => return Ok(child), + Err(error) + if is_exec_busy(&error) + && attempt < APE_EXEC_BUSY_ATTEMPTS + && process::ape_launch(args[0], cwd, env).is_some() => + { + tokio::time::sleep(EXEC_BUSY_BACKOFF * attempt).await; + attempt += 1; + } + Err(error) => return Err(spawn_err(args, error)), + } + } +} + fn build_command( args: &[&str], cwd: Option<&Path>, @@ -605,7 +629,38 @@ fn build_command( capture: bool, stdin_piped: bool, ) -> Result { - let mut cmd = TokioCommand::new(args[0]); + // APE (cosmocc) images can't be exec'd directly on hosts without an APE + // binfmt handler (e.g. NixOS); route them through a loader. + let launch = process::ape_launch(args[0], cwd, env); + let mut cmd = match &launch { + Some(launch) => { + let mut cmd = TokioCommand::new(&launch.loader); + cmd.arg(&launch.image); + cmd + } + None => TokioCommand::new(args[0]), + }; + // Expose fbuild's loader as `ape` on the child's PATH so APE programs the + // tool spawns itself (gcc -> cc1) resolve a loader too. + let ape_path = launch.as_ref().and_then(|launch| { + let inherited = env + .and_then(|vars| vars.iter().rev().find(|(k, _)| *k == "PATH")) + .map(|(_, v)| std::ffi::OsString::from(*v)) + .or_else(|| std::env::var_os("PATH")); + launch.child_path(inherited.as_deref())?.into_string().ok() + }); + let overlay: Option> = match (&ape_path, env) { + (Some(path), env) => Some( + env.unwrap_or_default() + .iter() + .copied() + .filter(|(k, _)| *k != "PATH") + .chain([("PATH", path.as_str())]) + .collect(), + ), + (None, env) => env.map(<[_]>::to_vec), + }; + let env = overlay.as_deref(); if args.len() > 1 { cmd.args(&args[1..]); } diff --git a/crates/fbuild-core/tests/it/main.rs b/crates/fbuild-core/tests/it/main.rs index 80ded676f..abccb6b2f 100644 --- a/crates/fbuild-core/tests/it/main.rs +++ b/crates/fbuild-core/tests/it/main.rs @@ -8,3 +8,4 @@ mod platformio_avr_resolution; mod platformio_ch32v_resolution; mod platformio_esp8266_resolution; mod platformio_package_resolution; +mod platformio_registry_system; diff --git a/crates/fbuild-core/tests/it/platformio_package_resolution.rs b/crates/fbuild-core/tests/it/platformio_package_resolution.rs index a7686f64d..676053131 100644 --- a/crates/fbuild-core/tests/it/platformio_package_resolution.rs +++ b/crates/fbuild-core/tests/it/platformio_package_resolution.rs @@ -139,6 +139,9 @@ fn resolves_exact_platform_payload_and_checksum() { ); } +/// A registry file with no `system` (or `"system": null`) is a host-universal +/// payload, e.g. one APE build of a tool for every host. It must parse and +/// resolve like `"*"`, while an exact host match still wins. #[test] fn selects_host_file_and_range_without_substituting_another_build() { let response = r#"{ diff --git a/crates/fbuild-core/tests/it/platformio_registry_system.rs b/crates/fbuild-core/tests/it/platformio_registry_system.rs new file mode 100644 index 000000000..5fc0bcf7c --- /dev/null +++ b/crates/fbuild-core/tests/it/platformio_registry_system.rs @@ -0,0 +1,53 @@ +//! Registry files that omit or null `system` are universal packages. + +use fbuild_core::platformio_package::{PackageKind, parse_package_spec, resolve_registry_json}; + +#[test] +fn registry_file_without_system_is_universal_and_exact_host_still_wins() { + let system = + fbuild_core::platformio_package::host_system(fbuild_core::platform::host::current()) + .unwrap_or("linux_x86_64"); + let spec = parse_package_spec("acme/tool-ape@1.0.0").unwrap(); + let sha = "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"; + + for universal in [ + format!( + r#"{{"download_url":"https://example.test/ape.zip","checksum":{{"sha256":"{sha}"}}}}"# + ), + format!( + r#"{{"system":null,"download_url":"https://example.test/ape.zip","checksum":{{"sha256":"{sha}"}}}}"# + ), + format!( + r#"{{"system":"*","download_url":"https://example.test/ape.zip","checksum":{{"sha256":"{sha}"}}}}"# + ), + ] { + let response = format!( + r#"{{"name":"tool-ape","owner":{{"username":"acme"}}, + "versions":[{{"name":"1.0.0","files":[{universal}]}}]}}"# + ); + let payload = resolve_registry_json( + spec.registry().unwrap(), + PackageKind::Tool, + system, + &response, + ) + .unwrap_or_else(|e| panic!("{universal} should resolve on {system}: {e}")); + assert_eq!(payload.url, "https://example.test/ape.zip"); + } + + let response = format!( + r#"{{"name":"tool-ape","owner":{{"username":"acme"}}, + "versions":[{{"name":"1.0.0","files":[ + {{"download_url":"https://example.test/ape.zip","checksum":{{"sha256":"{sha}"}}}}, + {{"system":["{system}"],"download_url":"https://example.test/native.zip","checksum":{{"sha256":"{sha}"}}}} + ]}}]}}"# + ); + let payload = resolve_registry_json( + spec.registry().unwrap(), + PackageKind::Tool, + system, + &response, + ) + .unwrap(); + assert_eq!(payload.url, "https://example.test/native.zip"); +} diff --git a/crates/fbuild-daemon/src/handlers/emulator/avr8js_headless.rs b/crates/fbuild-daemon/src/handlers/emulator/avr8js_headless.rs index 65e46a49d..32c8e7338 100644 --- a/crates/fbuild-daemon/src/handlers/emulator/avr8js_headless.rs +++ b/crates/fbuild-daemon/src/handlers/emulator/avr8js_headless.rs @@ -61,7 +61,7 @@ pub(crate) async fn run_avr8js_headless( } // allow-direct-spawn: tokio streaming emulator; blocking NativeProcess unsuitable. - let mut cmd = tokio::process::Command::new(node_path); + let mut cmd = fbuild_core::platform::process::tokio_command(node_path); cmd.arg(script_path) .arg("--hex") .arg(hex_path) diff --git a/crates/fbuild-daemon/src/handlers/emulator/runners.rs b/crates/fbuild-daemon/src/handlers/emulator/runners.rs index a781230a9..28069ee20 100644 --- a/crates/fbuild-daemon/src/handlers/emulator/runners.rs +++ b/crates/fbuild-daemon/src/handlers/emulator/runners.rs @@ -240,7 +240,22 @@ impl EmulatorRunner for Avr8jsRunner { /// - macOS: `brew install simavr` /// - Windows: build from source (MSYS2/MinGW) — limited support async fn find_simavr() -> fbuild_core::Result { - let simavr = fbuild_core::platform::executable::name("simavr", "simavr.exe"); + // Prefer an explicit PATH hit so a `simavr.com` APE is found on every + // host; otherwise probe the bare native name (the OS resolves PATH). + let resolved = std::env::var_os("PATH") + .and_then(|path| { + fbuild_core::platform::executable::find_tool_on_paths( + std::env::split_paths(&path), + "simavr", + ) + }) + .map(|path| path.to_string_lossy().into_owned()); + let simavr = resolved + .as_deref() + .unwrap_or(fbuild_core::platform::executable::name( + "simavr", + "simavr.exe", + )); // Try running simavr to verify it exists; route through containment // (issue #32). This is a short-lived probe so the containment // difference is purely consistency. diff --git a/crates/fbuild-daemon/src/handlers/emulator/shared.rs b/crates/fbuild-daemon/src/handlers/emulator/shared.rs index 0981113a0..1b708cfb5 100644 --- a/crates/fbuild-daemon/src/handlers/emulator/shared.rs +++ b/crates/fbuild-daemon/src/handlers/emulator/shared.rs @@ -238,7 +238,7 @@ pub(crate) async fn run_qemu_process( options: RunQemuOptions<'_>, ) -> fbuild_core::Result { // allow-direct-spawn: tokio streaming QEMU emulator; blocking NativeProcess unsuitable. - let mut cmd = tokio::process::Command::new(qemu_path); + let mut cmd = fbuild_core::platform::process::tokio_command(qemu_path); cmd.args(args) .stdin(Stdio::null()) .stdout(Stdio::piped()) diff --git a/crates/fbuild-daemon/src/handlers/operations/deploy.rs b/crates/fbuild-daemon/src/handlers/operations/deploy.rs index 9dae54f64..b10c293a5 100644 --- a/crates/fbuild-daemon/src/handlers/operations/deploy.rs +++ b/crates/fbuild-daemon/src/handlers/operations/deploy.rs @@ -6,6 +6,7 @@ use super::common::{ resolve_client_path, resolve_request_project_dir, trust_device_hash_enabled, }; use super::deploy_port::{append_warning_to_stderr, choose_deploy_port}; +use super::deploy_teensy::find_teensy_loader_cli; use super::monitor::{MonitorOutcome, run_monitor_loop}; use crate::context::DaemonContext; use crate::models::{DeployRequest, OperationResponse}; @@ -18,52 +19,6 @@ use std::sync::Arc; #[cfg(feature = "espflash-native")] use super::common::{native_verify_enabled, native_write_enabled}; -/// Resolve a usable `teensy_loader_cli` binary for the Teensy deploy arm. -/// -/// Search order: -/// 1. `$PATH` (`teensy_loader_cli` on Unix, `teensy_loader_cli.exe` on Win) -/// 2. `~/.platformio/packages/tool-teensy/teensy_loader_cli{.exe}` — the -/// well-known path PlatformIO installs it at on every PIO-using machine. -/// -/// Returns `None` if neither is found; the TeensyDeployer's default will then -/// try a bare `teensy_loader_cli` invocation, which will surface -/// `command not found` to the user — clearer than a silent abort here. -fn find_teensy_loader_cli() -> Option { - let exe_name = - fbuild_core::platform::executable::name("teensy_loader_cli", "teensy_loader_cli.exe"); - - if let Ok(path_env) = std::env::var("PATH") { - let sep = fbuild_core::platform::host::path_list_separator(); - for dir in path_env.split(sep) { - let candidate = PathBuf::from(dir).join(exe_name); - if candidate.is_file() { - return Some(candidate); - } - } - } - - // PlatformIO drops the binary here on every platform. Reusing it means a - // user who already has PIO working doesn't need to install anything else - // to deploy via fbuild. - let pio_root = if fbuild_core::platform::host::is_windows() { - std::env::var("USERPROFILE").ok() - } else { - std::env::var("HOME").ok() - }; - if let Some(home) = pio_root { - let pio_candidate = PathBuf::from(home) - .join(".platformio") - .join("packages") - .join("tool-teensy") - .join(exe_name); - if pio_candidate.is_file() { - return Some(pio_candidate); - } - } - - None -} - /// POST /api/deploy #[expect( clippy::cognitive_complexity, @@ -941,7 +896,8 @@ pub async fn deploy( Some(deploy_project.as_path()), ); let loader_params = fbuild_deploy::teensy::TeensyLoaderParams::default(); - let loader_path = find_teensy_loader_cli(); + let loader_path = find_teensy_loader_cli(deploy_caller_path.as_deref()) + .map(fbuild_core::path::NormalizedPath::into_path_buf); let deployer = fbuild_deploy::teensy::TeensyDeployer::new( &board_config.board.to_uppercase(), &loader_params, diff --git a/crates/fbuild-daemon/src/handlers/operations/deploy_teensy.rs b/crates/fbuild-daemon/src/handlers/operations/deploy_teensy.rs new file mode 100644 index 000000000..782c1d1fa --- /dev/null +++ b/crates/fbuild-daemon/src/handlers/operations/deploy_teensy.rs @@ -0,0 +1,133 @@ +//! `teensy_loader_cli` discovery for the Teensy deploy arm. + +use fbuild_core::path::NormalizedPath; +use fbuild_core::platform::executable::{find_tool_in, find_tool_on_paths}; +use std::ffi::OsStr; +use std::path::Path; + +const STEM: &str = "teensy_loader_cli"; + +/// Resolve a usable `teensy_loader_cli` binary for the Teensy deploy arm. +/// +/// Search order: +/// 1. the requesting CLI's forwarded `PATH` (`caller_path`, +/// FastLED/fbuild#1234) — the daemon's own environment can be stale or +/// minimal, so a tool the user installed in their shell must win; +/// 2. the daemon's `$PATH`; +/// 3. `~/.platformio/packages/tool-teensy/` — the well-known path +/// PlatformIO installs it at on every PIO-using machine. +/// +/// Each directory accepts `teensy_loader_cli` (Unix) / `teensy_loader_cli.exe` +/// (Windows), or a `.com`/`.exe` APE on any host. +/// +/// Returns `None` if nothing is found; the TeensyDeployer's default will then +/// try a bare `teensy_loader_cli` invocation, which will surface +/// `command not found` to the user — clearer than a silent abort here. +pub(super) fn find_teensy_loader_cli(caller_path: Option<&str>) -> Option { + let home = if fbuild_core::platform::host::is_windows() { + std::env::var_os("USERPROFILE") + } else { + std::env::var_os("HOME") + }; + find_teensy_loader_cli_in( + caller_path.map(OsStr::new), + std::env::var_os("PATH").as_deref(), + home.as_deref().map(Path::new), + ) +} + +/// [`find_teensy_loader_cli`] with every environment input passed explicitly. +fn find_teensy_loader_cli_in( + caller_path: Option<&OsStr>, + daemon_path: Option<&OsStr>, + home: Option<&Path>, +) -> Option { + for path_env in [caller_path, daemon_path].into_iter().flatten() { + if let Some(found) = find_tool_on_paths(std::env::split_paths(path_env), STEM) { + return Some(found); + } + } + + // PlatformIO drops the binary here on every platform. Reusing it means a + // user who already has PIO working doesn't need to install anything else + // to deploy via fbuild. + let pio_tool_dir = home? + .join(".platformio") + .join("packages") + .join("tool-teensy"); + find_tool_in(&pio_tool_dir, STEM) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::ffi::OsString; + + async fn write_ape(path: &Path) { + fbuild_core::fs::write(path, b"MZqFpD='\n#!/bin/sh\n") + .await + .unwrap(); + } + + fn join(dirs: &[&Path]) -> OsString { + std::env::join_paths(dirs).unwrap() + } + + /// CodeRabbit on FastLED/fbuild#1633: a `teensy_loader_cli.com` APE that + /// exists only on the requesting CLI's PATH must be found even though the + /// daemon's own PATH does not contain it. + #[tokio::test] + async fn caller_path_ape_is_found_when_daemon_path_lacks_it() { + let caller = tempfile::TempDir::new().unwrap(); + let daemon = tempfile::TempDir::new().unwrap(); + let tool = caller.path().join("teensy_loader_cli.com"); + write_ape(&tool).await; + + let found = find_teensy_loader_cli_in( + Some(&join(&[caller.path()])), + Some(&join(&[daemon.path()])), + None, + ); + assert_eq!(found, Some(NormalizedPath::from(tool))); + } + + #[tokio::test] + async fn caller_path_wins_over_daemon_path() { + let caller = tempfile::TempDir::new().unwrap(); + let daemon = tempfile::TempDir::new().unwrap(); + let caller_tool = caller.path().join("teensy_loader_cli.com"); + write_ape(&caller_tool).await; + write_ape(&daemon.path().join("teensy_loader_cli.com")).await; + + let found = find_teensy_loader_cli_in( + Some(&join(&[caller.path()])), + Some(&join(&[daemon.path()])), + None, + ); + assert_eq!(found, Some(NormalizedPath::from(caller_tool))); + } + + #[tokio::test] + async fn falls_back_to_daemon_path_then_platformio() { + let caller = tempfile::TempDir::new().unwrap(); + let daemon = tempfile::TempDir::new().unwrap(); + let home = tempfile::TempDir::new().unwrap(); + let daemon_tool = daemon.path().join("teensy_loader_cli.com"); + write_ape(&daemon_tool).await; + + let found = find_teensy_loader_cli_in( + Some(&join(&[caller.path()])), + Some(&join(&[daemon.path()])), + Some(home.path()), + ); + assert_eq!(found, Some(NormalizedPath::from(daemon_tool))); + + let pio_dir = home.path().join(".platformio/packages/tool-teensy"); + fbuild_core::fs::create_dir_all(&pio_dir).await.unwrap(); + let pio_tool = pio_dir.join("teensy_loader_cli.com"); + write_ape(&pio_tool).await; + let found = + find_teensy_loader_cli_in(Some(&join(&[caller.path()])), None, Some(home.path())); + assert_eq!(found, Some(NormalizedPath::from(pio_tool))); + } +} diff --git a/crates/fbuild-daemon/src/handlers/operations/mod.rs b/crates/fbuild-daemon/src/handlers/operations/mod.rs index 2f956bef6..6fe8a5920 100644 --- a/crates/fbuild-daemon/src/handlers/operations/mod.rs +++ b/crates/fbuild-daemon/src/handlers/operations/mod.rs @@ -9,6 +9,7 @@ mod build; mod common; mod deploy; mod deploy_port; +mod deploy_teensy; mod install_deps; mod monitor; mod recovery_request; diff --git a/crates/fbuild-daemon/src/main.rs b/crates/fbuild-daemon/src/main.rs index e744f3a66..18eb0b2d7 100644 --- a/crates/fbuild-daemon/src/main.rs +++ b/crates/fbuild-daemon/src/main.rs @@ -116,6 +116,9 @@ async fn main() { eprintln!("warning: failed to install process containment: {}", e); } + // APE (cosmocc) host tools: keep extracted loaders under fbuild's own cache. + fbuild_core::platform::ape::set_default_cache_root(fbuild_paths::get_cache_root().join("ape")); + let port = args.port.unwrap_or_else(fbuild_paths::get_daemon_port); // Build the broadcast hub before installing the tracing subscriber diff --git a/crates/fbuild-deploy/src/lpc.rs b/crates/fbuild-deploy/src/lpc.rs index f2337ad38..f5e3a0182 100644 --- a/crates/fbuild-deploy/src/lpc.rs +++ b/crates/fbuild-deploy/src/lpc.rs @@ -63,13 +63,8 @@ pub fn find_lpc21isp() -> Option { } } - if let Some(managed) = managed_lpc21isp_path() { - if managed.is_file() { - return Some(managed); - } - } - - None + fbuild_core::platform::executable::find_tool_in(&fbuild_paths::try_get_tools_dir()?, "lpc21isp") + .map(fbuild_core::path::NormalizedPath::into_path_buf) } /// Build the "install lpc21isp" hint that surfaces on the failing diff --git a/crates/fbuild-deploy/src/lpc_debugger_reflash.rs b/crates/fbuild-deploy/src/lpc_debugger_reflash.rs index 05bed7bd3..038a4e1e0 100644 --- a/crates/fbuild-deploy/src/lpc_debugger_reflash.rs +++ b/crates/fbuild-deploy/src/lpc_debugger_reflash.rs @@ -105,12 +105,7 @@ fn find_dfu_util_with_override(env_override: Option) -> Opti } } let tools = managed_tools_dir()?; - let exe = fbuild_core::platform::executable::native_name("dfu-util"); - let candidate = tools.join(exe); - if candidate.is_file() { - return Some(candidate); - } - None + fbuild_core::platform::executable::find_tool_in(&tools, "dfu-util") } /// Resolve the CMSIS-DAP hex to flash. Precedence: diff --git a/crates/fbuild-deploy/src/probe_rs.rs b/crates/fbuild-deploy/src/probe_rs.rs index fa4623e0c..8421df51c 100644 --- a/crates/fbuild-deploy/src/probe_rs.rs +++ b/crates/fbuild-deploy/src/probe_rs.rs @@ -133,7 +133,7 @@ pub fn managed_probe_rs_path() -> Option { /// Search order (first hit wins): /// /// 1. `FBUILD_PROBE_RS_PATH` — explicit override. -/// 2. `~/.fbuild/{prod|dev}/tools/probe-rs/probe-rs[.exe]` — the +/// 2. `~/.fbuild/{prod|dev}/tools/probe-rs/probe-rs[.exe|.com]` — the /// canonical fbuild-managed location populated from the pinned /// FastLED/probe-rs release. /// @@ -149,11 +149,7 @@ pub fn find_probe_rs() -> Option { } } - let managed = managed_probe_rs_path()?; - if managed.is_file() { - return Some(managed); - } - None + fbuild_core::platform::executable::find_tool_in(&managed_probe_rs_dir()?, "probe-rs") } /// Resolve and, if needed, install the pinned FastLED/probe-rs binary. @@ -292,8 +288,8 @@ fn probe_rs_temp_install_path(dest_path: &Path) -> NormalizedPath { } fn find_extracted_probe_rs_binary(root: &Path) -> Result { - let exe = fbuild_core::platform::executable::native_name("probe-rs"); - find_file_by_name(root, &exe).ok_or_else(|| { + fbuild_core::platform::executable::find_tool_in_tree(root, "probe-rs").ok_or_else(|| { + let exe = fbuild_core::platform::executable::native_name("probe-rs"); FbuildError::PackageError(format!( "probe-rs binary `{exe}` not found after extracting {}", root.display() @@ -301,27 +297,6 @@ fn find_extracted_probe_rs_binary(root: &Path) -> Result { }) } -fn find_file_by_name(root: &Path, file_name: &str) -> Option { - let entries = std::fs::read_dir(root).ok()?; - for entry in entries.flatten() { - let path = entry.path(); - if path.is_file() - && path - .file_name() - .and_then(|name| name.to_str()) - .is_some_and(|name| name == file_name) - { - return Some(NormalizedPath::from(path)); - } - if path.is_dir() { - if let Some(found) = find_file_by_name(&path, file_name) { - return Some(found); - } - } - } - None -} - /// Map an fbuild `BoardConfig` to the `--chip` name probe-rs expects. /// /// probe-rs uses the exact silicon SKU (e.g. `LPC845M301JBD48`) rather diff --git a/crates/fbuild-deploy/src/wchisp.rs b/crates/fbuild-deploy/src/wchisp.rs index 9c4eb29fc..a83fc7a80 100644 --- a/crates/fbuild-deploy/src/wchisp.rs +++ b/crates/fbuild-deploy/src/wchisp.rs @@ -103,9 +103,11 @@ fn extract_wchisp(archive: &Path, staging: &Path, dest: &Path) -> Result<()> { .unpack(&extract_dir) .map_err(|e| FbuildError::PackageError(format!("extract wchisp archive: {e}")))?; } - let name = fbuild_core::platform::executable::native_name("wchisp"); - let binary = find_file(&extract_dir, &name) - .ok_or_else(|| FbuildError::PackageError(format!("wchisp archive lacks {name}")))?; + let binary = fbuild_core::platform::executable::find_tool_in_tree(&extract_dir, "wchisp") + .ok_or_else(|| { + let name = fbuild_core::platform::executable::native_name("wchisp"); + FbuildError::PackageError(format!("wchisp archive lacks {name}")) + })?; if let Some(parent) = dest.parent() { std::fs::create_dir_all(parent) .map_err(|e| FbuildError::PackageError(format!("create wchisp install dir: {e}")))?; @@ -117,21 +119,6 @@ fn extract_wchisp(archive: &Path, staging: &Path, dest: &Path) -> Result<()> { Ok(()) } -fn find_file(root: &Path, name: &str) -> Option { - for entry in std::fs::read_dir(root).ok()?.flatten() { - let path = entry.path(); - if path.is_file() && path.file_name().and_then(|n| n.to_str()) == Some(name) { - return Some(path); - } - if path.is_dir() { - if let Some(found) = find_file(&path, name) { - return Some(found); - } - } - } - None -} - /// Factory USB-ISP support excludes V003/V006, which have no USB peripheral. pub fn supports_mcu(mcu: &str) -> bool { let upper = mcu.to_ascii_uppercase(); diff --git a/crates/fbuild-deploy/src/wlink.rs b/crates/fbuild-deploy/src/wlink.rs index e17c83cf2..f526d0923 100644 --- a/crates/fbuild-deploy/src/wlink.rs +++ b/crates/fbuild-deploy/src/wlink.rs @@ -101,9 +101,11 @@ fn extract_wlink(archive: &Path, staging: &Path, dest: &Path) -> Result<()> { tar.unpack(&extract_dir) .map_err(|e| FbuildError::PackageError(format!("extract wlink archive: {e}")))?; } - let binary_name = fbuild_core::platform::executable::native_name("wlink"); - let binary = find_file(&extract_dir, &binary_name) - .ok_or_else(|| FbuildError::PackageError(format!("wlink archive lacks {binary_name}")))?; + let binary = fbuild_core::platform::executable::find_tool_in_tree(&extract_dir, "wlink") + .ok_or_else(|| { + let binary_name = fbuild_core::platform::executable::native_name("wlink"); + FbuildError::PackageError(format!("wlink archive lacks {binary_name}")) + })?; if let Some(parent) = dest.parent() { std::fs::create_dir_all(parent) .map_err(|e| FbuildError::PackageError(format!("create wlink install dir: {e}")))?; @@ -115,21 +117,6 @@ fn extract_wlink(archive: &Path, staging: &Path, dest: &Path) -> Result<()> { Ok(()) } -fn find_file(root: &Path, name: &str) -> Option { - for entry in std::fs::read_dir(root).ok()?.flatten() { - let path = entry.path(); - if path.is_file() && path.file_name().and_then(|n| n.to_str()) == Some(name) { - return Some(path); - } - if path.is_dir() { - if let Some(found) = find_file(&path, name) { - return Some(found); - } - } - } - None -} - #[derive(Debug, Clone)] pub struct WlinkDeployer { executable: PathBuf, diff --git a/crates/fbuild-library/src/library/esptool.rs b/crates/fbuild-library/src/library/esptool.rs index e170b5202..43c242946 100644 --- a/crates/fbuild-library/src/library/esptool.rs +++ b/crates/fbuild-library/src/library/esptool.rs @@ -589,7 +589,8 @@ fn host_platform_tag() -> Result<&'static str> { }) } -/// Executable name for the current platform. +/// Native executable name for the current platform (test fixtures). +#[cfg(test)] fn esptool_bin_name() -> &'static str { fbuild_core::platform::executable::name("esptool", "esptool.exe") } @@ -599,9 +600,8 @@ fn esptool_bin_name() -> &'static str { /// `esptool-/esptool`). fn find_esptool_binary(root: &Path) -> Option { fn search(dir: &Path, depth: usize) -> Option { - let candidate = dir.join(esptool_bin_name()); - if candidate.is_file() { - return Some(NormalizedPath::from(candidate)); + if let Some(candidate) = fbuild_core::platform::executable::find_tool_in(dir, "esptool") { + return Some(candidate); } if depth == 0 { return None; diff --git a/crates/fbuild-library/src/library/esptool_tests.rs b/crates/fbuild-library/src/library/esptool_tests.rs index b318e4e4e..37ca5d7a3 100644 --- a/crates/fbuild-library/src/library/esptool_tests.rs +++ b/crates/fbuild-library/src/library/esptool_tests.rs @@ -99,6 +99,16 @@ fn find_binary_nested_one_level() { assert_eq!(found.as_path(), inner.join(esptool_bin_name())); } +#[test] +fn find_binary_accepts_a_nested_ape_com() { + let tmp = tempfile::TempDir::new().unwrap(); + let inner = tmp.path().join("esptool-any"); + std::fs::create_dir_all(&inner).unwrap(); + std::fs::write(inner.join("esptool.com"), b"MZqFpD='\n").unwrap(); + let found = find_esptool_binary(tmp.path()).unwrap(); + assert_eq!(found.as_path(), inner.join("esptool.com")); +} + #[test] fn find_binary_missing_returns_none() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-packages-fetch/src/ape_install_tests.rs b/crates/fbuild-packages-fetch/src/ape_install_tests.rs new file mode 100644 index 000000000..3c1e19a82 --- /dev/null +++ b/crates/fbuild-packages-fetch/src/ape_install_tests.rs @@ -0,0 +1,74 @@ +//! Staged installs restore the exec bit on APE (cosmocc) host tools. + +use super::toolchain_gcc_ar_tests::serve_once; +use super::*; + +/// A zip whose entries carry no exec bits (the `zip` writer default is +/// 0644, like DOS-attribute archives) must still install an APE host tool +/// as executable, while leaving non-APE files exactly as extracted. +#[tokio::test] +async fn staged_install_marks_ape_tools_executable() { + use std::io::{Cursor, Write}; + use zip::write::SimpleFileOptions; + + let mut zip = zip::ZipWriter::new(Cursor::new(Vec::new())); + zip.start_file("pkg/bin/tool.com", SimpleFileOptions::default()) + .unwrap(); + zip.write_all(b"MZqFpD='\n#!/bin/sh\n").unwrap(); + zip.start_file("pkg/bin/readme.txt", SimpleFileOptions::default()) + .unwrap(); + zip.write_all(b"not a tool").unwrap(); + let bytes = zip.finish().unwrap().into_inner(); + + let tmp = tempfile::TempDir::new().unwrap(); + // Plain extraction (not only staged installs) restores the APE exec + // bit too, and leaves other 0644 entries alone. + let archive = tmp.path().join("ref.zip"); + std::fs::write(&archive, &bytes).unwrap(); + let reference = tmp.path().join("ref"); + std::fs::create_dir_all(&reference).unwrap(); + extractor::extract(&archive, &reference).unwrap(); + if !fbuild_core::platform::host::current().is_windows() { + let ref_ape = std::fs::metadata(reference.join("pkg/bin/tool.com")).unwrap(); + assert!(fbuild_core::platform::fs::is_executable(&ref_ape)); + let ref_txt = std::fs::metadata(reference.join("pkg/bin/readme.txt")).unwrap(); + assert!(!fbuild_core::platform::fs::is_executable(&ref_txt)); + } + + let url = serve_once(bytes).await; + let base = PackageBase::with_cache_root( + "ape-tool", + "1.0", + &url, + "ape-tool", + None, + CacheSubdir::Toolchains, + tmp.path(), + &tmp.path().join("cache"), + ); + let installed = base + .staged_install(|staging| { + let meta = std::fs::metadata(staging.join("pkg/bin/tool.com")).unwrap(); + assert!( + fbuild_core::platform::fs::is_executable(&meta), + "APE must be executable before validation runs" + ); + Ok(()) + }) + .await + .expect("staged install should succeed"); + + assert!( + ape_perms::is_repaired(&installed), + "a fresh install records its APE scan so it is never repaired again" + ); + let ape = std::fs::metadata(installed.join("pkg/bin/tool.com")).unwrap(); + assert!(fbuild_core::platform::fs::is_executable(&ape)); + let txt = std::fs::metadata(installed.join("pkg/bin/readme.txt")).unwrap(); + let ref_txt = std::fs::metadata(reference.join("pkg/bin/readme.txt")).unwrap(); + assert_eq!( + txt.permissions(), + ref_txt.permissions(), + "non-APE files keep their extracted permissions" + ); +} diff --git a/crates/fbuild-packages-fetch/src/ape_perms.rs b/crates/fbuild-packages-fetch/src/ape_perms.rs new file mode 100644 index 000000000..49d9cc613 --- /dev/null +++ b/crates/fbuild-packages-fetch/src/ape_perms.rs @@ -0,0 +1,152 @@ +//! Once-per-install exec-bit repair for APE (cosmocc) host tools. +//! +//! Zip entries without Unix attributes extract as 0644. Fresh installs mark +//! APE tools executable during [`crate::PackageBase::staged_install`], but an +//! install extracted by an older fbuild can still hold a 0644 `tool.com`. +//! A marker file next to the `.install_complete` sentinel records that the +//! tree has been scanned, so every install pays for at most one scan, ever. + +use crate::extractor; +use std::path::Path; + +/// Marker written into an install dir once its APE tools are known executable. +/// Bump the suffix to force one more scan of every existing install. +const APE_PERMS_MARKER: &str = ".ape_perms_v1"; + +/// Whether `installed_dir` has already been scanned for APE exec bits. +pub(crate) fn is_repaired(installed_dir: &Path) -> bool { + installed_dir.join(APE_PERMS_MARKER).exists() +} + +/// Mark APE tools executable in a freshly extracted tree and record the scan, +/// so the committed install never needs a repair pass. +pub(crate) fn mark_fresh(dir: &Path) -> fbuild_core::Result<()> { + extractor::mark_ape_executables(dir)?; + write_marker(dir); + Ok(()) +} + +/// Repair an existing install's APE exec bits unless it was already scanned. +/// Best-effort: a failed scan is logged and retried on the next install hit. +/// Callers hold the package's install lock. +pub(crate) fn repair_once(installed_dir: &Path) { + if is_repaired(installed_dir) { + return; + } + match extractor::mark_ape_executables(installed_dir) { + Ok(marked) => { + if marked > 0 { + tracing::info!( + "marked {marked} APE tool(s) executable in {}", + installed_dir.display() + ); + } + write_marker(installed_dir); + } + Err(e) => tracing::warn!( + "failed to repair APE permissions in {}: {e}", + installed_dir.display() + ), + } +} + +fn write_marker(dir: &Path) { + if let Err(e) = std::fs::write(dir.join(APE_PERMS_MARKER), b"") { + tracing::warn!("failed to write APE permission marker: {e}"); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{CacheSubdir, PackageBase}; + + const APE_BYTES: &[u8] = b"MZqFpD='\n#!/bin/sh\n"; + + fn is_executable(path: &Path) -> bool { + fbuild_core::platform::fs::is_executable(&std::fs::metadata(path).unwrap()) + } + + /// Recreate the file so it carries no exec bit (fresh files never do). + fn clear_exec_bits(path: &Path) { + let bytes = std::fs::read(path).unwrap(); + std::fs::remove_file(path).unwrap(); + std::fs::write(path, bytes).unwrap(); + } + + #[test] + fn mark_fresh_writes_marker() { + let tmp = tempfile::TempDir::new().unwrap(); + std::fs::write(tmp.path().join("tool.com"), APE_BYTES).unwrap(); + mark_fresh(tmp.path()).unwrap(); + assert!(is_repaired(tmp.path())); + assert!(is_executable(&tmp.path().join("tool.com"))); + } + + /// The repair runs once: after the marker is written, a 0644 APE is no + /// longer rescanned (it stays non-executable), proving the scan is not + /// repeated on every install hit. + #[test] + fn repair_once_scans_only_once() { + if fbuild_core::platform::host::current().is_windows() { + return; + } + let tmp = tempfile::TempDir::new().unwrap(); + let tool = tmp.path().join("bin").join("tool.com"); + std::fs::create_dir_all(tool.parent().unwrap()).unwrap(); + std::fs::write(&tool, APE_BYTES).unwrap(); + clear_exec_bits(&tool); + + repair_once(tmp.path()); + assert!( + is_executable(&tool), + "first repair marks the APE executable" + ); + assert!(is_repaired(tmp.path())); + + clear_exec_bits(&tool); + repair_once(tmp.path()); + assert!(!is_executable(&tool), "a marked install is never rescanned"); + } + + /// CodeRabbit on FastLED/fbuild#1633: an install extracted before APE + /// support (0644 `tool.com`, sentinel present, no marker) is repaired by + /// the next `staged_install` hit, which never re-downloads. + #[tokio::test] + async fn staged_install_repairs_legacy_install_once() { + if fbuild_core::platform::host::current().is_windows() { + return; + } + let tmp = tempfile::TempDir::new().unwrap(); + let base = PackageBase::with_cache_root( + "legacy-ape", + "1.0", + "http://127.0.0.1:9/unreachable.zip", + "legacy-ape", + None, + CacheSubdir::Toolchains, + tmp.path(), + &tmp.path().join("cache"), + ); + let install_path = base.install_path(); + let tool = install_path.join("bin").join("tool.com"); + std::fs::create_dir_all(tool.parent().unwrap()).unwrap(); + std::fs::write(&tool, APE_BYTES).unwrap(); + clear_exec_bits(&tool); + std::fs::write( + crate::disk_cache::paths::install_complete_sentinel(&install_path), + b"", + ) + .unwrap(); + + let installed = base.staged_install(|_| Ok(())).await.unwrap(); + assert_eq!(installed, install_path); + assert!(is_executable(&tool), "legacy 0644 APE must be repaired"); + assert!(is_repaired(&install_path)); + + // Already-marked installs take the lock-free fast path untouched. + clear_exec_bits(&tool); + base.staged_install(|_| Ok(())).await.unwrap(); + assert!(!is_executable(&tool), "marked install is not rescanned"); + } +} diff --git a/crates/fbuild-packages-fetch/src/extractor.rs b/crates/fbuild-packages-fetch/src/extractor.rs index 23a11844f..fb71b563c 100644 --- a/crates/fbuild-packages-fetch/src/extractor.rs +++ b/crates/fbuild-packages-fetch/src/extractor.rs @@ -64,6 +64,43 @@ pub fn extract(archive_path: &Path, dest_dir: &Path) -> Result<()> { } } +/// Give every APE (Actually Portable Executable) host tool under `root` an +/// exec bit, returning how many files were inspected and found to be APE. +/// +/// Zip archives built without Unix attributes (DOS-attribute or no-attribute +/// entries) extract with mode 0644, so a package shipping one `tool.com` APE +/// for every host would be unrunnable on Linux/macOS. Only regular files whose +/// first bytes carry an APE magic are touched (an 8-byte read each), symlinks +/// are never followed, and the call is a no-op on Windows. +pub fn mark_ape_executables(root: &Path) -> Result { + let mut marked = 0; + let mut pending = vec![root.to_path_buf()]; + while let Some(dir) = pending.pop() { + let entries = std::fs::read_dir(&dir).map_err(|e| { + FbuildError::PackageError(format!("failed to read {}: {}", dir.display(), e)) + })?; + for entry in entries { + let entry = entry?; + // `DirEntry::file_type` does not follow symlinks. + let file_type = entry.file_type()?; + let path = entry.path(); + if file_type.is_dir() { + pending.push(path); + } else if file_type.is_file() && fbuild_core::platform::ape::is_ape_file(&path) { + fbuild_core::platform::fs::ensure_executable(&path).map_err(|e| { + FbuildError::PackageError(format!( + "failed to mark APE {} executable: {}", + path.display(), + e + )) + })?; + marked += 1; + } + } + } + Ok(marked) +} + /// Extract a `.zip` archive. Public so callers (e.g. lnk materializer) /// can dispatch without depending on the source file's extension. pub fn extract_zip_public(archive_path: &Path, dest_dir: &Path) -> Result<()> { @@ -160,6 +197,9 @@ fn extract_zip(archive_path: &Path, dest_dir: &Path) -> Result<()> { )) })?; validate_extraction(dest_dir)?; + // Zip entries without Unix attributes extract without the exec bit; every + // zip caller (not just staged package installs) needs APE tools runnable. + mark_ape_executables(dest_dir)?; Ok(()) } @@ -296,6 +336,48 @@ mod tests { assert_eq!(bytes, b"hello from tar.xz"); } + #[test] + fn mark_ape_executables_marks_only_apes_and_does_not_follow_symlinks() { + use fbuild_core::platform::fs::is_executable; + + let tmp = tempfile::TempDir::new().unwrap(); + let archive = tmp.path().join("tools.zip"); + make_zip(&archive, "bin/tool.com", b"MZqFpD='\n"); + let dest = tmp.path().join("out"); + std::fs::create_dir_all(&dest).unwrap(); + extract(&archive, &dest).unwrap(); + std::fs::write(dest.join("bin/notes.txt"), b"plain").unwrap(); + let notes_before = std::fs::metadata(dest.join("bin/notes.txt")) + .unwrap() + .permissions(); + + // An APE outside the tree, reachable only through a symlink. + let outside = tmp.path().join("outside.com"); + std::fs::write(&outside, b"MZqFpD='\n").unwrap(); + let outside_before = std::fs::metadata(&outside).unwrap().permissions(); + let linked = + fbuild_core::platform::fs::symlink_file(&outside, &dest.join("bin/link.com")).is_ok(); + + let marked = mark_ape_executables(&dest).unwrap(); + assert_eq!(marked, 1, "only the in-tree regular APE is marked"); + assert!(is_executable( + &std::fs::metadata(dest.join("bin/tool.com")).unwrap() + )); + assert_eq!( + std::fs::metadata(dest.join("bin/notes.txt")) + .unwrap() + .permissions(), + notes_before + ); + if linked { + assert_eq!( + std::fs::metadata(&outside).unwrap().permissions(), + outside_before, + "symlink targets outside the tree must not be modified" + ); + } + } + #[test] fn test_extract_zip() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-packages-fetch/src/lib.rs b/crates/fbuild-packages-fetch/src/lib.rs index 238c48fa8..0e94198b7 100644 --- a/crates/fbuild-packages-fetch/src/lib.rs +++ b/crates/fbuild-packages-fetch/src/lib.rs @@ -15,6 +15,7 @@ pub mod platformio_registry; pub mod platformio_repository; pub mod submodules; +mod ape_perms; mod install_lock; pub use cache::Cache; @@ -364,7 +365,7 @@ impl PackageBase { { let install_path = self.install_path(); - if install_path.exists() { + if install_path.exists() && ape_perms::is_repaired(&install_path) { // An existing-install hit is a use: refresh the DiskCache LRU so // background GC doesn't treat this package as the oldest unused // entry and delete it mid-build (FastLED/fbuild#1341). @@ -383,6 +384,7 @@ impl PackageBase { let _install_lock = install_lock::acquire_for_install(&install_path, &self.name, &self.version).await?; if install_path.exists() { + ape_perms::repair_once(&install_path); self.touch_disk_cache(); return Ok(install_path); } @@ -459,6 +461,10 @@ impl PackageBase { submodules::prepare_submodules(&self.name, &self.url, &staging_path, &self.submodules) .await?; + // Zip entries without Unix attributes extract as 0644; restore the + // exec bit on APE host tools so one `tool.com` runs on every host. + ape_perms::mark_fresh(&staging_path)?; + // Validate validate(&staging_path)?; @@ -541,11 +547,14 @@ fn clear_package_touch_cache_for_tests() { } } +#[cfg(test)] +mod ape_install_tests; + #[cfg(test)] mod toolchain_gcc_ar_tests { use super::*; - async fn serve_once(body: Vec) -> String { + pub(super) async fn serve_once(body: Vec) -> String { use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::TcpListener; diff --git a/crates/fbuild-python/src/daemon.rs b/crates/fbuild-python/src/daemon.rs index 295d3bdd8..f5d5baec8 100644 --- a/crates/fbuild-python/src/daemon.rs +++ b/crates/fbuild-python/src/daemon.rs @@ -255,9 +255,9 @@ async fn ensure_running_async_impl( // stale user-level daemon on PATH. let mut cmd = match spawn_target { // allow-direct-spawn: construction only; execution uses platform::process::spawn_detached. - Some(path) => std::process::Command::new(path), + Some(path) => fbuild_core::platform::process::command(path), // allow-direct-spawn: construction only; execution uses platform::process::spawn_detached. - None => std::process::Command::new(DAEMON_BIN_NAME), + None => fbuild_core::platform::process::command(DAEMON_BIN_NAME), }; if dev_mode { cmd.arg("--dev"); diff --git a/crates/fbuild-serial/src/crash_decoder.rs b/crates/fbuild-serial/src/crash_decoder.rs index 3b7ab5f09..4cc8b1ea5 100644 --- a/crates/fbuild-serial/src/crash_decoder.rs +++ b/crates/fbuild-serial/src/crash_decoder.rs @@ -381,25 +381,31 @@ impl CrashDecoder { /// The toolchain prefix is everything before `gcc` in the binary name: /// - `riscv32-esp-elf-gcc` → `riscv32-esp-elf-addr2line` /// - `xtensa-esp32s3-elf-gcc` → `xtensa-esp32s3-elf-addr2line` +/// +/// The compiler's executable suffix (`.exe`, or `.com` for a cosmocc APE) is +/// stripped for matching and tried first on the derived tool, so an APE +/// toolchain's `xtensa-esp32s3-elf-gcc.com` maps to +/// `xtensa-esp32s3-elf-addr2line.com`. That sibling obeys `find_tool_in`'s +/// acceptance rule: a non-native spelling counts only if it is an APE, so a +/// Windows-only `addr2line.exe` never shadows a runnable `addr2line.com` on +/// Unix. Otherwise the usual native/APE candidates are probed. pub fn derive_addr2line_path(cc_path: &Path) -> Option { - let name = cc_path.file_name()?.to_string_lossy(); - - // Strip .exe suffix for matching - let stem = name.replace(".exe", ""); - if !stem.ends_with("gcc") { - return None; - } + use fbuild_core::platform::ape::is_ape_file; + use fbuild_core::platform::executable::{find_tool_in, native_name, split_tool_suffix}; - let prefix = &stem[..stem.len() - 3]; // e.g. "riscv32-esp-elf-" - let addr2line_name = - fbuild_core::platform::executable::native_name(&format!("{prefix}addr2line")); - let addr2line = cc_path.parent()?.join(addr2line_name); - - if addr2line.exists() { - Some(addr2line) - } else { - None - } + let name = cc_path.file_name()?.to_string_lossy(); + let (stem, suffix) = split_tool_suffix(&name); + let prefix = stem.strip_suffix("gcc")?; // e.g. "riscv32-esp-elf-" + let bin_dir = cc_path.parent()?; + let tool_stem = format!("{prefix}addr2line"); + + let same_name = format!("{tool_stem}{suffix}"); + let same_suffix = bin_dir.join(&same_name); + if same_suffix.is_file() && (same_name == native_name(&tool_stem) || is_ape_file(&same_suffix)) + { + return Some(same_suffix); + } + find_tool_in(bin_dir, &tool_stem).map(fbuild_core::path::NormalizedPath::into_path_buf) } #[cfg(test)] @@ -636,6 +642,47 @@ mod tests { assert!(derive_addr2line_path(¬_gcc).is_none()); } + #[test] + fn derive_addr2line_keeps_the_com_suffix_of_an_ape_compiler() { + let dir = tempfile::tempdir().unwrap(); + let gcc = dir.path().join("xtensa-esp32s3-elf-gcc.com"); + let addr2line = dir.path().join("xtensa-esp32s3-elf-addr2line.com"); + std::fs::write(&gcc, b"MZqFpD='\n").unwrap(); + assert!(derive_addr2line_path(&gcc).is_none()); + std::fs::write(&addr2line, b"MZqFpD='\n").unwrap(); + assert_eq!(derive_addr2line_path(&gcc), Some(addr2line)); + } + + #[test] + fn derive_addr2line_from_exe_compiler_keeps_exe() { + // An APE `.exe` runs on every host, so the compiler's suffix is kept. + let dir = tempfile::tempdir().unwrap(); + let gcc = dir.path().join("riscv32-esp-elf-gcc.exe"); + let addr2line = dir.path().join("riscv32-esp-elf-addr2line.exe"); + std::fs::write(&addr2line, b"MZqFpD='\n").unwrap(); + assert_eq!(derive_addr2line_path(&gcc), Some(addr2line)); + } + + /// CodeRabbit on FastLED/fbuild#1633: the compiler's own non-native + /// suffix is accepted only for an APE. A Windows-only PE `addr2line.exe` + /// next to an APE `addr2line.com` must lose to the runnable `.com` on + /// Unix (on Windows the native `.exe` legitimately wins). + #[test] + fn derive_addr2line_skips_non_ape_foreign_suffix() { + let dir = tempfile::tempdir().unwrap(); + let gcc = dir.path().join("riscv32-esp-elf-gcc.exe"); + let pe = dir.path().join("riscv32-esp-elf-addr2line.exe"); + let ape = dir.path().join("riscv32-esp-elf-addr2line.com"); + std::fs::write(&pe, b"MZ\x90\x00").unwrap(); + std::fs::write(&ape, b"MZqFpD='\n").unwrap(); + let expected = if fbuild_core::platform::host::is_windows() { + pe + } else { + ape + }; + assert_eq!(derive_addr2line_path(&gcc), Some(expected)); + } + // --- Regex pattern tests --- #[test] diff --git a/crates/fbuild-toolchain/src/toolchain/arm.rs b/crates/fbuild-toolchain/src/toolchain/arm.rs index d788ea7e4..478241f24 100644 --- a/crates/fbuild-toolchain/src/toolchain/arm.rs +++ b/crates/fbuild-toolchain/src/toolchain/arm.rs @@ -119,9 +119,7 @@ impl crate::Package for ArmToolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin") - .join(tool_name("arm-none-eabi-gcc")) - .exists() + tool_binary(&root.join("bin"), "arm-none-eabi-gcc").exists() } fn get_info(&self) -> PackageInfo { @@ -247,14 +245,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -/// Get the tool binary name with .exe extension on Windows. -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - -/// Get the full path to a tool binary. +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] @@ -270,15 +263,6 @@ mod tests { assert!(url.contains("arm-none-eabi")); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("arm-none-eabi-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("arm-none-eabi-gcc") - ); - } - #[test] fn test_find_bin_root_direct() { let tmp = tempfile::TempDir::new().unwrap(); @@ -313,6 +297,31 @@ mod tests { assert!(!tc.is_installed()); } + /// A package shipping each host tool as one APE `bin/.com` validates, + /// reports installed, and resolves every tool to the `.com` file. + #[test] + fn test_arm_toolchain_finds_ape_com_tools() { + let tmp = tempfile::TempDir::new().unwrap(); + let tc = ArmToolchain::with_cache_root(tmp.path(), &tmp.path().join("cache")); + let install = tc.base.install_path(); + let bin = install.join("bin"); + std::fs::create_dir_all(&bin).unwrap(); + let tools = ["gcc", "g++", "ar", "objcopy", "size"]; + assert!(ArmToolchain::validate(&install).is_err()); + for tool in tools { + std::fs::write(bin.join(format!("arm-none-eabi-{tool}.com")), b"MZqFpD='\n").unwrap(); + } + ArmToolchain::validate(&install).expect("APE .com tools satisfy validation"); + std::fs::write( + crate::disk_cache::paths::install_complete_sentinel(&install), + "", + ) + .unwrap(); + assert!(tc.is_installed()); + assert_eq!(tc.get_gcc_path(), bin.join("arm-none-eabi-gcc.com")); + assert_eq!(tc.get_size_path(), bin.join("arm-none-eabi-size.com")); + } + /// Checksums that are present must be valid 64-char lowercase hex (SHA-256). /// Catches placeholder hashes like `a000...0002` on all platforms. #[test] diff --git a/crates/fbuild-toolchain/src/toolchain/arm_gcc8.rs b/crates/fbuild-toolchain/src/toolchain/arm_gcc8.rs index 7c820b597..644ace99d 100644 --- a/crates/fbuild-toolchain/src/toolchain/arm_gcc8.rs +++ b/crates/fbuild-toolchain/src/toolchain/arm_gcc8.rs @@ -88,9 +88,7 @@ impl crate::Package for ArmGcc8Toolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin") - .join(tool_name("arm-none-eabi-gcc")) - .exists() + tool_binary(&root.join("bin"), "arm-none-eabi-gcc").exists() } fn get_info(&self) -> PackageInfo { @@ -165,12 +163,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] diff --git a/crates/fbuild-toolchain/src/toolchain/avr.rs b/crates/fbuild-toolchain/src/toolchain/avr.rs index 5afcf2215..bdf2867e5 100644 --- a/crates/fbuild-toolchain/src/toolchain/avr.rs +++ b/crates/fbuild-toolchain/src/toolchain/avr.rs @@ -126,7 +126,7 @@ impl crate::Package for AvrToolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin").join(tool_name("avr-gcc")).exists() + tool_binary(&root.join("bin"), "avr-gcc").exists() } fn get_info(&self) -> PackageInfo { @@ -255,14 +255,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -/// Get the tool binary name with .exe extension on Windows. -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - -/// Get the full path to a tool binary. +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] @@ -382,15 +377,6 @@ mod tests { ); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("avr-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("avr-gcc") - ); - } - #[test] fn test_find_bin_root_direct() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-toolchain/src/toolchain/clang.rs b/crates/fbuild-toolchain/src/toolchain/clang.rs index a86b0b805..da8068b2b 100644 --- a/crates/fbuild-toolchain/src/toolchain/clang.rs +++ b/crates/fbuild-toolchain/src/toolchain/clang.rs @@ -106,8 +106,7 @@ impl ClangComponent { /// Calls `ensure_installed()` internally. pub async fn get_binary(&self, name: &str) -> fbuild_core::Result { let install_dir = self.ensure_installed().await?; - let binary_name = fbuild_core::platform::executable::native_name(name); - find_binary_in_dir(&install_dir, &binary_name).ok_or_else(|| { + find_binary_in_dir(&install_dir, name).ok_or_else(|| { fbuild_core::FbuildError::Other(format!( "'{}' not found in {} installation at {}", name, @@ -240,8 +239,7 @@ impl ClangComponent { fn validate(kind: ClangComponentKind, dir: &Path) -> fbuild_core::Result<()> { for name in kind.required_binaries() { - let binary_name = fbuild_core::platform::executable::native_name(name); - if find_binary_in_dir(dir, &binary_name).is_none() { + if find_binary_in_dir(dir, name).is_none() { return Err(fbuild_core::FbuildError::PackageError(format!( "'{}' not found in extracted {} archive", name, @@ -253,25 +251,25 @@ impl ClangComponent { } } -/// Recursively search for a binary by name in a directory. -/// Checks `dir/bin/name`, then `dir/*/bin/name` (one level of nesting). -pub fn find_binary_in_dir(dir: &Path, name: &str) -> Option { +/// Search for a tool by stem in a directory: the host-native spelling or a +/// `.com`/`.exe` APE (see [`fbuild_core::platform::executable::find_tool_in`]). +/// Checks `dir/bin/`, then `dir/*/bin/` (one level of nesting). +pub fn find_binary_in_dir(dir: &Path, stem: &str) -> Option { + use fbuild_core::platform::executable::find_tool_in; if !dir.exists() { return None; } - // Direct: dir/bin/name - let direct = dir.join("bin").join(name); - if direct.exists() { - return Some(direct); + // Direct: dir/bin/ + if let Some(direct) = find_tool_in(&dir.join("bin"), stem) { + return Some(direct.into_path_buf()); } // One level nested: dir/subdir/bin/name (archives often have a top-level folder) if let Ok(entries) = std::fs::read_dir(dir) { for entry in entries.filter_map(|e| e.ok()) { let p = entry.path(); if p.is_dir() { - let candidate = p.join("bin").join(name); - if candidate.exists() { - return Some(candidate); + if let Some(candidate) = find_tool_in(&p.join("bin"), stem) { + return Some(candidate.into_path_buf()); } } } @@ -427,7 +425,26 @@ mod tests { std::fs::create_dir_all(&nested).unwrap(); let name = fbuild_core::platform::executable::native_name("clang-tidy"); std::fs::write(nested.join(&name), b"fake").unwrap(); - assert!(find_binary_in_dir(dir.path(), &name).is_some()); + assert!(find_binary_in_dir(dir.path(), "clang-tidy").is_some()); + } + + #[test] + fn test_find_binary_accepts_ape_com_but_not_foreign_native_binary() { + let dir = tempfile::tempdir().unwrap(); + let bin = dir.path().join("llvm").join("bin"); + std::fs::create_dir_all(&bin).unwrap(); + let foreign = if fbuild_core::platform::host::current().is_windows() { + "clang-tidy" + } else { + "clang-tidy.exe" + }; + std::fs::write(bin.join(foreign), b"not an APE").unwrap(); + assert!(find_binary_in_dir(dir.path(), "clang-tidy").is_none()); + std::fs::write(bin.join("clang-tidy.com"), b"MZqFpD='\n").unwrap(); + assert_eq!( + find_binary_in_dir(dir.path(), "clang-tidy"), + Some(bin.join("clang-tidy.com")) + ); } #[test] diff --git a/crates/fbuild-toolchain/src/toolchain/esp32.rs b/crates/fbuild-toolchain/src/toolchain/esp32.rs index 57875e3ec..dab916aca 100644 --- a/crates/fbuild-toolchain/src/toolchain/esp32.rs +++ b/crates/fbuild-toolchain/src/toolchain/esp32.rs @@ -204,9 +204,7 @@ impl crate::Package for Esp32Toolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin") - .join(tool_name(&format!("{}gcc", self.prefix))) - .exists() + tool_binary(&root.join("bin"), &format!("{}gcc", self.prefix)).exists() } fn get_info(&self) -> PackageInfo { @@ -392,14 +390,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -/// Get the tool binary name with .exe extension on Windows. -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - -/// Get the full path to a tool binary. +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] @@ -421,15 +414,6 @@ mod tests { assert!(url.contains("espressif")); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("riscv32-esp-elf-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("riscv32-esp-elf-gcc") - ); - } - #[test] fn test_find_bin_root_direct() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-toolchain/src/toolchain/esp32_metadata.rs b/crates/fbuild-toolchain/src/toolchain/esp32_metadata.rs index 9de575bb3..2547b4e86 100644 --- a/crates/fbuild-toolchain/src/toolchain/esp32_metadata.rs +++ b/crates/fbuild-toolchain/src/toolchain/esp32_metadata.rs @@ -183,16 +183,21 @@ fn parse_tools_json(path: &Path, toolchain_name: &str) -> Result = version - .as_object() - .map(|m| m.keys().map(|k| k.as_str()).collect()) - .unwrap_or_default(); - FbuildError::PackageError(format!( - "platform '{}' not found for {}. Available: {:?}", - platform, toolchain_name, available - )) - })?; + // ESP-IDF marks host-independent tools (scripts, APE builds) with the + // `any` platform key; use it when the host has no dedicated entry. + let platform_info = version + .get(platform) + .or_else(|| version.get("any")) + .ok_or_else(|| { + let available: Vec<&str> = version + .as_object() + .map(|m| m.keys().map(|k| k.as_str()).collect()) + .unwrap_or_default(); + FbuildError::PackageError(format!( + "platform '{}' not found for {}. Available: {:?}", + platform, toolchain_name, available + )) + })?; let url = platform_info .get("url") @@ -281,6 +286,38 @@ mod tests { assert!(result.is_err()); } + #[test] + fn test_parse_tools_json_falls_back_to_any_platform() { + let tmp = tempfile::TempDir::new().unwrap(); + let tools_json = tmp.path().join("tools.json"); + std::fs::write( + &tools_json, + r#"{"tools": [{"name": "tc", "versions": [{ + "other-platform": {"url": "https://example.com/other.zip"}, + "any": {"url": "https://example.com/any.zip", "sha256": "beef"} + }]}]}"#, + ) + .unwrap(); + let result = parse_tools_json(&tools_json, "tc").unwrap(); + assert_eq!(result.url, "https://example.com/any.zip"); + assert_eq!(result.sha256.as_deref(), Some("beef")); + + // The host's own entry still wins over `any`. + std::fs::write( + &tools_json, + format!( + r#"{{"tools": [{{"name": "tc", "versions": [{{ + "any": {{"url": "https://example.com/any.zip"}}, + "{}": {{"url": "https://example.com/host.zip"}} + }}]}}]}}"#, + detect_platform() + ), + ) + .unwrap(); + let result = parse_tools_json(&tools_json, "tc").unwrap(); + assert_eq!(result.url, "https://example.com/host.zip"); + } + #[test] fn test_parse_tools_json_missing_platform() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-toolchain/src/toolchain/esp8266.rs b/crates/fbuild-toolchain/src/toolchain/esp8266.rs index 77db347a9..4b558874d 100644 --- a/crates/fbuild-toolchain/src/toolchain/esp8266.rs +++ b/crates/fbuild-toolchain/src/toolchain/esp8266.rs @@ -110,9 +110,7 @@ impl crate::Package for Esp8266Toolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin") - .join(tool_name(&format!("{PREFIX}gcc"))) - .exists() + tool_binary(&root.join("bin"), &format!("{PREFIX}gcc")).exists() } fn get_info(&self) -> PackageInfo { @@ -249,12 +247,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] @@ -269,15 +264,6 @@ mod tests { assert!(url.contains("earlephilhower")); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("xtensa-lx106-elf-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("xtensa-lx106-elf-gcc") - ); - } - #[test] fn test_find_bin_root_direct() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-toolchain/src/toolchain/esp_qemu.rs b/crates/fbuild-toolchain/src/toolchain/esp_qemu.rs index 1e957c6bb..449eb1ccb 100644 --- a/crates/fbuild-toolchain/src/toolchain/esp_qemu.rs +++ b/crates/fbuild-toolchain/src/toolchain/esp_qemu.rs @@ -71,6 +71,14 @@ impl EspQemuArch { } } + /// Executable stem, resolved via `find_tool_in` (native or APE spelling). + fn stem(self) -> &'static str { + match self { + Self::Xtensa => "qemu-system-xtensa", + Self::Riscv32 => "qemu-system-riscv32", + } + } + fn binary_name(self) -> &'static str { match self { Self::Xtensa => fbuild_core::platform::executable::name( @@ -127,7 +135,7 @@ impl EspQemu { hydrate_windows_runtime(&path)?; validate_windows_runtime(&path)?; path - } else if let Some(path) = find_on_path(self.arch.binary_name()) { + } else if let Some(path) = find_tool_on_path(self.arch.stem()) { hydrate_windows_runtime(&path)?; validate_windows_runtime(&path)?; path @@ -406,15 +414,15 @@ fn validate_qemu_path(path: PathBuf, source: &str) -> Result { } fn find_qemu_binary(root: &Path, arch: EspQemuArch) -> Result { + use fbuild_core::platform::executable::find_tool_in; let file_name = arch.binary_name(); - let direct = root.join(file_name); - if direct.is_file() { - return Ok(direct); + let stem = arch.stem(); + if let Some(direct) = find_tool_in(root, stem) { + return Ok(direct.into_path_buf()); } - let in_bin = root.join("bin").join(file_name); - if in_bin.is_file() { - return Ok(in_bin); + if let Some(in_bin) = find_tool_in(&root.join("bin"), stem) { + return Ok(in_bin.into_path_buf()); } if let Ok(entries) = std::fs::read_dir(root) { @@ -424,14 +432,12 @@ fn find_qemu_binary(root: &Path, arch: EspQemuArch) -> Result { continue; } - let nested_direct = path.join(file_name); - if nested_direct.is_file() { - return Ok(nested_direct); + if let Some(nested_direct) = find_tool_in(&path, stem) { + return Ok(nested_direct.into_path_buf()); } - let nested_bin = path.join("bin").join(file_name); - if nested_bin.is_file() { - return Ok(nested_bin); + if let Some(nested_bin) = find_tool_in(&path.join("bin"), stem) { + return Ok(nested_bin.into_path_buf()); } } } @@ -461,15 +467,20 @@ fn qemu_root(install_dir: &Path, arch: EspQemuArch) -> Result { } } -fn find_on_path(binary: &str) -> Option { +/// A host tool on PATH by stem, in any spelling `find_tool_on_paths` accepts +/// (native, or `.exe`/`.com`/extensionless APE). +fn find_tool_on_path(stem: &str) -> Option { let path_var = std::env::var_os("PATH")?; - for dir in std::env::split_paths(&path_var) { - let candidate = dir.join(binary); - if candidate.is_file() { - return Some(candidate); - } - } - None + fbuild_core::platform::executable::find_tool_on_paths(std::env::split_paths(&path_var), stem) + .map(fbuild_core::path::NormalizedPath::into_path_buf) +} + +/// An exact file name on PATH (e.g. a DLL — never a tool-name expansion). +fn find_on_path(file_name: &str) -> Option { + let path_var = std::env::var_os("PATH")?; + std::env::split_paths(&path_var) + .map(|dir| dir.join(file_name)) + .find(|candidate| candidate.is_file()) } fn find_existing_idf_qemu(arch: EspQemuArch) -> Option { diff --git a/crates/fbuild-toolchain/src/toolchain/riscv.rs b/crates/fbuild-toolchain/src/toolchain/riscv.rs index ab7727571..91f3733df 100644 --- a/crates/fbuild-toolchain/src/toolchain/riscv.rs +++ b/crates/fbuild-toolchain/src/toolchain/riscv.rs @@ -6,7 +6,6 @@ //! riscv-none-elf-size. use std::path::{Path, PathBuf}; -use std::process::Command; use crate::{CacheSubdir, PackageBase, PackageInfo, Toolchain}; @@ -175,14 +174,24 @@ impl RiscvToolchain { fn get_gcc_multilib_dir(&self, march: &str, mabi: &str) -> Option { // allow-direct-spawn: short synchronous GCC capability probe (-print-multi-directory). - let output = Command::new(self.get_gcc_path()) - .args([ - format!("-march={march}"), - format!("-mabi={mabi}"), - "-print-multi-directory".into(), - ]) - .output() - .ok()?; + let mut probe = fbuild_core::platform::process::command(self.get_gcc_path()); + probe.args([ + format!("-march={march}"), + format!("-mabi={mabi}"), + "-print-multi-directory".into(), + ]); + // Fork under the shared fork lock (released before the wait), with the + // stdio `output()` would set, so a stalled probe cannot block loader + // installation (coderabbit review on #1633). + probe + .stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()); + let child = { + let _fork = fbuild_core::platform::process::shared_fork_guard(); + probe.spawn().ok()? + }; + let output = child.wait_with_output().ok()?; if !output.status.success() { return None; } @@ -261,9 +270,11 @@ impl crate::Package for RiscvToolchain { if self.executable_prefix == "auto" { return detect_tool_prefix(&root.join("bin")).is_some(); } - root.join("bin") - .join(tool_name(&format!("{}-gcc", self.executable_prefix))) - .exists() + tool_binary( + &root.join("bin"), + &format!("{}-gcc", self.executable_prefix), + ) + .exists() } fn get_info(&self) -> PackageInfo { @@ -403,34 +414,34 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -/// Get the tool binary name with .exe extension on Windows. -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - -/// Get the full path to a tool binary. +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } /// Find one complete RISC-V compiler suite in an extracted PlatformIO /// package. A package with multiple suites is ambiguous and must not be /// selected by directory iteration order. fn detect_tool_prefix(bin_dir: &Path) -> Option { - let gcc_suffix = tool_name("-gcc"); - let mut candidates = std::fs::read_dir(bin_dir) + use fbuild_core::platform::executable::{find_tool_in, split_tool_suffix}; + + // A suite may spell gcc natively or as an APE (`.com`/`.exe`); dedupe so + // two spellings of one suite do not look like two suites. + let prefixes = std::fs::read_dir(bin_dir) .ok()? .filter_map(Result::ok) .filter_map(|entry| { let name = entry.file_name().to_string_lossy().into_owned(); - let prefix = name.strip_suffix(&gcc_suffix)?; - if !prefix.starts_with("riscv") || !entry.path().is_file() { - return None; - } - ["g++", "ar", "objcopy", "size"] + let prefix = split_tool_suffix(&name).0.strip_suffix("-gcc")?; + prefix.starts_with("riscv").then(|| prefix.to_string()) + }) + .collect::>(); + let mut candidates = prefixes + .into_iter() + .filter(|prefix| { + ["gcc", "g++", "ar", "objcopy", "size"] .iter() - .all(|suffix| tool_binary(bin_dir, &format!("{prefix}-{suffix}")).is_file()) - .then(|| prefix.to_string()) + .all(|suffix| find_tool_in(bin_dir, &format!("{prefix}-{suffix}")).is_some()) }) .collect::>(); if candidates.len() == 1 { @@ -453,15 +464,6 @@ mod tests { assert!(url.contains("riscv-none-elf")); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("riscv-none-elf-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("riscv-none-elf-gcc") - ); - } - #[test] fn ch32v_platform_toolchain_uses_wch_executable_prefix() { let tmp = tempfile::TempDir::new().unwrap(); @@ -473,11 +475,15 @@ mod tests { RiscvToolchain::with_platform_override(tmp.path(), override_package, "riscv-wch-elf"); assert_eq!( toolchain.get_gcc_path().file_name().unwrap(), - std::ffi::OsStr::new(&tool_name("riscv-wch-elf-gcc")) + std::ffi::OsStr::new(&fbuild_core::platform::executable::native_name( + "riscv-wch-elf-gcc" + )) ); assert_eq!( toolchain.get_ar_path().file_name().unwrap(), - std::ffi::OsStr::new(&tool_name("riscv-wch-elf-ar")) + std::ffi::OsStr::new(&fbuild_core::platform::executable::native_name( + "riscv-wch-elf-ar" + )) ); assert!(toolchain.get_info().url.contains("toolchain-riscv-linux")); } diff --git a/crates/fbuild-toolchain/src/toolchain/rp2040_picotool.rs b/crates/fbuild-toolchain/src/toolchain/rp2040_picotool.rs index d0be3c1a6..7d54aba34 100644 --- a/crates/fbuild-toolchain/src/toolchain/rp2040_picotool.rs +++ b/crates/fbuild-toolchain/src/toolchain/rp2040_picotool.rs @@ -67,11 +67,15 @@ impl Rp2040Picotool { } pub fn executable(&self) -> PathBuf { - self.resolved_dir().join(picotool_name()) + fbuild_core::platform::executable::resolve_tool_in(&self.resolved_dir(), PICOTOOL) + .into_path_buf() } fn validate(install_dir: &Path) -> fbuild_core::Result<()> { - let executable = find_picotool_root(install_dir).join(picotool_name()); + let executable = fbuild_core::platform::executable::resolve_tool_in( + &find_picotool_root(install_dir), + PICOTOOL, + ); if !executable.is_file() { return Err(fbuild_core::FbuildError::PackageError(format!( "managed picotool executable not found at {}; managed picotool assets exist for {} (the windows asset is x86_64 and is also what Windows ARM64 receives, relying on x64 emulation) — other host platforms are unsupported", @@ -184,19 +188,18 @@ fn platform_package() -> PlatformPackage { } fn find_picotool_root(install_dir: &Path) -> PathBuf { - if install_dir.join(picotool_name()).is_file() { + if fbuild_core::platform::executable::find_tool_in(install_dir, PICOTOOL).is_some() { return install_dir.to_path_buf(); } let nested = install_dir.join("picotool"); - if nested.join(picotool_name()).is_file() { + if fbuild_core::platform::executable::find_tool_in(&nested, PICOTOOL).is_some() { return nested; } install_dir.to_path_buf() } -fn picotool_name() -> &'static str { - fbuild_core::platform::executable::name("picotool", "picotool.exe") -} +/// Executable stem; the native spelling or a `.com`/`.exe` APE is accepted. +const PICOTOOL: &str = "picotool"; #[cfg(test)] mod tests { @@ -217,7 +220,11 @@ mod tests { let temp = tempfile::tempdir().unwrap(); let nested = temp.path().join("picotool"); std::fs::create_dir(&nested).unwrap(); - std::fs::write(nested.join(picotool_name()), []).unwrap(); + std::fs::write( + nested.join(fbuild_core::platform::executable::native_name(PICOTOOL)), + [], + ) + .unwrap(); assert_eq!(find_picotool_root(temp.path()), nested); } diff --git a/crates/fbuild-toolchain/src/toolchain/rp2040_pqt.rs b/crates/fbuild-toolchain/src/toolchain/rp2040_pqt.rs index ef9c0d04d..51cc32ad5 100644 --- a/crates/fbuild-toolchain/src/toolchain/rp2040_pqt.rs +++ b/crates/fbuild-toolchain/src/toolchain/rp2040_pqt.rs @@ -116,9 +116,7 @@ impl crate::Package for Rp2040PqtToolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin") - .join(tool_name("arm-none-eabi-gcc")) - .exists() + tool_binary(&root.join("bin"), "arm-none-eabi-gcc").exists() } fn get_info(&self) -> PackageInfo { @@ -245,12 +243,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] @@ -265,15 +260,6 @@ mod tests { assert!(url.contains("arm-none-eabi")); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("arm-none-eabi-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("arm-none-eabi-gcc") - ); - } - #[test] fn test_find_bin_root_direct() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/fbuild-toolchain/src/toolchain/teensy_arm.rs b/crates/fbuild-toolchain/src/toolchain/teensy_arm.rs index aa5437d09..9f8e69825 100644 --- a/crates/fbuild-toolchain/src/toolchain/teensy_arm.rs +++ b/crates/fbuild-toolchain/src/toolchain/teensy_arm.rs @@ -109,9 +109,7 @@ impl crate::Package for TeensyArmToolchain { return false; } let root = find_bin_root(&self.base.install_path()); - root.join("bin") - .join(tool_name("arm-none-eabi-gcc")) - .exists() + tool_binary(&root.join("bin"), "arm-none-eabi-gcc").exists() } fn get_info(&self) -> PackageInfo { @@ -245,12 +243,9 @@ fn find_bin_root(install_dir: &Path) -> PathBuf { install_dir.to_path_buf() } -fn tool_name(name: &str) -> String { - fbuild_core::platform::executable::native_name(name) -} - +/// Get the full path to a tool binary (native spelling, or a `.com`/`.exe` APE). fn tool_binary(bin_dir: &Path, name: &str) -> PathBuf { - bin_dir.join(tool_name(name)) + fbuild_core::platform::executable::resolve_tool_in(bin_dir, name).into_path_buf() } #[cfg(test)] @@ -268,15 +263,6 @@ mod tests { assert!(url.contains(TEENSY_ARM_TOOLCHAIN_VERSION)); } - #[test] - fn test_tool_name_platform() { - let name = tool_name("arm-none-eabi-gcc"); - assert_eq!( - name, - fbuild_core::platform::executable::native_name("arm-none-eabi-gcc") - ); - } - #[test] fn test_find_bin_root_direct() { let tmp = tempfile::TempDir::new().unwrap();