Repository navigation
Respect CPU affinity and cgroup CPU quota in the default thread count - #584
Open
manunicholasjacob wants to merge 2 commits into
Open
manunicholasjacob wants to merge 2 commits into
manunicholasjacob wants to merge 2 commits into
Conversation
detect_number_of_cores() returned SC_NPROCESSORS_ONLN, the machine's online core count, ignoring any CPU affinity mask the process runs under. A process pinned to a subset of cores by taskset, by `docker run --cpuset-cpus`, or by the cpuset a SLURM or Kubernetes job is placed in was still told the full core count, so the default thread pool oversubscribed the cores it was actually allowed to use. This is the common failure mode when numexpr is pulled in as an indirect dependency and nobody has set NUMEXPR_MAX_THREADS (pydatagh-328): pinned to 2 of 20 cores the default pool still started 16 threads. Prefer len(os.sched_getaffinity(0)) where the OS exposes it (Linux), falling back to the existing sysconf / sysctl / NUMBER_OF_PROCESSORS probes everywhere else. The returned count is only ever reduced, and only when the process is genuinely restricted; the unconstrained default and every explicitly configured path (NUMEXPR_MAX_THREADS, NUMEXPR_NUM_THREADS, OMP_NUM_THREADS) are unchanged. Verified on x86-64 (20 cores, pinned to 2 -> 2 threads) and on aarch64 (Raspberry Pi 5, 4 cores, pinned to 2 -> 2 threads). Adds a regression test that pins the process and asserts the detected count matches the affinity mask. Signed-off-by: Manu Nicholas Jacob <manunicholasjacob@gmail.com>
The affinity fix covers cpuset-style restriction (taskset, --cpuset-cpus, a SLURM cpuset). A CPU *quota* set without a cpuset does not change the affinity mask, so it was still invisible: `docker run --cpus=2` or a Kubernetes CPU limit on a 20-core host left the default pool at 16 threads, oversubscribing the 2-CPU share eightfold. This is the most common containerised case. When numexpr is left to auto-detect (no NUMEXPR_MAX_THREADS / NUMEXPR_NUM_THREADS / OMP_NUM_THREADS), cap the default by the cgroup CPU quota in addition to the affinity mask. The quota is read from cgroup v2 `cpu.max` or cgroup v1 `cpu.cfs_quota_us` / `cpu.cfs_period_us` for the process cgroup and its ancestors, taking the tightest, rounded up. No quota, or a non-Linux platform, leaves the count unchanged. Verified end to end: inside a `cpu.max = 200000 100000` (2.0 CPUs) cgroup with the affinity mask left at all 20 cores, numexpr.nthreads drops from 16 to 2; with no quota it stays 16. Adds a root-free regression test that parses v2 and v1 quota fixtures. Signed-off-by: Manu Nicholas Jacob <manunicholasjacob@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When numexpr is left to auto-detect its thread count (no
NUMEXPR_MAX_THREADS/NUMEXPR_NUM_THREADS/OMP_NUM_THREADS), it oversubscribes a process that has been confinedto part of the machine, in two ways:
detect_number_of_cores()returnsSC_NPROCESSORS_ONLN, the machine'sonline core count, and ignores the affinity mask. A process pinned by
taskset,docker run --cpuset-cpus, or the cpuset that SLURM / Kubernetes place a job in is stilltold the full core count.
docker run --cpus=2, a Kubernetes CPUlimit) does not change the affinity mask, so it was invisible as well.
This is the failure mode #328 was about: numexpr is almost always an indirect dependency
(
pandas->tables->numexpr), nobody downstream configures it, and the auto-default spinsup more threads than the job was given. The 16-thread cap from #328 limits the blast radius on
large machines but does nothing for a job confined to 2 or 4 cores.
Reproduction (before)
20-core x86-64. Pinned to 2 cores, and limited to 2 CPUs by quota, both still start 16 threads:
Raspberry Pi 5, aarch64, numexpr 2.8.4:
Fix
detect_number_of_cores()now preferslen(os.sched_getaffinity(0))where the OS exposes it(Linux), falling back to the existing
sysconf/sysctl/NUMBER_OF_PROCESSORSprobeseverywhere else.
_init_num_threads()additionally caps the auto-detected default by the cgroup CPU quota,read from cgroup v2
cpu.maxor cgroup v1cpu.cfs_quota_us/cpu.cfs_period_us(processcgroup and ancestors, tightest wins), rounded up. The quota cap lives here rather than in
detect_number_of_cores()because a fractional CPU-time budget is not a core count.The default is only ever reduced, and only when the process is genuinely restricted; the
unconstrained default and every explicitly configured path (
NUMEXPR_MAX_THREADS,NUMEXPR_NUM_THREADS,OMP_NUM_THREADS) are unchanged.After
Tests
test_detect_cores_respects_affinity: pins the process to a single CPU and asserts thedetected count matches the mask (skips where
sched_setaffinityis unavailable or fewer thantwo CPUs are allowed). Fails on
master, passes here.test_cgroup_cpu_quota_parsing: parses cgroup v2cpu.maxand cgroup v1cpu.cfs_quota_us/cpu.cfs_period_usfixtures (no root, no real cgroup required),including the
max= unlimited case.Full suite: 6044 passed. Verified end to end on x86-64 (20 cores) and aarch64 (Raspberry
Pi 5, 4 cores).