Skip to content

Respect CPU affinity and cgroup CPU quota in the default thread count - #584

Open
manunicholasjacob wants to merge 2 commits into
pydata:masterfrom
manunicholasjacob:fix/detect-cores-respects-cpu-affinity
Open

manunicholasjacob wants to merge 2 commits into
pydata:masterfrom
manunicholasjacob:fix/detect-cores-respects-cpu-affinity

Conversation

@manunicholasjacob

@manunicholasjacob manunicholasjacob commented Oct 30, 2026 •

Copy link
Copy Markdown

What

When numexpr is left to auto-detect its thread count (no NUMEXPR_MAX_THREADS /
NUMEXPR_NUM_THREADS / OMP_NUM_THREADS), it oversubscribes a process that has been confined
to part of the machine, in two ways:

  1. CPU affinity. detect_number_of_cores() returns SC_NPROCESSORS_ONLN, the machine's
    online core count, and ignores the affinity mask. A process pinned by taskset,
    docker run --cpuset-cpus, or the cpuset that SLURM / Kubernetes place a job in is still
    told the full core count.
  2. CPU quota. A CPU quota set without a cpuset (docker run --cpus=2, a Kubernetes CPU
    limit) does not change the affinity mask, so it was invisible as well.

This is the failure mode #328 was about: numexpr is almost always an indirect dependency
(pandas -> tables -> numexpr), nobody downstream configures it, and the auto-default spins
up more threads than the job was given. The 16-thread cap from #328 limits the blast radius on
large machines but does nothing for a job confined to 2 or 4 cores.

Reproduction (before)

20-core x86-64. Pinned to 2 cores, and limited to 2 CPUs by quota, both still start 16 threads:

$ python -c "import numexpr; print(numexpr.nthreads)"
16
$ taskset -c 0,1 python -c "import numexpr; print(numexpr.nthreads)"
16      # affinity: pinned to 2 cores, still 16 threads
# inside a cgroup with cpu.max = "200000 100000" (2.0 CPUs), affinity left at all 20:
$ python -c "import numexpr; print(numexpr.nthreads)"
16      # quota: limited to 2 CPUs, still 16 threads

Raspberry Pi 5, aarch64, numexpr 2.8.4:

$ taskset -c 0,1 python3 -c "import numexpr, numexpr.utils as u; print(u.detect_number_of_cores(), numexpr.nthreads)"
4 4     # pinned to 2 cores, still 4 threads

Fix

  • detect_number_of_cores() now prefers len(os.sched_getaffinity(0)) where the OS exposes it
    (Linux), falling back to the existing sysconf / sysctl / NUMBER_OF_PROCESSORS probes
    everywhere else.
  • _init_num_threads() additionally caps the auto-detected default by the cgroup CPU quota,
    read from cgroup v2 cpu.max or cgroup v1 cpu.cfs_quota_us / cpu.cfs_period_us (process
    cgroup and ancestors, tightest wins), rounded up. The quota cap lives here rather than in
    detect_number_of_cores() because a fractional CPU-time budget is not a core count.

The default is only ever reduced, and only when the process is genuinely restricted; the
unconstrained default and every explicitly configured path (NUMEXPR_MAX_THREADS,
NUMEXPR_NUM_THREADS, OMP_NUM_THREADS) are unchanged.

After

$ taskset -c 0,1 python -c "import numexpr; print(numexpr.nthreads)"
2       # affinity respected
# inside the cpu.max = 2.0 cgroup (affinity still all 20):
$ python -c "import numexpr; print(numexpr.nthreads)"
2       # quota respected
$ python -c "import numexpr; print(numexpr.nthreads)"
16      # unrestricted default unchanged

Tests

  • test_detect_cores_respects_affinity: pins the process to a single CPU and asserts the
    detected count matches the mask (skips where sched_setaffinity is unavailable or fewer than
    two CPUs are allowed). Fails on master, passes here.
  • test_cgroup_cpu_quota_parsing: parses cgroup v2 cpu.max and cgroup v1
    cpu.cfs_quota_us / cpu.cfs_period_us fixtures (no root, no real cgroup required),
    including the max = unlimited case.

Full suite: 6044 passed. Verified end to end on x86-64 (20 cores) and aarch64 (Raspberry
Pi 5, 4 cores).

detect_number_of_cores() returned SC_NPROCESSORS_ONLN, the machine's
online core count, ignoring any CPU affinity mask the process runs
under. A process pinned to a subset of cores by taskset, by
`docker run --cpuset-cpus`, or by the cpuset a SLURM or Kubernetes job
is placed in was still told the full core count, so the default thread
pool oversubscribed the cores it was actually allowed to use. This is
the common failure mode when numexpr is pulled in as an indirect
dependency and nobody has set NUMEXPR_MAX_THREADS (pydatagh-328): pinned to 2
of 20 cores the default pool still started 16 threads.

Prefer len(os.sched_getaffinity(0)) where the OS exposes it (Linux),
falling back to the existing sysconf / sysctl / NUMBER_OF_PROCESSORS
probes everywhere else. The returned count is only ever reduced, and
only when the process is genuinely restricted; the unconstrained
default and every explicitly configured path (NUMEXPR_MAX_THREADS,
NUMEXPR_NUM_THREADS, OMP_NUM_THREADS) are unchanged.

Verified on x86-64 (20 cores, pinned to 2 -> 2 threads) and on aarch64
(Raspberry Pi 5, 4 cores, pinned to 2 -> 2 threads). Adds a regression
test that pins the process and asserts the detected count matches the
affinity mask.

Signed-off-by: Manu Nicholas Jacob <manunicholasjacob@gmail.com>
The affinity fix covers cpuset-style restriction (taskset, --cpuset-cpus,
a SLURM cpuset). A CPU *quota* set without a cpuset does not change the
affinity mask, so it was still invisible: `docker run --cpus=2` or a
Kubernetes CPU limit on a 20-core host left the default pool at 16
threads, oversubscribing the 2-CPU share eightfold. This is the most
common containerised case.

When numexpr is left to auto-detect (no NUMEXPR_MAX_THREADS /
NUMEXPR_NUM_THREADS / OMP_NUM_THREADS), cap the default by the cgroup CPU
quota in addition to the affinity mask. The quota is read from cgroup v2
`cpu.max` or cgroup v1 `cpu.cfs_quota_us` / `cpu.cfs_period_us` for the
process cgroup and its ancestors, taking the tightest, rounded up. No
quota, or a non-Linux platform, leaves the count unchanged.

Verified end to end: inside a `cpu.max = 200000 100000` (2.0 CPUs) cgroup
with the affinity mask left at all 20 cores, numexpr.nthreads drops from
16 to 2; with no quota it stays 16. Adds a root-free regression test that
parses v2 and v1 quota fixtures.

Signed-off-by: Manu Nicholas Jacob <manunicholasjacob@gmail.com>
@manunicholasjacob manunicholasjacob changed the title Respect CPU affinity in detect_number_of_cores() Respect CPU affinity and cgroup CPU quota in the default thread count Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant