ci: constrain internal builds to exact local wheels - #2510
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test |
a571b64 to
472b7cd
Compare
472b7cd to
f39b4c0
Compare
This comment has been minimized.
This comment has been minimized.
f39b4c0 to
5fcc5fd
Compare
3e8866f to
9eb4470
Compare
|
@kkraus14 This PR now keeps dependency selection entirely in CI, uses exact constraints for both pathfinder and bindings across every in-scope workflow, and no longer changes either build hook. I plan to merge after clean CI unless you see a remaining blocker. (At this moment the CI is very unreliable b/o infrastructure issues; I'll keep retrying; we had a clean run before I merged 2520, therefore I believe it should work again.) |
kkraus14
left a comment
kkraus14
left a comment
There was a problem hiding this comment.
Reposting comment I made offline:
There is so much unnecessary added complexity here. We shouldn't need a whole python helper for constraint file generation. We already know the exactly wheel we built earlier for dependencies so we can easily just pass that information through in the GitHub Action workflows.
9eb4470 to
2126160
Compare
|
@kkraus14, I removed the Python helper and moved the constraint generation directly into the workflows, following your suggestion. CI is running, and I've updated the PR description to reflect the new implementation. Could you please take another look? |
| - name: Constrain cuda.core to the local cuda.bindings wheel | ||
| run: | | ||
| pathfinder_wheels=(cuda_pathfinder/cuda_pathfinder-*.whl) | ||
| bindings_wheels=("${CUDA_BINDINGS_ARTIFACTS_DIR}"/cuda_bindings-"${BUILD_CUDA_MAJOR}".*.whl) | ||
| test "${#pathfinder_wheels[@]}" -eq 1 | ||
| test "${#bindings_wheels[@]}" -eq 1 | ||
| test -f "${pathfinder_wheels[0]}" | ||
| test -f "${bindings_wheels[0]}" | ||
| mkdir -p wheel-constraints | ||
| if [[ "${{ inputs.host-platform }}" == win* ]]; then | ||
| pathfinder_uri="file:///$(cygpath -am "${pathfinder_wheels[0]}")" | ||
| bindings_uri="file:///$(cygpath -am "${bindings_wheels[0]}")" | ||
| else | ||
| pathfinder_uri="file:///host$(realpath "${pathfinder_wheels[0]}")" | ||
| bindings_uri="file:///host$(realpath "${bindings_wheels[0]}")" | ||
| fi | ||
| { | ||
| printf 'cuda-pathfinder @ %s\n' "${pathfinder_uri}" | ||
| printf 'cuda-bindings @ %s\n' "${bindings_uri}" | ||
| } | tee wheel-constraints/cuda-core.txt | ||
|
|
There was a problem hiding this comment.
I thought we explicitly want to test cuda-core against the already-published cuda-bindings to make sure we don't break backward compat. Does this remove that kind of testing (in favor of testing with latest)? I think ideally we need to do both.
There was a problem hiding this comment.
I thought we explicitly want to test cuda-core against the already-published cuda-bindings to make sure we don't break backward compat. Does this remove that kind of testing (in favor of testing with latest)?
Good question. I asked codex to carefully inspect the CI code. The answer is:
No, this PR does not remove that compatibility testing.
I think ideally we need to do both.
Agreed. That is already what our CI is meant to do. This PR makes the current-build side reliable without changing the published-wheel compatibility side.
This repository already explicitly implements both:
| Scenario | cuda-bindings used with current cuda-core |
|---|---|
| Exact build/test CUDA minor | Just-built wheel from the current run |
| Same major, older test minor | Published PyPI wheel |
| Different major | Wheel from the backport branch |
nightly-cuda-core |
Reverse direction: released cuda-core with main-built cuda-pathfinder and cuda-bindings |
At a high level, the build and test stages have deliberately different policies:
- While building the wheels, the packages from the current checkout are built in dependency order: pathfinder, then bindings, then core. This PR ensures that each downstream PEP 517 build actually uses the corresponding wheel produced earlier in that run.
- When testing the resulting core wheel,
BINDINGS_SOURCEselects which bindings wheel to install:- an exact CUDA major/minor match uses the just-built bindings wheel;
- a same-major, different-minor match uses the published PyPI wheel for that minor, specifically for the real-world backward-compatibility scenario;
- a major-version mismatch uses the bindings artifact from the backport branch.
For example, the current build version is CUDA 13.3, while the PR test matrix contains many CUDA 13.0.2 rows. Those rows install cuda-bindings==13.0.* from PyPI, then install the core wheel built by the current CI run and run the core tests. Thus the matrix exercises both current bindings and already-published bindings.
Evidence for published-wheel compatibility coverage
ci/tools/env-varsexplicitly definespublishedas installing from PyPI. For a same-major/minor-mismatch it selects that mode "to test the real-world backward-compat scenario."ci/tools/run-testsimplements the policy: inpublishedmode it installscuda-bindings==${TEST_CUDA_MAJOR}.${TEST_CUDA_MINOR}.*, then installs the current-run core wheel and runs its tests.ci/versions.ymland the CUDA 13.0.2 entries inci/test-matrix.ymlensure that this mode is exercised while the current build CUDA version is 13.3.- There is also a separate nightly test in the opposite direction: released cuda-core from PyPI against main-built pathfinder and bindings (
ci/test-matrix.ymlandci/tools/run-tests).
Evidence for current-run/local-wheel coverage
- The same
ci/tools/env-varslogic says that an exact CUDA major/minor match uses "the just-built bindings wheel." - The test workflows download the core artifact, and in
mainmode the bindings artifact, from the applicable CI run and pass those wheel paths directly to pip. - Before this PR, the ordinary build sequence and its
PIP_FIND_LINKSsettings already indicated that the newly built local wheels were intended to feed downstream builds. The weakness was thatPIP_FIND_LINKSonly added candidates; it did not guarantee that pip selected them over PyPI. - This PR closes that enforcement gap with direct local-wheel constraints. Those are build-environment constraints, not exact requirements embedded into the finished core wheel. The core wheel continues to declare the broad runtime dependency
cuda-bindings[all]==13.*, so a published 13.0 bindings wheel still satisfies it.
The PR changes only the build-wheel, coverage, and sdist-build workflows. It does not change ci/tools/env-vars, ci/tools/run-tests, the test matrix, or the test-wheel workflows that implement the published-wheel compatibility coverage.
One distinction worth calling out: the existing compatibility lane builds the core wheel with the current-run bindings, then installs and tests that wheel with an older published bindings version. It does not separately build/cythonize core against the older binding. If that second build-time scenario is what you meant, it would be additional coverage rather than something this PR removes; the previous unconstrained resolver behavior did not select an older release systematically enough to provide such coverage.
74ea497 to
39e9ac6
Compare
|
4 participants
Description
Fixes #2468.
xref: #2464, #2470
Same-checkout CI builds must use the exact CUDA Python wheel artifacts selected by the workflow. A compatible package from PyPI must not satisfy an internal build dependency when CI intends to exercise a local artifact.
This PR generates PEP 508 direct-reference constraints for the selected
cuda-pathfinderandcuda-bindingswheels, for example:The workflows apply each constraint file through both
PIP_BUILD_CONSTRAINTandPIP_CONSTRAINT. This constrains dependencies installed into PEP 517 isolated build environments as well as other pip dependency resolution in the build step, while leaving package indexes available for unrelated third-party dependencies.PRs #2509 and #2520 report the versions and import locations of build dependencies. Those diagnostics make the selected artifacts visible in CI logs; the constraints in this PR enforce their selection.
Dependency constraints
The protected build edges are:
cuda.bindingscuda-pathfinderwheelcuda.corecuda-pathfinderwheel and onecuda-bindingswheel matching the current CUDA majorcuda.corecuda-pathfinderwheel and one downloadedcuda-bindingswheel matching the previous CUDA majorEach constraint-generation step verifies that exactly one matching artifact exists before starting the consumer build. The previous-major bindings artifact is kept in a separate directory so current- and previous-major wheels cannot become an ambiguous input set.
The direct references use paths appropriate to each execution environment:
/hostmount.cygpathfor Git Bash and native pip subprocesses.Affected environments install pip 25.3 or newer for build-constraint support.
Workflow coverage
The constraints are applied consistently across:
cuda.corebuild;The policy is contained entirely in CI configuration. Package build requirements and ordinary builds without these constraint variables are unchanged.