Skip to content

Queue pull request CI runs per author and workflow - #1426

Open
dhruv8sh wants to merge 7 commits into
mainfrom
chore/ci-per-user-concurrency
Open

dhruv8sh wants to merge 7 commits into
mainfrom
chore/ci-per-user-concurrency

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • One author opening or pushing to many pull requests could fill the shared runner pool and make everyone else's CI wait. Each PR-triggered workflow now gets a concurrency group keyed on the workflow and the PR author (github.event.pull_request.user.login), so an author has at most one run in progress per workflow and their extra runs wait as pending without holding a runner. Keying on the author rather than github.actor means a maintainer pushing to or clicking "Update branch" on contributors' PRs does not pile those runs into the maintainer's queue.
  • queue: max keeps every waiting run and starts them in order. The default single-pending queue would silently cancel an author's older waiting run whenever they open another PR. Every job now sets timeout-minutes, so a hung job cannot hold an author's queue for GitHub's 6-hour default.
  • Pushes to main, scheduled runs, manual dispatches and bot-authored PRs (for example Dependabot) fall back to github.run_id, so they get their own group and are never held behind anyone's queue.

Changes

File Change
.github/workflows/test.yml Add per-PR-author concurrency group with queue: max; add timeout-minutes to the six jobs that lacked one
.github/workflows/format.yml Same concurrency group; add timeout-minutes to all five jobs
.github/workflows/codeql.yml Same concurrency group; add timeout-minutes to analyze
.github/workflows/integration-tests.yml Same concurrency group (jobs already had timeouts)
.github/workflows/pbs-example.yml Same concurrency group (job already had a timeout)

deploy-docs.yml and docs-proposal.yml are untouched: they only run on main and already define their own concurrency.

concurrency:
  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }}
  cancel-in-progress: false
  queue: max

New timeouts are about 3x the longest of each job's last five main runs, with a 10-minute floor (5 for the symlink guard):

Job Longest recent run Timeout
test-axum 10.8 min 30
test-cloudflare 2.9 min 15
test-spin 5.2 min 20
test-parity 3.1 min 15
test-cli (macOS leg) 14.9 min 45
test-typescript 1.1 min 10
check-claude-md-symlink 0.1 min 5
format-rust 7.2 min 25
format-typescript 0.6 min 10
format-docs 0.8 min 10
docs-proposal-scripts 0.3 min 10
CodeQL analyze (rust leg) 11.7 min 45

Known trade-offs: several pushes to the same PR now run one after another rather than in parallel, and a re-run joins the back of the author's queue. Cancelling a PR's superseded runs needs a separate privileged workflow, since a workflow can only have one concurrency group; that is left for a follow-up.

Closes

Closes #1425

Spec

Spec: Not a feature (CI change)

Test plan

  • cargo test-fastly && cargo test-axum (and cargo test-cloudflare)
  • cargo clippy-fastly && cargo clippy-axum (and cargo clippy-cloudflare)
  • cargo fmt --all -- --check
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run
  • JS format: cd crates/trusted-server-js/lib && npm run format
  • Docs format: cd docs && npm run format
  • WASM build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
  • Manual testing via fastly compute serve
  • Other:
    • Live GitHub run: ran the concurrency block with queue: max in a throwaway private repo. One author opened three PRs back to back, then pushed to main while they were queued. The PR runs ran one at a time, in creation order, none cancelled (pr-2 started 4s after pr-1 finished, pr-3 4s after pr-2). The main push started 4s after it was created while two PR runs were still pending.
    • Group evaluation with act: evaluated each workflow's actual group: expression. Two PRs by one author map to one group, different authors to separate groups, a maintainer pushing to another author's PR to that author's group, and Dependabot PRs, push, schedule and workflow_dispatch to run-ID groups. The same script asserts every job sets timeout-minutes. Against the previous revision it fails on the maintainer, Dependabot and timeout checks, as expected.
    • actionlint 1.7.12 and act 0.2.89 both flag queue as an unknown key; their schemas predate it. GitHub accepts it, and docs-proposal.yml already uses queue: max on main.

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds (n/a, not a feature)
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...") (no Rust changes)
  • Uses tracing macros (not println!) (no Rust changes)
  • New code has tests (workflow config only; verified by the live run and act checks above)
  • No secrets or credentials committed

One author opening or pushing many pull requests could fill the runner queue and delay everyone else. Each PR-triggered workflow now uses a concurrency group keyed on the workflow and the actor, so an author has one run in progress per workflow while the rest wait in order (queue: max) rather than being cancelled. Pushes to main, scheduled runs and manual dispatches fall back to the run ID and are never held back.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh dhruv8sh self-assigned this Oct 20, 2026
dhruv8sh and others added 2 commits October 21, 2026 00:47
Group pull request runs by the PR author instead of the triggering actor, so a maintainer pushing to or updating contributors' branches does not pile those runs into the maintainer's queue. Bot-authored PRs fall back to the run ID so a Dependabot batch is not serialized.

Give every job in the queued workflows a timeout of roughly three times its longest recent run, so a hung job cannot hold an author's queue for GitHub's six-hour default.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed all five changed workflows at 3d448f910c2c0445b979441452b29d401074d42d, including job dependencies, matrices, shared setup, triggers, permissions and concurrency policies. No blocking defects found; the implementation matches issue #1425. Author rather than actor grouping correctly handles maintainer updates, and all 18 job definitions have explicit timeouts.

Non-blocking observations

  • 🤔 Workflow limits do not guarantee runner fairness — The concurrency block in .github/workflows/test.yml:16–19 admits one workflow run per author, but that run can launch eight jobs. Format and CodeQL can launch another eight simultaneously, alongside integration jobs. This substantially reduces burst multiplication, but still permits significant runner occupancy by one author. Measure contention after rollout before adding further serialization.

  • 📝 Clarify queue capacity and ordering — The repeated comments in the five workflows say queued runs are preserved and run in order. GitHub permits 100 pending runs per group, then cancels additional arrivals. FIFO follows when runs begin waiting on the concurrency group, rather than workflow dispatch order. Consider documenting those qualifications in the shared wording and PR description. See GitHub concurrency documentation. Apply manually: this clarification spans five workflow files.

  • 🏕 Retain the policy verification script — The PR describes useful expression checks with act, but the verification script is not included in the change. Keeping it in the repository would help catch drift across the five duplicated policies. Cover author versus actor, bots, non-PR events, group separation and timeout presence. Apply manually: retaining the script introduces a new file.

Suggested follow-up

Retain this author-level cap, then separately consider cancelling superseded runs within the same PR. That would reduce obsolete work while preserving the cap across different PRs. Any privileged cancellation workflow should manipulate Actions metadata without checking out or executing contributor code. Stale-run queueing is explicitly acknowledged in this PR and is not a blocker.

Validation

Local YAML/policy assertions passed for all five identical concurrency policies and all 18 job definitions; git diff --check passed. Local actionlint reported its outdated schema for queue and shell warnings on unchanged lines. Current GitHub documentation confirms queue: max is supported. Rust/JS suites were not rerun locally for this workflow-only change, and the reported live multi-PR experiment was not independently reproduced.

CI Status

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Queuing PR CI runs per author and workflow, with bounded job timeouts and independent groups for bots and non-PR runs, is a focused improvement to shared runner fairness.

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This adds a per-PR-author concurrency group with queue: max to the five PR-triggered workflows, and a timeout-minutes on every job. The timeouts hold up against the last 400 runs of each workflow and against the cold runs after the Rust 1.98.1 upgrade. Two things block: without superseded-run cancellation, the per-author queue delays required checks by hours and can hit the 100-pending cap, and nothing keeps the new invariants in place as open PRs land.

1 of the inline comments below carries a one-click GitHub suggestion — use Commit suggestion to apply it as a commit on the PR branch. The remaining comments describe the fix in prose because the change touches multiple files or lines outside the diff and can't be auto-applied.

Blocking

🔧 wrench

  • Per-author queue without superseded-run cancellation — see inline at .github/workflows/test.yml:17
  • Enforce the shared block and job timeouts — see inline at .github/workflows/format.yml:58

Non-blocking

🤔 thinking

  • Bot exemption covers the runs nobody waits on — see inline at .github/workflows/test.yml:14

📝 note

  • Windows axum leg from the module stack — see inline at .github/workflows/test.yml:81

CI Status

  • integration tests: PASS
  • browser integration tests: PASS
  • integration tests (Fastly EC lifecycle): PASS
  • CodeQL: PASS
  • cargo test (ts CLI, native) (macos-latest): PASS
  • Analyze (python): PASS
  • docs-proposal scripts: PASS
  • Analyze (javascript-typescript): PASS (both runs)
  • Analyze (rust): PASS
  • vitest: PASS
  • cargo fmt: PASS (required)
  • cargo test (axum native): PASS
  • CLAUDE.md symlink guard: PASS
  • format-typescript: PASS (required)
  • prepare integration artifacts: PASS
  • cargo test (ts CLI, native) (ubuntu-latest): PASS
  • format-docs: PASS (required)
  • cargo check/build/test (spin native + wasm32-wasip1): PASS
  • cargo test (cross-adapter parity): PASS
  • PBS Terraform and Compose checks: PASS
  • cargo check (cloudflare native + wasm32-unknown-unknown): PASS
  • cargo test: PASS (required)
  • Analyze (actions): PASS

Comment thread .github/workflows/test.yml Outdated
Comment thread .github/workflows/format.yml
Comment thread .github/workflows/test.yml Outdated
Comment thread .github/workflows/test.yml
A workflow-level group per pull request now cancels a run when a newer push arrives, so stale runs no longer hold an author's queue or count toward GitHub's 100-pending cap. The per-author queue moves to each job's own concurrency block with queue: max, keyed on the matrix for matrix jobs so their legs do not queue behind each other. Bot-authored PRs and non-PR events still key on the run ID.

The shared comment now states the FIFO ordering and the 100-pending limit.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The required cargo fmt job now fails when a PR-triggered workflow lacks the per-PR cancellation block from test.yml, or when a job lacks timeout-minutes or a per-author queue keyed on pull_request.user.login with queue: max. Matrix jobs must also include the matrix in their group. Jobs that call a reusable workflow are skipped. The step uses the runner image's yq.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh
dhruv8sh requested a review from aram356 October 27, 2026 16:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Queue pull request CI runs per author to stop one contributor blocking the runner pool

4 participants