feat(job)!: adaptive stopping for compare jobs by kannwism · Pull Request #990 · RapidataAI/rapidata-python-sdk

kannwism · 2026-10-29T10:38:38Z

Why

Compare jobs paid for a fixed 10 votes per pair whether the pair was unanimous after 7 or a coin flip after 30. Adaptive stopping (agreed with marian@rapidata.ai, sibling legs in rapidata-backend, rapidata-evaluator and app-frontend) stops each pair at its first settled verdict, and the SDK makes it the compare default. Breaking: callers who never passed responses_per_datapoint go from a fixed 10 to adaptive (~25 votes per pair on average (measured on past compare jobs), ~31 at the hardest split; max_responses=100 is only a safety cap — the longest possible run is 78); this needs a major release.

Flow

flowchart LR
  user["create_compare_job_definition"]:::changed ==>|"referee: AdaptiveStoppingReferee"| orders["POST job definition / revision"]
  jd["job_def.responses_per_datapoint"]:::changed ==>|"IReferee round-trip"| orders
  cost["job.estimated_cost"]:::changed ==>|"GET cost-estimate (expected)"| orders
  prog["job.get_progress()"]:::changed ==>|"GET progress (completed/total)"| orders
  res["RapidataResults"]:::changed ==>|"stoppingState → verdict, responses"| export[("results export (evaluator)")]
  classDef new fill:#d3f9d8,stroke:#2f9e44,color:#000
  classDef changed fill:#fff3bf,stroke:#f08c00,color:#000
  classDef removed fill:#ffe3e3,stroke:#e03131,color:#000,stroke-dasharray:4

API	Change	Before → After
`create_compare_job_definition(responses_per_datapoint=…)`	changed	`int = 10` → `int \| AdaptiveStopping \| QuorumStopping = AdaptiveStopping()`
`create_classification_job_definition(responses_per_datapoint=…)`	changed	`int` → `int \| QuorumStopping`; `AdaptiveStopping` raises
`confidence_threshold`, `quorum_threshold`	deprecated	`DeprecationWarning`; raise with a config object
`RapidataJobDefinition.responses_per_datapoint`	changed	`int` → `int \| AdaptiveStopping \| QuorumStopping`
`CostEstimate`	changed	`estimated_cost` is now the expected cost (no max cost: per review, it is statistically near-unreachable)
`JobProgress`	changed	+ `completed`, `total`
compare result datapoint	changed	+ `verdict`, `responses` (from `stoppingState`)

Decisions

Deprecated thresholds with the compare default keep their old cap of 10 instead of raising — otherwise every existing confidence_threshold= caller breaks; only an explicitly passed config object raises.
An int assigned to job_def.responses_per_datapoint now means a fixed count (matching creation), except on a legacy confidence-threshold referee, which has no config object and keeps the int-tunes-the-cap behaviour.
IRefereeAdaptiveStoppingReferee and its IReferee entry are hand-written placeholders in the generator's shape so this builds before the backend ships; they get replaced by the regenerated client before merge. IRefereeConfig's new AdaptiveStoppingRefereeConfig arrives with that regen (no hand-written code reads IRefereeConfig).

Migration note in docs/migration.md; skill (SKILL.md, reference.md, examples.md) updated, including the "Unsure" example that would now raise.

Verified

pytest (512, incl. 28 new in tests/rapidata_client/job/test_adaptive_stopping.py against mocked generated client), pyright src/rapidata/rapidata_client 0 errors, mkdocs build clean.

BREAKING CHANGE: compare jobs default to adaptive stopping (~25 responses per pair on average) instead of a fixed 10.

🔗 Session: ves-14z8til7twlbzp

🤖 Generated with Claude Code

Compare jobs default to responses_per_datapoint=AdaptiveStopping(): each pair stops at its first settled verdict (A, B, ambiguous) or at max_responses (undecided). QuorumStopping replaces quorum_threshold; confidence_threshold and quorum_threshold are deprecated. Results gain verdict/responses from stoppingState, CostEstimate gains max_cost (estimated_cost is now the expected cost), JobProgress gains completed/total. The generated IRefereeAdaptiveStoppingReferee and maxCost fields are placeholders until the client is regenerated from the deployed backend. BREAKING CHANGE: create_compare_job_definition without responses_per_datapoint now uses adaptive stopping (up to 100 responses per pair) instead of a fixed 10. Pass responses_per_datapoint=10 to keep the old behaviour. job_definition.responses_per_datapoint returns AdaptiveStopping/QuorumStopping for definitions that use them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

kannwism

Let's change these things here first, then I'll review the more. But I keep reading about the wrong 100 cap

kannwism · 2026-10-29T12:52:56Z

+| an `int` | Collects exactly that many responses. Default (`10`) for every other job type. | All |

-You can use one or the other, but not both at the same time.
+## Adaptive Stopping


Don't really need this heading

Removed in cbd3f48 — the page is now about adaptive stopping, so its sections are top-level.

kannwism · 2026-10-29T12:53:30Z

@@ -1,25 +1,144 @@
 # Early Stopping


This is called adaptive stopping now

Renamed the page and its nav entry to Adaptive Stopping, and updated the links that pointed to it.

kannwism · 2026-10-29T14:28:30Z

+
+The pair stops at the first verdict. A clear pair stops after as few as 7 votes, a 50/50 pair after about 16, and a pair needs about 21 votes on average.
+
+The guarantee holds no matter when the pair stops: the chance that a verdict is wrong stays below `1 - min_confidence`.


remove this sentence

kannwism · 2026-10-29T14:30:21Z

+)
+```
+
+1. The same as leaving the argument out. `AdaptiveStopping(max_responses=50)` lowers the cap.


Remove this sentence. If you want, put tightening to interval [0.25, 0.75] as an example.

Removed. Replaced it with AdaptiveStopping(ambiguity=0.375), which gives the 25/75–75/25 band: about 28 votes on average (vs 21) and about 49 at the hardest split (unit weights).

kannwism · 2026-10-29T14:31:09Z

+The defaults (`min_confidence=0.9`, `ambiguity=0.32`, `max_responses=100`) are calibrated; change them only if you know why. Raising `ambiguity` narrows the ambiguous band, so near-even pairs resolve to a winner — at the cost of many more votes.
+
+!!! note
+    Adaptive stopping is only available for compare jobs, and cannot be combined with `AllowNeitherBothSetting` (the "Unsure" button). Use `QuorumStopping` or a fixed `int` there.


remove the cannot be combined section, we're gonna make this compatible soon

Removed from the docs, including the parameters page and the migration note. The code still raises in v1 because that's the agreed contract. The agent skill keeps one line about it so agents don't hit the error. I'll drop both once the backend supports the combination.

kannwism · 2026-10-29T14:33:02Z

 |---|---|
-| `estimated_cost` | The estimated total cost of running the job to completion, in your account's billing currency. |
+| `estimated_cost` | The expected total cost of running the job to completion, in your account's billing currency. |
+| `max_cost` | The cost if every datapoint collects its maximum number of responses. |


no max cost

kannwism · 2026-10-29T14:34:41Z

+from rapidata import AdaptiveStopping, QuorumStopping
+
+responses_per_datapoint=15                                          # fixed
+responses_per_datapoint=AdaptiveStopping(max_responses=50)          # compare only


don't put setting the cap as an example, that's misleading as this is not an important param. Use the default or tighten ambiguity if you want an example

Replaced the cap example with AdaptiveStopping(ambiguity=0.375).

kannwism · 2026-10-29T14:35:02Z

 #### `AllowNeitherBothSetting()`

-For Compare jobs, allows labelers to select "Neither" or "Both" instead of forcing a choice.
+For Compare jobs, allows labelers to select "Neither" or "Both" instead of forcing a choice. It cannot be combined with adaptive stopping, the compare default, so pass a fixed `responses_per_datapoint` or a `QuorumStopping`.


Don't mention for now

Removed the sentence. The example still passes responses_per_datapoint=10 so that it runs while v1 rejects the combination.

kannwism · 2026-10-29T14:36:43Z

+
+## Compare jobs stop adaptively (4.0)
+
+`create_compare_job_definition` now defaults to `responses_per_datapoint=AdaptiveStopping()`. Each pair stops once its verdict is settled, using up to 100 responses instead of exactly 10.


Not true, there is no actuall combination where one could use 100 responses. The 100 is just such that no cap is in place. In reality, the max. would be around 80 with default settings, but this near impossible to reach. The realistic value to report is the expected votes at the worst shares q

Agreed, and I checked it against the reference rule: with default settings the longest possible run is 78 votes, the average over all splits is about 21, and the worst-case split (≈35/65) expects about 31. The docs, docstrings and skill now report 21 on average and 31 for the hardest pairs, and call max_responses a safety cap that the defaults never reach.

kannwism · 2026-10-29T14:36:56Z

+
+| Before | Now |
+|--------|-----|
+| no `responses_per_datapoint` on a compare job: 10 responses | adaptive stopping, up to 100 (about 21 on average) |


Not up to 100

Fixed: about 21 on average.

max_responses is a safety cap the defaults never reach (the longest possible run is 78 votes), so report the expected votes instead: ~21 on average, ~31 at the hardest split. CostEstimate no longer exposes max_cost; estimated_cost is the expected cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

The cost estimate is priced with the backend's measured n-hat of 25.1 votes on past compare jobs; 21 was the uniform-prior figure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

RapidPoseidon · 2026-10-29T14:42:57Z

Correction to my earlier thread replies: the docs now say ~25 votes per pair on average, not 21. 25 is the backend's measured n̂ = 25.1 on historical compare jobs, and estimated_cost is priced with it; 21 was the average under a uniform prior. The worst split (q ≈ 0.35) stays at ~31. The ambiguity=0.375 example now says "about a third more votes" rather than giving an absolute number, because only uniform-prior figures exist for it. (aeb8b81)

AdaptiveStopping takes ambiguity_range (wire ambiguityRange), the width of the ambiguous band centred on 0.5. The default 0.6 is the old ambiguity 0.32, so the rule and calibration are unchanged. BREAKING CHANGE: AdaptiveStopping(ambiguity=...) is now AdaptiveStopping(ambiguity_range=...); there is no alias. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

RapidPoseidon · 2026-10-29T15:23:13Z

Parameter rename, decided by the requester: AdaptiveStopping(ambiguity=0.32) is now AdaptiveStopping(ambiguity_range=0.6), wire ambiguityRange, with no alias. It is the width of the ambiguous band centred on 0.5, so the band is [0.5 − r/2, 0.5 + r/2]. The default 0.6 is the same 20/80–80/20 band as before, so the rule and calibration don't change. Lowering it narrows the band and pushes closer pairs to a winner at the cost of more votes. The docs example is now ambiguity_range=0.5 (25/75–75/25), and they also mention 0.3 (35/65–65/35). This supersedes the ambiguity=0.375 example I mentioned earlier in the review threads. (bd105ad)

RapidPoseidon · 2026-10-29T16:04:39Z

Merge gate besides approval: src/rapidata/api_client/models/i_referee_adaptive_stopping_referee.py and its IReferee entry are hand-written placeholders. I'll regenerate the client from the deployed backend before merging, because the compare default sends AdaptiveStoppingReferee and a release ahead of the backend would break every new compare job. I'll merge once the regen is pushed, CI is green and the PR is approved.

…State fields verdict gains "open" (no verdict yet) and is never None on adaptive datapoints; responses is stoppingState.responseCount, which replaces decidedAtResponse, so the aggregatedResults fallback is gone. BREAKING CHANGE: an adaptive datapoint without a verdict now reports verdict "open" instead of None. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

RapidPoseidon · 2026-10-29T16:39:56Z

Contract update, decided by the requester (3856a82d9): stoppingState.verdict is now required and gains Open, so the SDK reports "open" where it used to return None. responses now reads stoppingState.responseCount, which replaces decidedAtResponse, and I removed the fallback that summed aggregatedResults. Datapoints from non-adaptive jobs still carry no verdict or responses at all.

The orders-side AdaptiveStoppingReferee may omit maxVotes, which the backend defaults to 100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

RapidPoseidon · 2026-10-30T13:14:49Z

Back to draft: the backend PR for this wire contract (rapidata-backend#5764) has changes requested and needs a second review, so the contract isn't final. Everything I had in flight is pushed (196a2357c). I won't change the contract or merge until the backend review settles, and then I'll regenerate the client against it.

kannwism commented Oct 29, 2026

View reviewed changes

RapidPoseidon and others added 2 commits October 29, 2026 14:40

kannwism marked this pull request as ready for review October 29, 2026 16:04

kannwism requested a review from LinoGiger as a code owner October 29, 2026 16:04

fix(job): read a missing adaptive maxVotes as 100

196a235

The orders-side AdaptiveStoppingReferee may omit maxVotes, which the backend defaults to 100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: marian@rapidata.ai <18029781+kannwism@users.noreply.github.com>

RapidPoseidon marked this pull request as draft October 30, 2026 13:14


		The pair stops at the first verdict. A clear pair stops after as few as 7 votes, a 50/50 pair after about 16, and a pair needs about 21 votes on average.

		The guarantee holds no matter when the pair stops: the chance that a verdict is wrong stays below `1 - min_confidence`.


		## Compare jobs stop adaptively (4.0)

		`create_compare_job_definition` now defaults to `responses_per_datapoint=AdaptiveStopping()`. Each pair stops once its verdict is settled, using up to 100 responses instead of exactly 10.

Conversation

kannwism commented Oct 29, 2026 • edited by RapidPoseidon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Flow

Decisions

Verified

Uh oh!

kannwism left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RapidPoseidon commented Oct 29, 2026

Uh oh!

RapidPoseidon commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RapidPoseidon commented Oct 29, 2026

Uh oh!

RapidPoseidon commented Oct 29, 2026

Uh oh!

RapidPoseidon commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kannwism commented Oct 29, 2026 •

edited by RapidPoseidon

Loading

RapidPoseidon commented Oct 29, 2026 •

edited

Loading