feat(agents)!: honor run_limits.max_turns on codex and antigravity

[bai-uipath]

Closes #108. Advances #68 (max_turns only — see Scope below).

run_limits.max_turns was accepted and silently ignored on Codex and Antigravity. A task file saying max_turns: 6 ran capped on Claude Code and unbounded on the other two, so the same file was not the same task.

max_turns on codex and antigravity

Both now cap on visible turns: resolved tool calls, the unit visible_turn_count already reports and the list TurnRecord.commands already holds. Both read it live off a shared EventCollector.visible_turn_count so one value means one thing on both, rather than two counters that happen to agree.

They need their own counter because each delivers a single SDK turn per communicate(), so a native cap would clamp at 1 regardless of what the task asked for.

The cap fires on the same loop boundary as the cooperative early stop: the step that reaches it is processed whole, the next is never pulled, the in-flight turn is cancelled server-side, and the run finalizes as max_turns_exhausted — a clean stop, not a crash, not retried. Criteria are still checked against whatever the agent produced.

On Antigravity that boundary lives inside _drain(), which #111 introduced along with a poll loop for backgrounded work. Putting the check there means the initial drain and every poll re-drain honor the cap identically, and the poll loop stops polling once the cap is reached rather than waiting out a background job on a run that is already over. A regression test covers that seam: without the guard, a capped run keeps polling for its full 120-cycle budget.

Verified on the eval VM

Two fixtures added under tasks/run_limits/, one prompt per limit, run on all three harnesses (claude-code on Bedrock, codex gpt-5.4, antigravity gemini-3.5-flash).

max_turns_cap.yaml — prompt asks for 12 sequential file writes, max_turns: 4:

Harness	resolved tool calls	max_turns_exhausted	final_status
claude-code	4	true	SUCCESS
codex	4	true	SUCCESS
antigravity	4	true	SUCCESS

All three stop at exactly 4 and finish cleanly, with the first file on disk so the criteria grade real work. On main, codex and antigravity run the prompt to completion and write all 12.

Batching prompt (parallel tool calls encouraged), max_turns: 2:

Harness	resolved tool calls	assistant messages
claude-code	12	14
codex	2 (+1 in-flight, recorded unresolved)	1
antigravity	2	1

This quantifies the residual divergence rather than hiding it. claude-code keeps its native SDK cap, whose unit is the SDK's own agent-loop turn, and that turn absorbs however many parallel calls the model emits — so a cap of 2 permitted all 12 writes. Holding max_turns constant across harnesses does not hold the budget constant. If you are A/B-ing across backends and the cap is close to binding, that is the number to distrust. Reimplementing claude-code's cap in the other unit would mean throwing away a real, working SDK cap, so it is documented instead.

The codex +1 is the tool already in flight when the cap fired: the run stopped after 2 completed calls, and the third is force-closed with result_status: unknown rather than dropped, so the trajectory shows what was interrupted.

turn_timeout.yaml — sleep 240 under a 45s watchdog:

Harness	outcome	duration
claude-code	turn timeout, partial turn captured, crashed: true	45.6s
codex	turn timeout, partial turn captured, crashed: true	46.4s
antigravity	poll budget exhausted, tool force-closed, turn graded, crashed: false	40.0s

Claude Code and Codex behave identically here. Antigravity stops earlier and more gently, for the reason below.

Antigravity: what the 10-second boundary still costs you

The Antigravity localharness has a 10-second maximum synchronous wait for shell commands. Past it the command becomes a background task and the model gets a task id instead of a result. That is harness behavior, not something coder_eval configures; sleep 5 resolving normally in 10.4s pins the boundary.

#111 made that survivable: the turn polls for the backgrounded result instead of finalizing on an idle step stream. Measured here on a command that finishes inside the budget (sleep 60 writing a file, turn_timeout: 300), same task and model on either side of that change:

	outcome	duration
without the poll loop	FAILURE — file never written, tool left unresolved	19.5s
with it	SUCCESS — file written, exit code reported back	75.0s

What remains is a bounded wait rather than an unlimited one: 80% of turn_timeout (or 120 five-second cycles when the task sets no timeout), not turn_timeout itself. A job that outlives that bound is force-closed as unresolved and graded on everything else, where Claude Code and Codex raise a turn timeout and mark the turn crashed.

So the residual divergence is the terminal signal, not whether slow work completes: a long npm install or build now runs to completion here the way it does on the other two, but a command that never finishes reads as an ordinary low score rather than a timeout. Documented in ANTIGRAVITY.md and HARNESS_PARITY.md.

Smaller items

• Drops the report's N turn(s) avoided claim. It derived from max_turns - sdk_turn_index, which on the single-SDK-turn backends advertised the whole budget as saved when only a tool-call tail was cut. The bound is still persisted on EarlyStopInfo, labelled there for what it is.

• Pins the user simulator's model. It passed model=None, so BEDROCK_MODEL decided who the simulated user was — an A/B varying the subject model silently varied its interlocutor too. Now a config field with its own default, translated per backend the way the LLM judge's is, and recorded on the result so simulator cost prices from a fact.

• Antigravity's mock-CLI PATH shadowing moves to the SDK's env seam (shipped in the 0.1.7 we already pin), deleting the process-wide os.environ mutation and the lock that serialized every harness start. Concurrent tasks now get genuinely separate environments instead of a time-sliced global one.

  Verified against the real SDK rather than only the fake: LocalAgentConfig.env → env=self.env → merged_env = {**os.environ, **env_map} → subprocess.Popen(env=merged_env), so our PATH wins over the inherited one and omitting env still spawns with plain inheritance. Confirmed live on the VM too: a record_cli shim for uip reached the agent's PATH, the agent got MOCK-UIP-SENTINEL 9.9.9 back, and the invocation landed in cli_mocks/calls.jsonl. A test now pins the SDK-side field, since the unit tests stub LocalAgentConfig and would all stay green if a version bump dropped it while mock CLIs silently stopped shadowing.

• Refreshes the model ids in docs examples (Sonnet 4-6 / Opus 4-7 / the dated 2025 ids → Sonnet 5 / Opus 5). The anthropic.claude-sonnet-4-6 values under llm_judge and simulation are left alone: those are the documented code default, and moving them changes verdicts rather than docs.

Scope

Three things were built and then deliberately cut, each because it turned out to be its own design problem rather than a side note to the turn cap:

• system_prompt parity is fix(agent): append system_prompt to the Claude Code preset instead of replacing #92's, which is further along in review and additionally carries the system_prompt_mode escape hatch that judge sub-agents need. Harness config parity: a base-config field must mean the same thing on every harness #68 stays open until it lands.
• config_support (declare per-agent divergences, reject unhonored config at resolution) had no real user: every declaration came out approximated, so the guard only ever fired for a synthetic test agent, and the one field that genuinely warranted unhonored could not be declared without hard-erroring live nightlies.
• Antigravity allowed_tools mapping works, but the questions around it are not small: whether the config field should be spelled in Claude's tool vocabulary at all, and what an allowlist that maps to nothing should do (falling back to every tool hands the model more than it asked for, which is the wrong direction for an eval harness).

Reference

docs/agents/HARNESS_PARITY.md carries the run-limit contract: what each field means per harness, what a capped run looks like, and the Antigravity backgrounding behavior. The measurements stay here rather than on the page, since they are a snapshot of one box on one day and the page is published to coder-eval.com. tasks/run_limits/ holds the fixtures to re-measure with.

🤖 Generated with Claude Code

[github-actions]

Claude finished @bai-uipath's task in 1m 49s —— View job

---

🔍 PR Review in Progress

Todo List:

• Read code review guidelines (.github/code_review.md)
• Read project conventions (CLAUDE.md)
• Analyze the full PR diff
• Review each changed file with full context
• Check cross-file consistency
• Analyze what's missing
• Post comprehensive review

Starting review...

[akshaylive]

Review: feat(agents)!: honor run_limits.max_turns on codex and antigravity

PR #110 by @bai-uipath · bai/harness-config-parity → main · OPEN · reviewed against 7142fdf

This is a well-executed piece of harness-parity work: it closes a real gap (Codex and Antigravity previously ignored run_limits.max_turns entirely), the two agents' new enforcement logic is structurally identical to each other and correctly diverges from Claude Code's native counter only where the SDKs actually differ, and the breaking change is documented unusually well for a repo this size — a new dedicated page (HARNESS_PARITY.md), cross-links from both agent docs, and worked task fixtures. Full test suite passes (4304 passed, 13 skipped), and security/type/error-handling checks came back clean. The main things worth addressing before merge are a couple of test gaps around the new "clean stop" exception path and a documentation gap around max_turns's per-agent unit semantics. Overall score 9.5/10, weakest axes are Code Quality and Harness Quality (tied at 8.9/10) — driven entirely by Medium/Low findings, no Critical or High issues.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.9/10	0	0	2	1	max_turns_reached()/ended_cleanly duplicated verbatim across codex/antigravity turn-state classes
2. Type Safety	10.0/10	0	0	0	0	No findings
3. Test Health	9.0/10	0	0	2	0	No test drives an exception during teardown after ended_cleanly (max_turns/stopped_early)
4. Security	10.0/10	0	0	0	0	No findings
5. Architecture & Design	9.7/10	0	0	0	3	max_turns carries two incompatible units across backends by design (documented, not a defect)
6. Error Handling & Resilience	10.0/10	0	0	0	0	No findings
7. API Surface & Maintainability	9.4/10	0	0	1	1	TASK_DEFINITION_GUIDE.md still documents the removed "N turn(s) avoided" report note
8. Evaluation Harness Quality	8.9/10	0	0	2	1	reports_stats.visible_turn_count can read cap + 1 under Antigravity's background-poll edge case

Overall Score: 9.5 / 10 · Weakest Axis: Code Quality (1) and Harness Quality (8), tied at 8.9 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 7 · 🔵 6 across 8 axes reviewed.

Blockers

No blockers found.

Non-blocking, but please consider before merge

Scoring/reporting correctness

1. run_limits.max_turns now means two structurally different things per agent (a native SDK inner-loop turn on Claude Code, which can bundle parallel tool calls, vs. one resolved tool call on Codex/Antigravity), and only docs/agents/HARNESS_PARITY.md says so. The field's own Pydantic description (src/coder_eval/models/limits.py:32-36) gives no hint of the divergence, so a user setting one max_turns for a cross-agent A/B can reasonably (and wrongly) assume it's a portable unit. Add a one-line pointer from the description to HARNESS_PARITY.md.
2. reports_stats.visible_turn_count (src/coder_eval/reports_stats.py:422-432) counts all TurnRecord.commands entries regardless of result_status, while the live cap check (EventCollector.visible_turn_count) counts only resolved calls. Under Antigravity's background-poll edge case (a >10s shell command still in flight when the cap fires), the orphaned call gets force-closed as result_status: unknown and lands in commands after the cap decision — so the persisted metric can read cap + 1, contradicting HARNESS_PARITY.md's own claim that "the resolved count still matches" the cap. Doesn't affect final_status/score, but does affect the expected_turns_overage threshold metric and report badges.

Test coverage

3. The PR widens the "ignore exception on a clean break" guard from stopped_early_hit alone to ended_cleanly (stopped_early_hit or max_turns_hit) in both agents specifically to stop a cap-triggered break from escalating into a crash/retry — but no test in either test_codex_agent.py or test_antigravity_agent.py actually raises during teardown after max_turns_hit=True to prove the turn still finalizes cleanly instead of raising AgentCrashError.
4. The bundled SimulationConfig.model/SimulationTelemetry.simulator_model addition reprioritizes simulator_cost_usd()'s pricing precedence and changes how orchestrator.py resolves sim_model_id — but no test in the diff exercises either the new precedence or the new resolution path (tests/test_run_limits_orchestrator.py's only change is a mock-satisfaction tweak, not a behavioral assertion).

Docs

5. docs/TASK_DEFINITION_GUIDE.md:486 still shows the removed "N turn(s) avoided" report-note phrase, but reports.py's own diff deliberately deletes that claim (its comment explains why: it overstated how much work an early-stopped row actually skipped). Update the doc's example to match.
6. SimulationConfig.model (a field tracked by CE030) got a docs update only in the simulation section of TASK_DEFINITION_GUIDE.md, not in any cross-referencing table — likely fine, flagged for completeness.

Minor code quality

7. _CodexTurnState/_AntigravityTurnState's max_turns_reached()/ended_cleanly are near-identical copy-pasted implementations with no shared base — both just delegate to EventCollector.visible_turn_count. A small shared helper would remove the two-copies-to-keep-in-sync risk.
8. The end-of-turn status precedence (timeout > stopped_early > max_turns_exhausted > completed) is now hand-implemented three times (Codex, Antigravity, and the pre-existing Claude Code path) with three different shapes — worth a shared resolve_end_status(...) helper if this precedence rule ever needs to change again.

Nits

• docs/agents/HARNESS_PARITY.md is a substantial new page — worth confirming it's linked from the docs nav (make docs-indexes/CE028) rather than orphaned.
• UserSimulator._resolve_model (src/coder_eval/simulation/user_simulator.py:233-234) imports to_anthropic_alias/to_bedrock_model/BedrockRoute locally inside the method with no circular-import hazard and no comment explaining why — hoist to module scope or explain the choice.
• SimulationTelemetry.simulator_model (new field) currently has only one reader (simulator_cost_usd) — legitimate, but consider surfacing it in the human-readable simulation report block in a follow-up.

What's Missing

• Tests: covered above (items 3, 4) — no additional gaps found beyond those.
• Downstream consumers: covered above (item 2) — no other counting/classification consumer missed the new max_turns-cap status.
• Display & mapping dicts: AgentEndStatus.MAX_TURNS_EXHAUSTED/FinalStatus.MAX_TURNS_EXHAUSTED are pre-existing (from the Claude Code path), so no new enum value was introduced — all report renderers already handle it. Nothing identified.
• Parallel paths: Codex and Antigravity's implementations were checked structurally identical to each other; the only divergence risk is the code-duplication note under item 7 above.

Harness & Lint Improvements

Static checks (lint / type):

• A lint rule that greps a diff for removed user-facing string literals in reports*.py and flags any doc under docs/ still quoting the removed phrase would have caught item 5 mechanically (Technique 1 applied to doc/code string parity — neither CE028 nor CE030 currently check this).

Harness improvements:

• A golden/parity test comparing EventCollector.visible_turn_count against the persisted reports_stats.visible_turn_count under a synthetic Antigravity background-poll-then-cap scenario would catch item 2's drift directly — this needs a live event stream to diff, so it can't be a pure static check.
• Runtime fault-injection tests for "raise during post-cap/post-stop teardown" (item 3) are inherently a test-authoring gap, not something static analysis can surface.

Top 5 Priority Actions

1. Add a one-line pointer from RunLimits.max_turns's field description (and its row in TASK_DEFINITION_GUIDE.md) to HARNESS_PARITY.md, since the field now means a structurally different unit per agent and the schema description gives no hint of this.
2. Add tests on both agents that raise an exception during teardown after ended_cleanly (stopped_early or max_turns) and assert the turn still finalizes cleanly with max_turns_exhausted=True instead of escalating to AgentCrashError.
3. Reconcile reports_stats.visible_turn_count's counting of force-closed unknown-status commands against the cap-time decision, so a capped Antigravity run's persisted turn-count metric can't silently read cap + 1.
4. Add a direct test asserting simulator_cost_usd() prices from the new SimulationConfig.model/SimulationTelemetry.simulator_model when set, and an orchestrator-level test confirming sim_model_id resolves from the simulator's actual model.
5. Extract the duplicated max_turns_reached()/ended_cleanly implementations (and consider a shared resolve_end_status(...) helper for the three hand-synced status-precedence cascades) across the agent implementations.

---

Change class: complex — introduces new max_turns enforcement/control-flow mirrored across two Agent implementations, changes turn-end status precedence logic, and is a documented breaking change
Stats: 0 🔴 · 0 🟠 · 7 🟡 · 6 🔵 across 8 axes reviewed.
Full per-axis breakdown: tmp/code-review-260813-1942/01-code-quality.md … tmp/code-review-260813-1942/08-harness-quality.md.