[MaxGhenis]

This PR moves litellm from 1.88.6 to 1.95.0 in uv.lock for Dependabot alert #96 (high, GHSA-g5ff-637f-6q2m, "internal_user_viewer arbitrary local file read via vertex_ai_credentials"). It also adds 1.95.0 to the litellm releases that the tool pilot's Gemini search patch accepts, since the lock bump alone trips the patch's version gate. It stays separate from #251 (pyasn1, tornado, filelock) because litellm builds every synchronous benchmark request. The two PRs touch different lock entries, so either can land first.

The board's requests don't change. Under 1.88.6 and 1.95.0, litellm builds the same method, URL and JSON body for all 635 requests we captured from policybench.config.MODELS, and only the user-agent header differs. In the tool pilot, 10 of 11 captured provider-and-arm cases send identical bodies. The Gemini web+pe continuation turn changes one field: 1.95.0 sends each Google Search part's own thoughtSignature back on that part, where 1.88.6 sent one signature on both parts.

pyproject.toml doesn't change. policyengine, policyengine-us, policyengine-uk and policyengine-core keep their exact lock entries from main, so no reference answer moves.

The advisory and the version

Package	Alert (severity)	Locked	First patched
litellm (direct, litellm[caching]>=1.30)	#96 (high)	1.88.6 → 1.95.0	1.95.0

• No 1.88.x release clears the alert. PyPI lists 1.88.6 as the last 1.88.x release, and the advisory covers every release below 1.95.0, so no 1.89–1.94 release clears it either. uv lock --upgrade-package litellm alone picked 1.106.6, so we ran uv lock --upgrade-package 'litellm==1.95.0'.

• No other package moves. litellm 1.95.0's lock entry lists the same dependencies as 1.88.6's. Compared field by field, the two lockfiles differ only in litellm's version, sdist and wheels, and their lock metadata matches.

• 1.88.6 already carries the fix commit. The advisory points to BerriAI/litellm 065faf6 (#34189). That commit changes seven library files:

  • proxy/auth/auth_utils.py adds vertex_ai_credentials to the request-body blocklist and checks fallback targets for banned parameters and URL destinations;
  • proxy/auth/user_api_key_auth.py validates fallbacks through the new fallback helper;
  • litellm_core_utils/url_utils.py adds provider_url_destination_candidates, and proxy/litellm_pre_call_utils.py checks URL destinations with it;
  • the Oobabooga and Hugging Face embedding handlers match a URL model only by its http:// or https:// prefix, and huggingface/embedding/transformation.py drops its own get_api_base.

  The installed 1.88.6 and 1.95.0 packages both contain every one of those changes. In clean installs of litellm[proxy] 1.88.6, 1.94.3 and 1.95.0, is_request_body_safe(request_body={"model": "gemini-pro", "vertex_ai_credentials": "/etc/passwd"}, ...) raises Rejected Request: vertex_ai_credentials is not allowed in request body on all three. Dependabot flags any release below 1.95.0 all the same, and landing 1.95.0 clears the alert.

• 1.95.0 has no macOS wheel. PyPI lists manylinux and Windows wheels and the sdist, and litellm 1.95.0 includes a Rust extension, so uv sync on a Mac builds it from source with maturin. On a Mac with Python 3.14 it built cp314-cp314-macosx_11_0_arm64. litellm 1.95.1 and later ship macOS wheels.

• PolicyBench doesn't run the litellm proxy. git grep finds no litellm.proxy import or proxy launch in policybench/, scripts/ or pyproject.toml. The vulnerable endpoint (/utils/transform_request) belongs to the proxy server.

What changes in litellm on PolicyBench's request path

We installed 1.88.6 (from main's lock) and 1.95.0 (from this lock) into separate virtual environments. We sorted each file as byte-identical, format-only (its Python syntax trees match, since 1.95.0 rewrapped much of the code at a longer line length) or changed.

Path under litellm/	Files	Identical	Format-only	Changed	Added
main.py	1	0	0	1	0
responses/	15	1	3	9	2
utils.py	1	0	0	1	0
llms/anthropic/	53	22	10	20	1
llms/openai/	60	22	27	11	0
llms/gemini/	23	7	6	9	1
llms/vertex_ai/gemini/	3	1	0	2	0
litellm_core_utils/prompt_templates/factory.py	1	0	0	1	0
litellm_core_utils/llm_response_utils/	5	1	2	2	0

To find the other modules on the request path, we traced every litellm file whose code ran during the request captures below: 90 files under 1.95.0 and 84 under 1.88.6. Code changed in 77 files across both lists, among them __init__.py, cost_calculator.py, exceptions.py, caching/caching.py and caching_handler.py, litellm_core_utils/exception_mapping_utils.py, get_llm_provider_logic.py, completion_timeout.py, llms/custom_httpx/llm_http_handler.py, http_handler.py and aiohttp_transport.py, and the xai, deepseek and openrouter chat transformations. Six files first appear in 1.95.0, among them litellm_core_utils/request_timeout_resolver.py, litellm_core_utils/fallback_generalizations.py and llms/xai/oauth.py.

1.95.0 restructures completion() in main.py: it adds a dispatch context and an agentic-loop hook, and calls AnthropicCacheControlHook.maybe_seed_default_injection_points from both completion() and acompletion(). We read the diffs and ran both versions against faked provider responses and errors.

Unchanged for PolicyBench's calls

• Applied timeouts stay the same. PolicyBench always passes timeout=_request_timeout_seconds(model_id) (policybench/eval_no_tools.py), which takes priority over the new global-timeout fallback in completion_timeout.py and request_timeout_resolver.py.
• Neither version adds retries: each error case sends the same number of requests.
• For HTTP 400, 401, 403, 404, 408, 409, 413, 422, 429 (including insufficient_quota), 500, 502, 503, 504 and 529, a context-window 400, and connect, read-timeout and protocol errors, both versions raise the same exception class, message, status_code and llm_provider. PolicyBench's _should_retry, _is_model_fatal_error and _is_retryable_provider_error return the same results. exceptions.py only adds attributes to RateLimitError.
• policybench/cache.py builds identical keys, and 1.95.0 reads entries 1.88.6 wrote as cache hits.
• For faked Anthropic, OpenAI Responses, Gemini, xAI, DeepSeek and OpenRouter responses, both versions parse the same content, tool_calls, function_call, finish reason, id, system_fingerprint, model, reasoning and cached-token counts, and _hidden_params["cache_hit"].
• litellm still sets a Gemini response's model to the requested name (model_response.model = model, llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py:2451). This PR names 1.95.0 beside 1.88.x in the two notes that record it, in policybench/config.py (gemini-3.7-flash entry) and docs/runbook.md:211.
• The Anthropic prompt-caching injection in integrations/anthropic_cache_control_hook.py returns [] unless litellm.enable_anthropic_prompt_caching is True. litellm reads that setting from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING, "false" when unset, and nothing in policybench/ or scripts/ sets it or passes cache_control_injection_points.

Changed, and reached by PolicyBench only in edge cases

• When an OpenAI-compatible chat provider (xAI, DeepSeek, OpenRouter) returns a 200 response with "choices": [], 1.88.6 rejected only a None choices and returned the empty list, so PolicyBench's response.choices[0] raised IndexError. 1.95.0 raises APIError(status_code=500, "LiteLLM: provider returned a response with no 'choices'...") (litellm_core_utils/llm_response_utils/convert_dict_to_response.py:631), and the board handles the household differently. Under 1.88.6, run_no_tools_eval recorded a failed row with the IndexError, and since is_infrastructure_error_text returns False for that error, the row scored as a model miss. Under 1.95.0, _is_retryable_provider_error returns True for the APIError, as it does for a provider's HTTP 500, so the run saves a checkpoint and raises (policybench/eval_no_tools.py:3597). That ends the eval-no-tools call, so the models after this one in --models don't run. A supervised run starts one eval-no-tools worker per household, each with one --model (policybench/supervisor.py:689-711), and a resume retries the household. No committed result row carries an IndexError error.
• litellm falls back to its bundled model-cost map only when it can't fetch the remote map or LITELLM_LOCAL_MODEL_COST_MAP tells it to, and PolicyBench sets that variable only in two tests. 1.95.0's bundled map adds xai/grok-4.5, claude-opus-5 and claude-sonnet-5. On the bundled map, 1.88.6 can't route claude-opus-5 or claude-sonnet-5 ("LLM Provider NOT provided") and can't price xai/grok-4.5, while 1.95.0 routes and prices all three. Both versions fetch the remote map from the same URL at import, and on it they price all three the same (litellm.cost_per_token).
• The async reference explainer (policybench/case_reference_explanations.py, litellm.acompletion) goes through llms/custom_httpx/aiohttp_transport.py, where 1.95.0 raises httpx.ReadError when a connection closes mid-body; 1.88.6 returned the partial body. We did not test this.
• Usage objects gain mirrored cache_write_tokens/cache_creation_tokens, service_tier and iterations fields. PolicyBench reads cache_write_tokens first, falls back to cache_creation_tokens, and extracts the same value. 1.95.0 drops message.provider_specific_fields["reasoning_content"] and keeps message.reasoning_content; PolicyBench reads neither.

Changed, behind settings PolicyBench doesn't use

1.95.0 rewrites reasoning_effort for OpenRouter ("max" → "xhigh") and DeepSeek ("none" → thinking: disabled), and PolicyBench's REASONING_EFFORT_OVERRIDES holds no entries. It raises a Responses max_output_tokens below 16 to 16. Its Anthropic adaptive-thinking and native-structured-output checks now read the cost map, and only a request with thinking, reasoning_effort or a json_schema response format reaches them. A drop_params= argument to responses() now takes effect. xAI OAuth (llms/xai/oauth.py) runs only with use_xai_oauth and no key. 1.95.0 also changes streaming code, and PolicyBench doesn't stream.

Board request bodies

We patched httpx.Client.send and httpx.AsyncClient.send after importing litellm and PolicyBench, so the import-time model-map fetch went out normally. Each patched send recorded the request and raised, so nothing reached a provider. With dummy API keys and POLICYBENCH_MAX_ATTEMPTS=1, we drove PolicyBench's own functions:

• run_single_no_tools for every one of the 52 MODELS entries, on a US household (TX, joint, two children) and a UK household (London, one adult), each with include_explanations=True and False. The capture includes the board's chunking: a model whose explanation contract chunks the outputs sends several requests.
• _request_predictions_once(..., repair=True) with and without explanations, and _request_explanations_once, for every model.
• case_reference_explanations._generate_one, which asks claude-haiku-4-5 with temperature: 0.0 and max_tokens: 500; both versions forward temperature.

Each version sent 635 requests in 365 groups, to api.anthropic.com/v1/messages, api.openai.com/v1/responses, generativelanguage.googleapis.com (v1alpha and v1beta generateContent), api.x.ai, api.deepseek.com/beta and openrouter.ai.

• With the remote cost map, PolicyBench's default, all 635 bodies and URLs match, and only the user-agent header differs.
• With the bundled map (LITELLM_LOCAL_MODEL_COST_MAP=True), 1.88.6 builds no request for claude-opus-5 or claude-sonnet-5, as above, and 1.95.0 builds the same 51 requests for them that both versions build with the remote map. The other 584 match.

We checked only the bodies litellm builds from PolicyBench's arguments. The board never sends a provider's reply back in a later request. The tool pilot does, so we captured its conversations separately. The batch path (policybench/batch_eval.py) calls provider SDKs directly, so litellm never touches it.

The tool pilot

allow_gemini_search_with_functions in scripts/tool_pilot_argon.py replaces two VertexGeminiConfig methods, _resolve_search_tool_conflict and _drop_search_tools_mixed_with_functions, so the pilot's Gemini web arms keep Google Search beside their function tools. It refuses any litellm outside GEMINI_SEARCH_PATCH_LITELLM, and this PR adds 1.95.0 to that set.

The patched methods

1.95.0 rewraps the lines of both patched methods, and their syntax trees match 1.88.6's. It adds _apply_include_server_side_tool_invocations, which map_openai_params calls before it maps any parameter. Its docstring reads "map_openai_params iterates non_default_params in request order; if tools appear before this flag, _resolve_search_tool_conflict would drop search tools before the flag is applied." The pilot's web arms always set that flag, so 1.95.0 keeps the search tool by itself. The test's own call, get_optional_params(model="gemini-4-argon", tools=[{"googleSearch": {}}, calculate_tool], tool_choice="required", include_server_side_tool_invocations=True), returns these tools:

	unpatched	patched
1.88.6	function_declarations	function_declarations, googleSearch
1.95.0	function_declarations, googleSearch	function_declarations, googleSearch

Without the flag, unpatched 1.95.0 still drops search.

Pilot request bodies

We drove the pilot's own run_household through litellm's completion and responses under each version, with httpx.Client.send patched to record each request and answer with a canned provider reply. The 11 cases cover Gemini 4 Argon, Claude Opus 5.5 and GPT-6 Sol on the pe, web and web+pe arms, plus Claude's two web arms with the search result arriving in the next response. In 10 cases every request body matches between versions. Our canned Claude answers missed the pilot's JSON answer format, so those runs also went through its two repair prompts and ended incomplete under both versions, with matching bodies request for request.

In the Gemini web+pe case, the continuation turn re-sends the first reply's Google Search parts with different signatures:

Signatures on Gemini's search parts	1.88.6 sends back	1.95.0 sends back
call SIG_CALL, result SIG_RESP	SIG_CALL on both	SIG_CALL on the call, SIG_RESP on the result
result SIG_RESP only	SIG_RESP on both	none on the call, SIG_RESP on the result

Google's guide to combining built-in tools with function calling (ai.google.dev/gemini-api/docs/tool-combination) says every built-in tool call and result carries its own signature, and asks a client that keeps its own history to send each one back. 1.95.0 does that.

Two other changes reach the pilot:

• 1.95.0 adds toolUsePromptTokenCount to prompt_tokens unless the response carries search grounding (groundingMetadata.webSearchQueries). In a canned Gemini web+pe run, for a reply that reported 1,000 prompt tokens and 150 tool-use tokens without grounding, the pilot recorded 1,150 prompt tokens under 1.95.0 and 1,000 under 1.88.6. The pilot's live probe found no grounding metadata on function-calling turns (docstring of turn_searches), so on the web arms 1.95.0 adds the tool-use tokens on every turn, search turns included. If live Gemini reports tool-use tokens on those turns, a 1.95.0 run prices them at the input rate on top of the per-query search fee. Google's guide says it doesn't charge for Google Search's tool tokens, since it prices each search per query (ai.google.dev/gemini-api/docs/tool-combination). 1.95.0 also stops counting thoughts twice in completion_tokens when candidatesTokenCount already includes them and a response reports tool-use tokens. A 1.95.0 web-arm run's token costs don't compare with a 1.88.x run's. Each run's manifest records its litellm version.
• Neither version re-sends a Claude search result that arrives in the response after its call. Without the pilot's _attach_deferred_results, no later request in either version carries it; with it, both versions carry it from the third request on. This PR names 1.95.0 beside 1.88.x in that method's docstring.

Tests

• UNPATCHED_GEMINI_TOOLS and RESENT_SEARCH_SIGNATURES record, for each gated release, the tools litellm keeps without the patch and the signatures it sends back. Two tests require an entry for every release in GEMINI_SEARCH_PATCH_LITELLM, so the next bump has to record both.

• test_the_gemini_patch_keeps_search_beside_functions checks the installed release's unpatched tools against its entry, then checks that the patch keeps search.

• test_the_gemini_web_and_tool_arm_runs_through_the_locked_litellm runs the web+pe arm through litellm's own Gemini code with canned replies. It checks that turn 0 sends googleSearch beside function_declarations with includeServerSideToolInvocations set, and that turn 1 sends the installed release's signatures on the search parts and SIG_FN on the function call.

• Each of these mutants fails a test:

  • on 1.95.0, giving 1.95.0 the 1.88.x signatures;
  • on 1.95.0, giving 1.95.0 the 1.88.x unpatched tools;
  • on 1.88.6, running the web+pe arm without the patch.

  Running that arm without the patch on 1.95.0 passes, since 1.95.0 keeps the search tool by itself, so CI, which runs the locked 1.95.0, can't catch a broken patch body. Only a test run on 1.88.x exercises it, as the patch's docstring says.

Verification

• CI at 77ebb5e: lint, test (7,010 passed, 90 skipped), app, paper and the Vercel preview pass.
• 77ebb5e changes one word in a comment (bill_searches → turn_searches), and scripts/tool_pilot_argon.py's syntax tree matches dc43c51's.
• CI at dc43c51: lint, test (7,010 passed, 90 skipped), app, paper and the Vercel preview pass.
• 081727b merges main again, bringing in Merge batch repair rounds cell by cell, like sync #180, Record PolicyEngine provenance without importing policyengine #186, Pin the reference adversary's leaderboard impact to the pass's inputs #205 and Patch pyasn1, tornado and filelock in uv.lock (16 Dependabot alerts) #251; git show --remerge-diff prints nothing for it. dc43c51 changes only comments and the patch's docstring: the two files' syntax trees match apart from that docstring.
• CI at d9f99ab: lint, test (6,903 passed, 90 skipped), app, paper and the Vercel preview pass.
• d9f99ab merges main, which brings in Count only real request timeouts in the supervisor's concurrency backoff #183 and Probe card-less models under the card the onboarding gauntlet derives #189. Count only real request timeouts in the supervisor's concurrency backoff #183's supervisor finds request timeouts in worker logs with log_shows_request_timeout, and its tests use log lines that workers wrote under litellm 1.88.1. On 1.95.0, log_shows_request_timeout still matches a printed litellm.Timeout and passes over litellm's warning that its cost-map fetch timed out.
• CI at c55e34a, the lock bump alone: lint, app, paper and the Vercel preview passed. The test job failed only the two gate tests, test_the_gemini_patch_keeps_search_beside_functions and test_the_locked_litellm_is_one_the_gemini_patch_was_checked_against (assert '1.95.0' in frozenset({'1.88.1', '1.88.6'})), with 6,781 passed and 90 skipped.
• Locally, tests/test_tool_pilot_argon.py gives 150 passed and 1 skipped on 1.95.0 (Python 3.14.0) at d9f99ab and at dc43c51, and the 9 tests whose names match gemini, checked_litellm or signatures pass on 1.88.6 too. ruff check and ruff format --check pass on the changed files.

Invariants

• The scoring inputs don't move: policyengine, policyengine-us, policyengine-uk and policyengine-core keep their exact lock entries from main.
• The board's request bodies don't change: 635 of 635 captured requests match with the remote cost map.
• In the tool pilot, 10 of 11 cases send identical bodies, and the Gemini web+pe continuation changes only the signatures on its search parts.
• Every litellm release the Gemini patch accepts has an entry for its unpatched tools and for the signatures it sends back.
• uv sync --locked accepts the lock, and uv lock --check passes under uv 0.11.32, 0.13.4 and 0.13.6. A newer uv rewrote the lockfile's revision = 3 header and pexpect's ptyprocess marker; both lines now match main's.

axiom: n/a: dependency security bump, no policy logic

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 2, 2026 8:09pm UTC

[MaxGhenis]

Ready for hub review

Head: c55e34aec8d32060420e4cd3e25282729e3dcfd0

Alert and how this PR resolves it

• litellm Publish PolicyBench dataset v1.1 (corrected ground truth, weights unchanged) #96 (high, GHSA-g5ff-637f-6q2m): 1.88.6 → 1.95.0, the first patched release named by the advisory (range < 1.95.0). No 1.88.x release after 1.88.6 exists.
• Nothing else in uv.lock moves.

Request bodies: none change. 635 captured requests are identical in method, URL and body under 1.88.6 and 1.95.0, with litellm's remote cost map. That covers every MODELS id, US and UK households, board requests with and without explanations, prediction repair, explanation repair, and the temperature=0.0 reference explainer. Only user-agent differs. With the bundled map (LITELLM_LOCAL_MODEL_COST_MAP), 1.95.0 can also route claude-opus-5 and claude-sonnet-5, which 1.88.6 cannot. Their bodies match the remote-map ones.

Tests

• CI at this head: lint, app, paper and the Vercel preview pass. The test job fails with 2 failed, 6781 passed, 90 skipped. The two failures are the tool pilot's Gemini-patch gate tests (assert '1.95.0' in frozenset({'1.88.1', '1.88.6'})).
• Local pytest -m "not slow" (Python 3.13): 6755 passed, 29 failed, 89 skipped. The failures are those 2 plus the same 27 tests/test_scenarios.py proxy-403 failures that occur on main's environment in this session.
• ruff check . and ruff format --check . pass.
• The lock comparison shows only litellm's version, sdist and wheels changing.

Judgment calls

1. Left CI red. litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py is not byte-identical in 1.95.0, so per the rule for this bump I did not add 1.95.0 to GEMINI_SEARCH_PATCH_LITELLM. The two patched methods are unchanged. 1.95.0's new _apply_include_server_side_tool_invocations keeps googleSearch beside functions on its own when the flag is set, and the pilot always sets it. Gemini continuation turns now re-send thought signatures differently. Landing needs the owner's decision on the gate.
2. 1.88.6 may already be fixed. 1.88.6's proxy/auth/auth_utils.py already bans vertex_ai_credentials, and its is_request_body_safe rejects it, as do 1.94.3 and 1.95.0. I could not read the upstream fix commit from this session, so I moved to 1.95.0 as the advisory states. Dismissing Publish PolicyBench dataset v1.1 (corrected ground truth, weights unchanged) #96 instead is an option. PolicyBench does not run the litellm proxy.
3. Used uv lock --upgrade-package 'litellm==1.95.0' because the plain upgrade picked 1.106.6.
4. Restored the lock revision = 3 header and pexpect's marker by hand, as in Patch pyasn1, tornado and filelock in uv.lock (16 Dependabot alerts) #251.
5. The code-diff review compared ASTs to set apart the format-only files (1.95.0 re-wrapped lines) and traced which litellm files actually run on the request path. Four parallel reviewers each read part of the diff, and I spot-checked their main claims in the code. Response-side edge cases that changed are listed in the description: empty choices now raises APIError, bundled-map pricing, and aiohttp mid-body close.
6. I did not edit the litellm 1.88.x notes in policybench/config.py and docs/runbook.md. The behaviour they describe (Gemini response model set to the requested name) is unchanged in 1.95.0.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

This is an independent review by a Claude Opus 5.5 cloud session for the PolicyBench owner session.

Verdict: REQUEST_CHANGES at d9f99ab812c10835788934793553b6677da9ad32.

The PR head on GitHub moved to 081727b9b1f81dc75f049dbcd4bd1bed1708ea88 while I was reviewing. That commit merges main (#205, #186, #180, #251) and git show --remerge-diff shows no hand edits. As the brief asked, I reviewed d9f99ab, not the new head.

The code and tests hold up. I'm asking for changes because of one description claim about scored rows that my run contradicts. The fix is text only. The other findings are non-blocking.

Setup: a clone at the head, and uv sync --locked --extra dev gave litellm 1.95.0 on Python 3.13.18. I also built a separate venv with litellm[caching]==1.88.6 (/tmp/l88), used uv run --with 'litellm[caching]==1.88.6' for the side-by-side runs, and synced a Python 3.12.3 env (UV_PROJECT_ENVIRONMENT=/tmp/v312 uv sync --locked --extra dev --python 3.12). Tests ran on a scratch merge of the head into origin/main at ec5413c3. litellm fetched its remote cost map normally, except where I say otherwise.

Blocking

1. Empty choices: 1.95.0 changes how the row is handled, not just the error text

The description says: "PolicyBench retries both, but a failed row's recorded error changes from IndexError to APIError." Under 1.95.0 no failed row is recorded at all. Instead the board's run loop raises and stops that model's run. Under 1.88.6 the same reply gave scored rows that counted as model misses.

Code at the head:

• policybench/eval_no_tools.py:3557-3560 (run_no_tools_eval): if _is_retryable_provider_error(e): ... _save_checkpoint(...); raise. The same branch is at :3818 in run_no_tools_single_output_eval.
• _is_retryable_provider_error (:445-451) returns True for 1.95.0's APIError and False for IndexError.
• is_infrastructure_error_text (:405-415) is what resume (:3071, :3430), retry_eval.py:42, chunked_eval.py:74 and row_repair.py:178 use to decide whether an error row counts as a model miss. It returns True for the 1.95.0 error text and False for IndexError: list index out of range.

What I ran: a fake xAI reply {"choices": []} with HTTP 200, POLICYBENCH_MAX_ATTEMPTS=1, and run_no_tools_eval called on two TX households, models={"grok-4.6": "xai/grok-4.6"}, programs=["snap"] (script /tmp/caps/emptyeval.py):

1.95.0 raised APIError litellm.APIError: LiteLLM: provider returned a response with no 'choic
1.88.6 returned rows 2 errors: ['IndexError: list index out of range', 'IndexError: list index out of range'] infra: [False, False]

The classification functions, called directly:

"APIError: litellm.APIError: LiteLLM: provider returned a response with no 'choices'. ..." infra: True retryable_text: True
'IndexError: list index out of range' infra: False retryable_text: False

Failure scenario: a provider sends an empty-choices 200 during a board run. Under 1.88.6 the household was recorded as a model failure and scored as a miss. Under 1.95.0 the model's run stops at that household, and resume retries it. 1.95.0's handling is arguably the better one, and git grep "IndexError" finds no committed result rows with that error, so nothing published moves. But the description tells the owner the only effect is the error text, and that is wrong on a path that decides what gets scored.

Replacement text for that bullet:

When an OpenAI-compatible chat provider (xAI, DeepSeek, OpenRouter) returns a 200 response with "choices": [], 1.88.6 returned the empty list, so PolicyBench's response.choices[0] raised IndexError. 1.95.0 raises APIError(status_code=500, "LiteLLM: provider returned a response with no 'choices'...") (litellm_core_utils/llm_response_utils/convert_dict_to_response.py:630). This changes how the board handles the row. Under 1.88.6, run_no_tools_eval recorded the household as a failed row with an IndexError error, which is_infrastructure_error_text counts as a model miss. Under 1.95.0, _is_retryable_provider_error accepts the APIError, so the run checkpoints and raises for that model (eval_no_tools.py:3557), and resume retries the household. No committed result row carries the IndexError error.

Non-blocking

2. Token accounting: the grounding exemption probably never applies on the web arms, and completion tokens change too

The description and the new comment at scripts/tool_pilot_argon.py:438-442 say 1.95.0 counts toolUsePromptTokenCount in prompt tokens "when a response carries no search grounding". That rule matches the code. 1.95.0 checks _response_has_search_grounding, which reads candidate["groundingMetadata"].webSearchQueries, at vertex_and_google_ai_studio_gemini.py:1748-1769 and :1925-1937. Two things are missing.

(a) The pilot's own docstring at scripts/tool_pilot_argon.py:1650-1652 says "litellm's grounding metadata is empty on function-calling turns (live probe, 2026-10-20)". litellm's grounding metadata is candidate["groundingMetadata"] (:2078-2082, :2126-2128), the same field the exemption reads. Every web-arm turn declares functions. If that probe holds, a search turn never counts as grounded, so under 1.95.0 any toolUsePromptTokenCount Gemini reports on it gets priced at the input rate. The pilot also bills the search per query (search_cost_usd). litellm's own docstring (:1755-1757) says Google does not bill grounding-with-search tokens as input. So on the arm that matters, 1.95.0 probably overstates cost. The author's capture put groundingMetadata on the search turn, which the pilot's probe says doesn't happen, so it showed turn 0 unchanged.

I re-ran capture_gemini_web.py both with no grounding on turn 0 (/tmp/caps/capture_gemini_web_nogrounding.py):

88 [(1000, 0.0035), (1000, 0.0035), (1000, 0.0035), (1000, 0.0035)]
95 [(1150, 0.0038), (1150, 0.0038), (1150, 0.0038), (1150, 0.0038)]

The pairs are (prompt_tokens, total_cost_usd) per turn, and every 1.95.0 turn rises, including the search turn. UNVERIFIED: whether live Gemini 4 Argon reports toolUsePromptTokenCount on function-calling search turns. A single live web+pe turn under 1.95.0 would settle it, comparing usageMetadata with prompt_tokens.

(b) is_candidate_token_count_inclusive (:1729-1744) now adds toolUsePromptTokenCount to the left-hand side. When Gemini reports candidatesTokenCount with thoughts included and toolUsePromptTokenCount > 0, 1.88.6 added the thoughts a second time and 1.95.0 doesn't. I called VertexGeminiConfig._calculate_usage directly (/tmp/caps/usage.py) with prompt 1000, candidates 150, thoughts 50, toolUse 150, total 1300:

1.95.0 nogrnd thoughts_in_candidates_tooluse_nogrounding prompt 1150 completion 150 total 1300
1.88.6 nogrnd thoughts_in_candidates_tooluse_nogrounding prompt 1000 completion 200 total 1300

The same completion change shows with grounding present. With thoughts reported separately (candidates 100), completion stays 150 under both versions.

Nothing published depends on this. The published tool-pilot note (app/src/notes/2026-10-20-…json, notes/data/tool_pilot_20261015.meta.json) is built from 1.88.x records, and _versions() (scripts/tool_pilot_argon.py:2369-2376) records the litellm version per run. The board never sends Gemini server-side tools, so toolUsePromptTokenCount should not appear on board responses (UNVERIFIED live).

Suggested replacement for the description's token bullet:

1.95.0 adds toolUsePromptTokenCount to prompt_tokens unless the response carries search grounding (groundingMetadata.webSearchQueries). The pilot's live probe (docstring of bill_searches) found no grounding metadata on function-calling turns, so on the web arms this exemption likely never applies, and a search turn's tool-use tokens would be priced at the input rate on top of the per-query search fee. 1.95.0 also stops double-counting thoughts in completion_tokens when candidatesTokenCount already includes them and toolUsePromptTokenCount is present. A 1.95.0 web-arm run's token costs are not comparable to a 1.88.x run's. Each run's manifest records its litellm version.

The comment at scripts/tool_pilot_argon.py:440-442 could add "(the pilot's function-calling turns carry none, per the live probe noted in bill_searches)".

3. On the locked litellm, nothing tests the patch's body

The description says this itself, and I confirmed it. On 1.95.0 the patch is inert for the web arms, because _apply_include_server_side_tool_invocations sets the flag first. So CI, which runs only 1.95.0, can't detect a broken patch. Mutant M3 below survives. That is fine while 1.95.0 is the only release in use. But GEMINI_SEARCH_PATCH_LITELLM still accepts 1.88.1 and 1.88.6, and on those the patch is the only thing keeping search. A later edit to the patch would pass CI and break a 1.88.x run. Options: drop 1.88.x from the gate now that the lock is 1.95.0, or say in the docstring that only a 1.88.x run exercises the patch. UNPATCHED_GEMINI_TOOLS and RESENT_SEARCH_SIGNATURES would need matching edits if you drop 1.88.x.

4. litellm 1.95.0 ships no macOS wheel

1.88.6 had a py3-none-any wheel. 1.95.0 is built with maturin (a Rust extension, litellm/rust_bridge/_native*.so), and PyPI lists only manylinux and win_amd64 wheels for cp310–cp314, plus the sdist:

curl -s https://pypi.org/pypi/litellm/1.95.0/json | ... → litellm-1.95.0-cp3{10..14}-cp3*-{manylinux_2_28_aarch64,manylinux_2_28_x86_64,win_amd64}.whl, litellm-1.95.0.tar.gz

The repository's docs show the owner works under /Users/maxghenis/..., which is macOS, where uv sync --locked will build litellm from the sdist. The sdist's pyproject.toml declares build-backend = "maturin". On Linux, uv pip install --no-binary litellm 'litellm==1.95.0', with ~/.cargo off PATH, still built and installed (Building litellm==1.95.0 → wheel tag cp312-cp312-linux_x86_64), so this probably isn't a blocker. I did not check how the toolchain was obtained, and I couldn't test macOS. The native bridge is opt-in at runtime (llm_http_handler.py:2278 returns early unless rust=True or a Rust env var is set), so request behaviour shouldn't depend on it. Worth one line in the description: "1.95.0 has no macOS wheel; uv sync on macOS builds it from source (maturin/Rust)."

5. The test comment cites a different Google page than the description

tests/test_tool_pilot_argon.py:2788-2791 says 1.95.0 returns each signature "as ai.google.dev/gemini-api/docs/thought-signatures asks". The description cites ai.google.dev/gemini-api/docs/tool-combination. UNVERIFIED: I didn't fetch either page. Cite the page that actually says it.

Checked and sound

• Patch certification. ast.dump of _resolve_search_tool_conflict and _drop_search_tools_mixed_with_functions in VertexGeminiConfig is equal between 1.88.6 and 1.95.0 (script astcmp.py). The only change to map_openai_params (per ast.unparse diff) is the added first line self._apply_include_server_side_tool_invocations(non_default_params, optional_params). That method (:1068-1085) sets optional_params["include_server_side_tool_invocations"] = True when either spelling of the flag is True. 1.95.0 adds one new caller of _drop_search_tools_mixed_with_functions, llms/gemini/common_utils.py:239 (map_gemini_image_tools_params), which only the image-generation transformations reach, not the pilot. The comment and docstring claims at scripts/tool_pilot_argon.py:436-442 and :1240-1243 match.
• Patched vs unpatched tools on 1.95.0. These match the description's table (test_the_gemini_patch_keeps_search_beside_functions passes on both versions, and mutant M2 is killed).
• thoughtSignature. In the 1.95.0 source, _extract_server_side_tool_invocations (:1461-1518) stores the response part's signature as response_thought_signature, and transformation.py:948-962 re-sends it on the toolResponse part. 1.88.6 (:1608-1611) merged it into thought_signature only when the call had none. I re-ran capture_gemini_web.py under both versions:

  both:      req1-3 .contents[1].parts[2].thoughtSignature  SIG_CALL_1 -> SIG_RESP_1
  resp_only: req1-3 .contents[1].parts[1].thoughtSignature  SIG_RESP_1 -> None
  

  No other body differences, and turn 0 sends [['function_declarations'], ['googleSearch']] with includeServerSideToolInvocations: True. capture_pilot_arms.py reproduces the author's arms-compare.txt line for line: 11 cases, 10 with 0 body diffs, and Gemini web+pe req1.contents[1].parts[4].thoughtSignature G_SIG_CALL -> G_SIG_RESP.
• The pilot doesn't depend on the signature change. _has_thought_signature (scripts/tool_pilot_argon.py:1102-1110) reads only tool-call thought_signature and message thought_signatures, and has_thought_signature per turn is True under both versions in every capture. bill_searches reads only tool_type, id and args from server_side_tool_invocations (:1712-1729). _message_dict (:1092-1099) passes the dict through, so a 1.95.0 transcript's invocations gain a response_thought_signature key, and git grep finds nothing in policybench/, app/src or notes/ that reads it.
• Deferred Claude search result. I re-ran no_workaround.py. With the pilot's _attach_deferred_results, A_ENC_1 shows up in requests [False, False, True, True] under both versions. Without it, [False, False, False, False] under both. The docstring change at :1479-1483 is accurate.
• Tests on the scratch merge. tests/test_tool_pilot_argon.py gave 150 passed, 1 skipped. The 9 tests matching gemini or checked_litellm or signatures pass on 1.88.6 too. Count only real request timeouts in the supervisor's concurrency backoff #183's and Probe card-less models under the card the onboarding gauntlet derives #189's test files (test_request_timeout_signature.py, test_supervisor.py, test_model_cards.py, test_onboard.py) gave 302 passed. CI at the head is lint, test, app, paper and the Vercel preview, all success.
• Hermeticity of the new end-to-end test. On Python 3.12.3 with GEMINI_API_KEY unset and HTTP(S)_PROXY=http://127.0.0.1:9, so network is blocked, the e2e test, the patch test and a scratch test that runs after them all passed. The scratch test asserted VertexGeminiConfig.__dict__ still holds litellm's own two methods, httpx.Client.send is restored, and GEMINI_API_KEY is unset. Three repeat runs passed.
• Mutants (on 1.95.0 unless noted, each against the 9 tests above):
  • M1, giving 1.95.0 the 1.88.x signatures: killed (1 failed).
  • M2, giving 1.95.0 the 1.88.x unpatched tools: killed.
  • M3, making the patch body a no-op: survives on 1.95.0 (finding 3), killed on 1.88.6 (2 failed).
  • M4, removing the version gate: killed by test_the_gemini_patch_refuses_another_litellm.
  • M5, dropping include_server_side_tool_invocations from the web arm's base kwargs: killed (2 failed).
  • M7, stripping provider_specific_fields from assistant messages: killed.
  • M8, dropping only server_side_tool_invocations: killed.
  • M9, on 1.88.6, the patch replacing only _resolve_search_tool_conflict: killed (2 failed).
• Merge. git show --remerge-diff --format= d9f99ab is empty, and git diff d9f99ab^2 d9f99ab --stat lists only the PR's 5 files, so there are no hand edits. Under both versions, log_shows_request_timeout matches the printed litellm.Timeout for OpenAI, Anthropic and Gemini on an httpx.ReadTimeout. It returns False for the import log when the cost-map fetch is refused ([Errno 111]) and when it times out, which I produced with a local socket that accepts and never replies (LiteLLM: Failed to fetch remote model cost map from http://127.0.0.1:8977/map.json: timed out. Falling back to local backup.).
• Board path spot checks.
  • Applied timeouts: litellm.completion(..., timeout=_request_timeout_seconds(mid)) for all 52 MODELS gave identical request.extensions["timeout"] under both versions (tmo.py, identical: True).
  • Cache keys: Cache(type="disk").get_cache_key(...) gave the same key for a Gemini chat request and a Responses-style request under both versions.
  • Empty choices: the litellm behaviour is as described, but see finding 1.
  • Error classification: _should_retry returns True for both the 1.88.6 and 1.95.0 error.
• Advisory, checked in the installed 1.88.6. proxy/auth/auth_utils.py:290 lists "vertex_ai_credentials". litellm_core_utils/url_utils.py:151 defines provider_url_destination_candidates, and proxy/litellm_pre_call_utils.py:24,232 uses it. llms/huggingface/embedding/transformation.py has no get_api_base, and oobabooga/chat/oobabooga.py:37,99 and huggingface/embedding/handler.py:351 match with startswith(("http://", "https://")). git grep "litellm.proxy\|litellm\[proxy\]" over policybench/, scripts/ and pyproject.toml finds nothing.
• Lock. The uv.lock diff outside litellm's sdist and wheel URLs is only version = "1.88.6" → "1.95.0", and uv lock --check passes under uv 0.11.32. litellm's non-extra requirements match except that 1.95.0 lowers the aiohttp floor from >=3.14.2 to >=3.10. That has no effect, since the lock pins aiohttp.

What I couldn't check

• Live Gemini usage on web-arm turns (finding 2). No provider calls were allowed.
• macOS install of the 1.95.0 sdist (finding 4).
• The full 635-request board capture, and the HTTP-status exception-mapping matrix. I spot-checked timeouts, cache keys, empty choices and litellm.Timeout only.
• The advisory's affected-version range and the claim that no 1.89–1.94 release clears it. I didn't read the GHSA.
• The two Google doc pages cited for per-part signatures (finding 5).
• The OpenRouter/DeepSeek reasoning_effort rewrites, the Responses max_output_tokens floor, and the aiohttp mid-body ReadError. I didn't test these.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

This is an independent review by a Claude Opus 5.5 cloud session for the PolicyBench owner session.

Verdict: APPROVE at dc43c517f09a5daf3f2a5e7dc53beb173c479b66.

The PR head on GitHub is still dc43c517. The head resolves round 1's blocking finding, and its new text holds up against the code and my runs. I have three non-blocking text findings: one misnamed function in a citation, one overstated scope ("stops that model"), and one round 1 point the description leaves out. CI at the head passes: lint, test, app, paper and the Vercel preview.

Setup: a clone at the head, and uv sync --locked --extra dev gave litellm 1.95.0 on Python 3.13.18. For 1.88.6 I used uv run --with 'litellm[caching]==1.88.6'. Tests ran on a scratch merge of the head into origin/main (4e5a8459). This session can't reach ai.google.dev (getaddrinfo ENOTFOUND), so every claim about Google's docs is UNVERIFIED.

Blocking

None.

Non-blocking

1. The probe the comment and description cite is in turn_searches, not bill_searches

scripts/tool_pilot_argon.py:442-443 says "The pilot's function-calling turns carry no grounding metadata (see bill_searches)". The description says "(docstring of bill_searches)". bill_searches (:1921-1922) has a one-line docstring: "Log the response's searches and add their price to the spend." The live probe is in turn_searches's docstring at :1656-1658: "litellm's grounding metadata is empty on function-calling turns (live probe, 2026-10-20)". bill_searches calls turn_searches, so a reader can find it in one hop, but the name is wrong. Round 1 made the same mistake, which is probably where this came from.

$ grep -n "live probe\|grounding metadata\|def bill_searches\|def turn_searches" scripts/tool_pilot_argon.py
442:# grounding. The pilot's function-calling turns carry no grounding metadata
1640:def turn_searches(
1657:      litellm's grounding metadata is empty on function-calling turns (live
1921:    def bill_searches(response, turn) -> dict:

Replacement: in scripts/tool_pilot_argon.py:443, (see bill_searches) → (see turn_searches). In the description, (docstring of \bill_searches`)→(docstring of `turn_searches`)`.

2. An empty-choices reply ends the whole eval-no-tools call, not just that model

The description says "the run saves a checkpoint and stops that model (policybench/eval_no_tools.py:3597)". At :3597-3600, the branch is _save_checkpoint(...) then raise. The raise leaves the for model_name, model_id in models.items() loop (:3532), so models later in the --models list don't run either. I ran run_no_tools_eval on two TX households with models={"grok-4.6": ..., "grok-4.7": ...}, POLICYBENCH_MAX_ATTEMPTS=1, and httpx.Client.send patched to return HTTP 200 {"choices": []}:

1.95.0 raised APIError litellm.APIError: LiteLLM: provider returned a response with no 'choices'. Raw k
requests by model: ['grok-4.6']
1.88.6 returned 4 rows [['grok-4.6', 'IndexError: list index out of range'], ['grok-4.6', ...], ['grok-4.7', ...], ['grok-4.7', ...]]
requests by model: ['grok-4.6', 'grok-4.6', 'grok-4.7', 'grok-4.7']

On the supervised board path this makes no difference. Supervisor._spawn (policybench/supervisor.py:689-711) starts one eval-no-tools worker per model and household (--model, --scenario-start i --scenario-end i+1). It does matter for a manual eval-no-tools --models a,b call or run_repeated_no_tools_eval.

Replacement for that clause: "so the run saves a checkpoint and raises, which ends that eval-no-tools call, including any models listed after this one (policybench/eval_no_tools.py:3597). A supervised worker runs one model on one household, and a resume retries the household."

3. The description doesn't mention round 1's completion-token change

Round 1's finding 2(b): when candidatesTokenCount already includes thoughts and toolUsePromptTokenCount > 0, 1.88.6 added the thoughts to completion_tokens a second time and 1.95.0 doesn't (is_candidate_token_count_inclusive, vertex_and_google_ai_studio_gemini.py:1734-1744 in the installed 1.95.0, which now includes toolUsePromptTokenCount in the sum). The new token bullet covers only prompt tokens. The bullet's last sentence, "A 1.95.0 web-arm run's token costs don't compare with a 1.88.x run's", already covers this in effect, so this is optional. Text to add before that sentence: "1.95.0 also stops counting thoughts twice in completion_tokens when candidatesTokenCount already includes them and a response reports tool-use tokens."

Round 1 findings at the head

1. Empty choices (blocking in round 1): resolved. Each part of the new bullet checks out:
   • convert_dict_to_response.py:631 in the installed 1.95.0 is if not response_object.get("choices") or not isinstance(response_object["choices"], Iterable):, followed by raise APIError(status_code=500, message=("LiteLLM: provider returned a response with no 'choices'. " ....
   • 1.88.6's same function (:539-541) has assert response_object["choices"] is not None and isinstance(response_object["choices"], Iterable), so an empty list passes and PolicyBench's choices[0] raises IndexError. The run above shows the IndexError rows. The description's "rejected only a None choices" leaves out the non-iterable check, which doesn't matter here.
   • policybench/eval_no_tools.py:3597 is if _is_retryable_provider_error(e):.
   • Classification, called directly at the head on 1.95.0:

     APIError retryable: True
     InternalServerError retryable: True
     IndexError retryable: False
     IndexError row infra: False 'IndexError: list index out of range'
     

     (APIError(status_code=500, "LiteLLM: provider returned a response with no 'choices'..."), litellm.InternalServerError, IndexError("list index out of range"); the last line is is_infrastructure_error_text(_format_error(IndexError(...))).)
   • "No committed result row carries an IndexError error": git grep -l IndexError at the head finds only four tests/test_finish_*.py files and a reference-audit note about a PolicyEngine trace (reference_audit/2026-09-22/verification/v3_r08_r09_r11_r12.md:153), and zcat of all 13 tracked .gz files finds none.
   • The scope wording is finding 2 above.
2. Token accounting: resolved in substance, with the citation in finding 1 and the omission in finding 3. In the installed 1.95.0, _calculate_usage sets billable_tool_use_prompt_tokens = 0 if VertexGeminiConfig._response_has_search_grounding(completion_response) else (tool_use_prompt_tokens or 0) and adds it to prompt_tokens (vertex_and_google_ai_studio_gemini.py:1925-1937). _response_has_search_grounding (:1748-1769) returns True only when a candidate's grounding metadata has web search queries. The turn_searches probe says that metadata is empty on function-calling turns, so the new comment at scripts/tool_pilot_argon.py:442-447 and the description's "adds the tool-use tokens on every turn, search turns included" follow from the code, given that probe. litellm's own docstring (:1755-1757) says "Google bills grounding-with-Google-Search retrieved tokens separately (a per-request / per-query search fee) and excludes them from input token billing". That matches what the comment and description attribute to Google's page, but I couldn't read the page itself (UNVERIFIED). The 1,000 → 1,150 canned figure matches round 1's rerun. _versions() records litellm among its packages (scripts/tool_pilot_argon.py:2377).
3. Untested patch body: resolved by the docstring. scripts/tool_pilot_argon.py:1247-1249 now says CI can't catch a broken patch body and only a 1.88.x test run exercises it. CI runs uv sync --locked --extra dev then uv run pytest -m "not slow" (.github/workflows/ci.yml:46-47), so it runs 1.95.0. Mutant: I replaced the two VertexGeminiConfig assignments at the end of allow_gemini_search_with_functions with return None and ran pytest tests/test_tool_pilot_argon.py -k "gemini or checked_litellm or signatures":

   1.95.0:            9 passed, 142 deselected   (survives)
   1.88.6:            2 failed, 7 passed         (killed)
   1.88.6 unmutated:  9 passed
   

   The sentence is accurate. The gate still accepts 1.88.x, which round 1 flagged as an option. Keeping it is the owner's call.
4. macOS wheel: resolved, and the PyPI claims hold. From https://pypi.org/pypi/litellm/json:

   1.95.0 files 16 mac 0 ['manylinux_2_28_aarch64', 'manylinux_2_28_x86_64', 'win_amd64']
   1.95.1 files 36 mac 10 ['macosx_10_12_x86_64', 'macosx_11_0_arm64', 'manylinux_2_28_aarch64', 'manylinux_2_28_x86_64', 'musllinux_1_2_aarch64', 'musllinux_1_2_x86_64']
   

   Every release from 1.95.1 through 1.106.6 (63 releases) lists macosx_10_12_x86_64 and macosx_11_0_arm64 wheels. The 16 files for 1.95.0 are 15 wheels (cp310–cp314 × three platforms) plus the sdist. The installed 1.95.0 contains litellm/rust_bridge/_native.cpython-313-x86_64-linux-gnu.so, so "includes a Rust extension" holds. UNVERIFIED: "On a Mac with Python 3.14 it built cp314-cp314-macosx_11_0_arm64". I can't run macOS here. The owner's own uv sync log on the Mac would settle it.
5. Test comment citation: resolved as text. tests/test_tool_pilot_argon.py:2791 now cites ai.google.dev/gemini-api/docs/tool-combination, the same page as the description. The commit message says the thought-signatures page covers only the Interactions API. UNVERIFIED: whether the tool-combination page says each built-in tool call and result carries its own signature and asks the client to send each back. Reading that page (or allowing ai.google.dev in this environment) would settle it.

Checked and sound

• Head. git rev-parse HEAD → dc43c517f09a5daf3f2a5e7dc53beb173c479b66, which is also the PR's head.sha on GitHub.
• Merge 081727b9. git show --remerge-diff --format= 081727b9 | wc -c → 0. Its parents are d9f99ab8 and 4e5a8459 (origin/main). Outside uv.lock, git diff 081727b9^1 081727b9 is byte-identical to git diff $(git merge-base d9f99ab 4e5a845) 4e5a845, which covers Pin the reference adversary's leaderboard impact to the pass's inputs #205, Record PolicyEngine provenance without importing policyengine #186, Merge batch repair rounds cell by cell, like sync #180 and Patch pyasn1, tornado and filelock in uv.lock (16 Dependabot alerts) #251. The +/- lines of the uv.lock part are identical to main's own lock change (filelock 3.24.3 → 4.2.6, pyasn1 0.6.4 → 0.6.5, tornado 6.5.10 → 6.6).
• Lock against main. git diff origin/main dc43c517 -- uv.lock has two hunks, both inside the litellm package entry: version = "1.88.6" → "1.95.0", and the sdist and wheel URLs. uv lock --check passes under uv 0.11.32, 0.13.4 and 0.13.6 (uvx --from uv==0.13.x uv lock --check → Resolved 137 packages), and uv sync --locked --extra dev succeeds.
• dc43c517 changes no code. It touches scripts/tool_pilot_argon.py and tests/test_tool_pilot_argon.py. With every module, class and function docstring blanked, ast.dump before (081727b9) and after is equal for both files. Without blanking, the test file is still equal, and the script differs only in the allow_gemini_search_with_functions docstring:

  scripts/tool_pilot_argon.py blanked-equal: True raw-equal: False
  tests/test_tool_pilot_argon.py blanked-equal: True raw-equal: True
  

• Focused tests on the scratch merge (8f2cb66, origin/main + dc43c517): uv run pytest tests/test_tool_pilot_argon.py -q → 150 passed, 1 skipped. On 1.88.6, the 9 tests matching gemini or checked_litellm or signatures pass.
• Other new and edited description text I checked: the Verification lines about 081727b and dc43c51 (above), the "Tests" bullet's three mutants (the 1.88.6 no-patch mutant is the one above, and it's killed), and the Invariants line about uv versions and the lock header (the lock now differs from main only in litellm, so the revision = 3 header and pexpect marker match main's).

CI at the head

gh api repos/PolicyEngine/policybench/commits/dc43c517.../check-runs:

Vercel Preview Comments	completed	success
test	completed	success
app	completed	success
lint	completed	success
paper	completed	success

The test job (run 41509798511) started at 2026-11-02T19:27:00Z and completed at 19:45:49Z, so it took 18m49s against its 30-minute limit (.github/workflows/ci.yml:35). Its log ends 7010 passed, 90 skipped, 38 deselected, 3 warnings in 1085.62s (0:18:05). For comparison, the test job at d9f99ab ran 17:10:16Z–17:33:44Z, which is 23m28s.

What I couldn't check

• ai.google.dev/gemini-api/docs/tool-combination: the network policy blocks the host. That leaves both Google-page claims (no charge for Google Search tool tokens, and a per-part signature that a client sends back) UNVERIFIED.
• Building the 1.95.0 sdist on macOS.
• Whether live Gemini reports toolUsePromptTokenCount on the pilot's function-calling search turns. No provider calls were allowed.
• The full test suite locally. CI is the evidence for it, and it passes (above).

---

Generated by Claude Code

[MaxGhenis]

The round 2 reviewer (#252 (comment)) approved dc43c51 with three non-blocking findings. We applied the first in 77ebb5e, a one-word comment change (bill_searches → turn_searches) that leaves the syntax tree of scripts/tool_pilot_argon.py unchanged, and the other two in the description. Lint, test (7,010 passed, 90 skipped), app, paper and the Vercel preview pass at 77ebb5e, and we squash-merged that head as fe6797a.