Run agent actions and workspace search together instead of one or the other - #1333
Merged
Paul Lizer (paullizer) merged 3 commits intoAug 21, 2026
Merged
Conversation
… other Selecting an agent with actions and enabling a workspace produced answers that never invoked any action and reported numbers absent from the cited spreadsheet. Three independent defects combined to cause this. The retrieval augmentation prompt instructed the model to base its answer only on the retrieved excerpts. Agent actions were attached and available -- agents are built with FunctionChoiceBehavior.Auto() and nothing disables tools when documents are in scope -- but the model was told not to reach for them. Excerpts are now framed as starting evidence, and the model is directed to call an available action when they lack what the question needs, then reason over both. The no-fabrication rule is preserved and strengthened. should_run_tabular_evidence() suppressed computation whenever any narrative source was in scope, and treated topic words such as "report" and "policy" as evidence-type signals. Because only a truncated three-row preview of a spreadsheet is indexed for search, skipping computation left the model deriving totals from those preview rows. Tabular sources in scope are now computed unless the question unambiguously names a narrative artifact, restoring parity with the legacy path used when mixed-source search is disabled. A skipped tabular source previously reported that processing "was not needed", implying irrelevance. It now states the full table was never read, that indexed excerpts are a truncated preview, and that the tabular analysis action should be called if values are required. No new setting is introduced; these are correctness fixes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Development shipped v0.260.023 (semantic kernel startup request context) and v0.260.024 (inline media cited-only gating) while this branch was open, so both the version constant and the release notes collided. Resolution: - config.py takes 0.260.025, since 0.260.023 and 0.260.024 are both taken. - release_notes.md keeps Development's v0.260.024 and v0.260.023 sections and moves this branch's three entries into a new v0.260.025 section at the top. - Renumbered the version references in the fix documentation and in the two affected functional test headers. - Registered the fix page in docs/explanation/fixes/index.md, which Development added while this branch was open. - Regenerated docs/explanation/release-notes/ with scripts/build_release_notes_pages.py, since those pages are generated from release_notes.md. Refs #1332 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Selecting an agent that has actions and enabling a workspace produced answers that never invoked any action and reported numbers that were not in the cited spreadsheet. The turn behaved as "retrieval or actions" instead of "retrieval and actions".
Three independent defects combined to cause this.
1. Agent actions were never attempted.
build_search_augmentation_system_prompt()instructed the model to "base your answer only on information supported by the retrieved excerpts". The actions were attached the whole time — agents are built withFunctionChoiceBehavior.Auto()and nothing in the code disables tools when documents are in scope. The model simply obeyed the instruction not to look anywhere else.2. The spreadsheet was never computed.
should_run_tabular_evidence()ended with a blanketif has_narrative_sources: return False, so a single PDF in the relevance results suppressed computation over an authorized spreadsheet. It also treated topic words (report,policy,memo,contract) as evidence-type signals, so it misfired frequently.3. That is why the numbers were wrong. Only a truncated three-row schema preview of a spreadsheet is indexed for search — intentionally, since the tabular engine reads the full file from blob storage. With computation skipped, the model derived totals and averages from those three preview rows. The indexed chunk even ends with "available for detailed analysis via the Tabular Processing plugin", so the model read the advertisement for the tool while being told not to use it.
Worth noting: when
enable_mixed_source_chat_searchis disabled, the legacy path computes workspace tabular sources unconditionally. The mixed-source path had regressed that; this restores parity.Changes
functions_mixed_source_orchestration.pypdf,docx,presentation,paragraph,section, ...).route_backend_chats.pybuild_search_augmentation_system_prompt(). Excerpts are now framed as starting evidence; the model is directed to call an available action when they lack what the question needs, then reason over excerpts and action results together. The no-fabrication rule is preserved and strengthened, and deriving any numeric conclusion from tabular preview rows is explicitly forbidden.config.py— version0.260.024→0.260.025.No new setting is introduced. These are correctness fixes and apply unconditionally.
Rebased onto Development
Development shipped v0.260.023 (semantic kernel startup request context) and v0.260.024 (inline media cited-only gating) while this branch was open, which collided on the version constant and the release notes. Resolved by:
0.260.025, since0.260.023and0.260.024are both claimed.v0.260.025section.docs/explanation/fixes/index.md, which Development added while this branch was open.docs/explanation/release-notes/withscripts/build_release_notes_pages.py, since those pages are generated fromrelease_notes.md.Validation
test_agent_actions_with_workspace_evidence.py(new)test_mixed_source_manifest_contracts.pytest_tabular_computed_results_prompt_priority.pytest_semantic_kernel_startup_without_request_context.py(Development's)test_docs_app_surface_coverage.pytest_docs_site_quality.pypy_compileon modified modulesAll re-run after the merge.
The new regression test asserts that a quantitative question with narrative sources present now computes the tabular source, that an unambiguous narrative-artifact question still skips it, that topic words no longer suppress computation, that the skipped envelope warns against preview-row math, and that both prompts permit action invocation.
Both existing prompt contracts asserted by
test_tabular_computed_results_prompt_priority.pyare preserved verbatim.One assertion in
test_mixed_source_chat_search_consistency.pywas updated because it encoded the old behavior being fixed — a generic question skipping computation of an in-scope spreadsheet.Pre-existing failures, not introduced here
Seven tests across
test_mixed_source_chat_search_consistency.py,test_mixed_source_hardening.py, andtest_mixed_source_analyze_workflow.pyfail identically before and after this change, at the same assertions. This was verified by stashing the change and re-running. Their harness builds a synthetic namespace for_execute_mixed_source_tabular_evidencethat is missingmaybe_queue_search_tabular_generated_output, so the stubbed tabular runner raises and every source reportsfailed. Flagged as follow-up in the issue rather than fixed here.Documentation
docs/explanation/fixes/AGENT_ACTIONS_WITH_WORKSPACE_EVIDENCE_FIX.md### **(v0.260.025)**indocs/explanation/release_notes.md, with generated pages refreshedFixes #1332
Related to #1021 (turn-level orchestration across chat capabilities), which is the strategic solution to this class of problem — treating selected capabilities as required attempts and reporting skips or failures. This PR is the targeted correctness fix for the concrete symptom and does not close that initiative.