Skip to content

Run agent actions and workspace search together instead of one or the other - #1333

Merged
Paul Lizer (paullizer) merged 3 commits into
Developmentfrom
paullizer-agent-actions-vs-workspace-search
Aug 21, 2026
Merged

Run agent actions and workspace search together instead of one or the other#1333
Paul Lizer (paullizer) merged 3 commits into
Developmentfrom
paullizer-agent-actions-vs-workspace-search

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Selecting an agent that has actions and enabling a workspace produced answers that never invoked any action and reported numbers that were not in the cited spreadsheet. The turn behaved as "retrieval or actions" instead of "retrieval and actions".

Three independent defects combined to cause this.

1. Agent actions were never attempted. build_search_augmentation_system_prompt() instructed the model to "base your answer only on information supported by the retrieved excerpts". The actions were attached the whole time — agents are built with FunctionChoiceBehavior.Auto() and nothing in the code disables tools when documents are in scope. The model simply obeyed the instruction not to look anywhere else.

2. The spreadsheet was never computed. should_run_tabular_evidence() ended with a blanket if has_narrative_sources: return False, so a single PDF in the relevance results suppressed computation over an authorized spreadsheet. It also treated topic words (report, policy, memo, contract) as evidence-type signals, so it misfired frequently.

3. That is why the numbers were wrong. Only a truncated three-row schema preview of a spreadsheet is indexed for search — intentionally, since the tabular engine reads the full file from blob storage. With computation skipped, the model derived totals and averages from those three preview rows. The indexed chunk even ends with "available for detailed analysis via the Tabular Processing plugin", so the model read the advertisement for the tool while being told not to use it.

Worth noting: when enable_mixed_source_chat_search is disabled, the legacy path computes workspace tabular sources unconditionally. The mixed-source path had regressed that; this restores parity.

Changes

functions_mixed_source_orchestration.py

  • Inverted the tabular gate: an in-scope tabular source is computed unless the question unambiguously names a narrative artifact (pdf, docx, presentation, paragraph, section, ...).
  • Narrowed narrative markers to artifact words only, so topic words no longer suppress computation.
  • Rewrote the skipped-source evidence summary. It previously said processing "was not needed", implying irrelevance; it now states the full table was never read, that indexed excerpts are a truncated preview, that numeric conclusions must not be drawn from them, and that the tabular analysis action should be called if values are required.
  • Added action permission and a preview-row guard to the mixed-source evidence handoff.

route_backend_chats.py

  • Rewrote build_search_augmentation_system_prompt(). Excerpts are now framed as starting evidence; the model is directed to call an available action when they lack what the question needs, then reason over excerpts and action results together. The no-fabrication rule is preserved and strengthened, and deriving any numeric conclusion from tabular preview rows is explicitly forbidden.

config.py — version 0.260.0240.260.025.

No new setting is introduced. These are correctness fixes and apply unconditionally.

Rebased onto Development

Development shipped v0.260.023 (semantic kernel startup request context) and v0.260.024 (inline media cited-only gating) while this branch was open, which collided on the version constant and the release notes. Resolved by:

  • Taking 0.260.025, since 0.260.023 and 0.260.024 are both claimed.
  • Keeping both of Development's release-note sections intact and moving this branch's three entries into a new v0.260.025 section.
  • Renumbering the version references in the fix doc and the two affected test headers.
  • Registering the fix page in docs/explanation/fixes/index.md, which Development added while this branch was open.
  • Regenerating docs/explanation/release-notes/ with scripts/build_release_notes_pages.py, since those pages are generated from release_notes.md.

Validation

Check Result
test_agent_actions_with_workspace_evidence.py (new) 4/4 pass
test_mixed_source_manifest_contracts.py pass
test_tabular_computed_results_prompt_priority.py pass
test_semantic_kernel_startup_without_request_context.py (Development's) pass
test_docs_app_surface_coverage.py 7/7 pass
test_docs_site_quality.py 6/6 pass
py_compile on modified modules clean

All re-run after the merge.

The new regression test asserts that a quantitative question with narrative sources present now computes the tabular source, that an unambiguous narrative-artifact question still skips it, that topic words no longer suppress computation, that the skipped envelope warns against preview-row math, and that both prompts permit action invocation.

Both existing prompt contracts asserted by test_tabular_computed_results_prompt_priority.py are preserved verbatim.

One assertion in test_mixed_source_chat_search_consistency.py was updated because it encoded the old behavior being fixed — a generic question skipping computation of an in-scope spreadsheet.

Pre-existing failures, not introduced here

Seven tests across test_mixed_source_chat_search_consistency.py, test_mixed_source_hardening.py, and test_mixed_source_analyze_workflow.py fail identically before and after this change, at the same assertions. This was verified by stashing the change and re-running. Their harness builds a synthetic namespace for _execute_mixed_source_tabular_evidence that is missing maybe_queue_search_tabular_generated_output, so the stubbed tabular runner raises and every source reports failed. Flagged as follow-up in the issue rather than fixed here.

Documentation

  • docs/explanation/fixes/AGENT_ACTIONS_WITH_WORKSPACE_EVIDENCE_FIX.md
  • Three entries under ### **(v0.260.025)** in docs/explanation/release_notes.md, with generated pages refreshed

Fixes #1332

Related to #1021 (turn-level orchestration across chat capabilities), which is the strategic solution to this class of problem — treating selected capabilities as required attempts and reporting skips or failures. This PR is the targeted correctness fix for the concrete symptom and does not close that initiative.

Paul Lizer (paullizer) and others added 3 commits August 20, 2026 20:20
… other

Selecting an agent with actions and enabling a workspace produced answers that
never invoked any action and reported numbers absent from the cited spreadsheet.
Three independent defects combined to cause this.

The retrieval augmentation prompt instructed the model to base its answer only
on the retrieved excerpts. Agent actions were attached and available -- agents
are built with FunctionChoiceBehavior.Auto() and nothing disables tools when
documents are in scope -- but the model was told not to reach for them. Excerpts
are now framed as starting evidence, and the model is directed to call an
available action when they lack what the question needs, then reason over both.
The no-fabrication rule is preserved and strengthened.

should_run_tabular_evidence() suppressed computation whenever any narrative
source was in scope, and treated topic words such as "report" and "policy" as
evidence-type signals. Because only a truncated three-row preview of a
spreadsheet is indexed for search, skipping computation left the model deriving
totals from those preview rows. Tabular sources in scope are now computed unless
the question unambiguously names a narrative artifact, restoring parity with the
legacy path used when mixed-source search is disabled.

A skipped tabular source previously reported that processing "was not needed",
implying irrelevance. It now states the full table was never read, that indexed
excerpts are a truncated preview, and that the tabular analysis action should be
called if values are required.

No new setting is introduced; these are correctness fixes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds the issue reference to the three v0.260.023 release note entries and to the
fix documentation header, and records #1021 as the related strategic initiative
that this targeted fix does not close.

Refs #1332

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Development shipped v0.260.023 (semantic kernel startup request context) and
v0.260.024 (inline media cited-only gating) while this branch was open, so both
the version constant and the release notes collided.

Resolution:
- config.py takes 0.260.025, since 0.260.023 and 0.260.024 are both taken.
- release_notes.md keeps Development's v0.260.024 and v0.260.023 sections and
  moves this branch's three entries into a new v0.260.025 section at the top.
- Renumbered the version references in the fix documentation and in the two
  affected functional test headers.
- Registered the fix page in docs/explanation/fixes/index.md, which Development
  added while this branch was open.
- Regenerated docs/explanation/release-notes/ with
  scripts/build_release_notes_pages.py, since those pages are generated from
  release_notes.md.

Refs #1332

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit 06cbf00 into Development Aug 21, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant