Skip to content

Gate inline image and video galleries to cited media only - #1331

Merged
Paul Lizer (paullizer) merged 1 commit into
Developmentfrom
paullizer-inline-media-citation-gating
Aug 21, 2026
Merged

Gate inline image and video galleries to cited media only#1331
Paul Lizer (paullizer) merged 1 commit into
Developmentfrom
paullizer-inline-media-citation-gating

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Fixes #1329
Refs #1249

Problem

Assistant messages rendered inline image and video galleries for every media document returned by retrieval, not just the media the response actually cited. A workspace search that surfaced five image files produced five inline gallery tiles even when the answer referenced only one of them, or none at all.

Because the galleries sit directly inside the message bubble, unrelated media was presented as though it supported the answer. The five-item gallery cap could be consumed entirely by retrieval noise, pushing genuinely cited media out of view, and each unreferenced workspace file triggered an additional enhanced-citation fetch.

Root cause

renderInlineImageGalleries and renderInlineVideoGalleries received the full hybrid_citations and web_search_citations arrays from appendMessage, then selected workspace media purely by file extension via extractWorkspaceCitationImageItems / extractWorkspaceCitationVideoItems.

#1249 already built the retrieved-vs-cited separation: every assistant message persists cited_hybrid_citations and cited_web_search_citations, and those subsets already reach the browser on all delivery paths. The inline gallery renderers were simply never switched over — no frontend module read the cited subsets at all.

Change

Frontend only. No backend change was required.

  • New application/single_app/static/js/chat/chat-citation-tracking.js — the browser mirror of functions_citation_tracking._message_has_citation_tracking(). Exports messageHasCitationTracking, getCitedHybridCitations, getCitedWebCitations. A message counts as tracked when citation_tracking_version >= 1 or either cited_* key is present; each getter normalizes non-array values to an empty list.
  • appendMessage derives citedHybridCitations / citedWebCitations from the assistant message object and passes those to both gallery renderers.
  • Renamed the gallery entry-point and extraction helper parameters to the cited names so the narrowed contract is explicit at the call boundary.
  • Corrected the "Linked images" / "Linked videos" summaries, which described the links as merely returned with the response rather than cited by it.

The Sources disclosure, its count badges, and the metadata drawer still receive the complete retrieved arrays, so nothing became harder to find.

Deliberate scope boundaries

  • Agent and tool galleries still render. An action that returns an image or video gallery is an executed tool result, not an unused retrieval candidate — matching how [P1] Distinguish retrieved sources from cited documents #1249 treats agent records.
  • Legacy messages keep prior behavior. Messages saved before citation tracking existed carry no cited arrays and fall back to the full retrieved set, matching the no-migration legacy fallback in get_message_reference_citation_buckets() and [P1] Distinguish retrieved sources from cited documents #1249's decision to avoid read-time history parsing.
  • A tracked response that cited nothing renders no workspace or linked gallery. Both renderers already collapse .inline-visualizations-container with d-none when it has no children, so an empty result leaves no visual gap.

Behavior

Scenario Before After
Tracked response, 1 of 5 retrieved images cited 5 inline tiles 1 inline tile
Tracked response, no documents cited Every retrieved image tiled No workspace gallery
Tracked response, no documents cited, media action ran Retrieved images plus action gallery Action gallery only
Legacy untracked response Every retrieved image tiled Unchanged
Sources disclosure Complete retrieved set Unchanged

Validation

Check Result
functional_tests/test_inline_media_cited_only_gating.py (new) 5/5 passed
functional_tests/test_chat_cited_source_tracking.py 17/17 passed
functional_tests/test_docs_site_quality.py 6/6 passed
functional_tests/test_docs_app_surface_coverage.py 7/7 passed
ui_tests/test_chat_inline_*_gallery_rendering.py 4 collected, skip without a live authenticated instance

The new functional test executes the real chat-citation-tracking.js module under Node across tracked, untracked, empty-cited, key-presence-only, missing-message, and malformed-value inputs, so the fallback rules are covered behaviorally rather than by source assertion alone.

Both Playwright gallery tests gained a gating regression covering three messages on one page: a tracked response that cited one of two retrieved media files, a tracked response that cited nothing but ran a media action, and a legacy untracked response.

Note

functional_tests/test_inline_image_gallery_visualization.py and its video counterpart cannot execute locally because they import functions_workflow_runner, which pulls in config.py and constructs a live Cosmos client at import time. That is a pre-existing environment constraint unrelated to this change; their updated source assertions were verified directly against the modified files.

Documentation

  • docs/explanation/fixes/INLINE_MEDIA_CITED_ONLY_GATING_FIX.md
  • Release notes entry under ### **(v0.260.024)**, with the generated Jekyll release-note pages regenerated via scripts/build_release_notes_pages.py

Version bumped to 0.260.024. This branch was rebased onto Development after #1330 landed and claimed 0.260.023.

Inline galleries were built from the full retrieved citation arrays, so
every workspace or web media file returned by search became a tile inside
the message bubble even when the response never referenced it. That
presented unrelated media as though it supported the answer, consumed the
five-item gallery cap with retrieval noise, and issued enhanced-citation
fetches for documents that were never cited.

Issue #1249 already persists the exact cited subsets on each assistant
message and delivers them to the browser on every path, but no frontend
module read them. Add chat-citation-tracking.js as the browser mirror of
_message_has_citation_tracking, and feed the cited subsets to both gallery
renderers from appendMessage. The Sources disclosure keeps the complete
retrieved set.

Agent and tool galleries stay ungated because they are executed results,
and legacy messages without tracking keep prior behavior rather than being
parsed at read time.

Fixes #1329
Refs #1249

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) force-pushed the paullizer-inline-media-citation-gating branch from c2fc062 to f9b8179 Compare August 21, 2026 00:36
@paullizer
Paul Lizer (paullizer) merged commit a8a3fbf into Development Aug 21, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant