Skip to content

Fix ClickHouse lookup pagination for UNION queries - #958

Open
CheekyNox wants to merge 1 commit into
PlayPro:masterfrom
CheekyNox:fix/clickhouse-union-pagination
Open

Fix ClickHouse lookup pagination for UNION queries#958
CheekyNox wants to merge 1 commit into
PlayPro:masterfrom
CheekyNox:fix/clickhouse-union-pagination

Conversation

@CheekyNox

@CheekyNox CheekyNox commented Aug 20, 2026

Copy link
Copy Markdown

Summary

  • Paginate ClickHouse union lookups globally instead of limiting only one union branch.
  • Use a two-phase lookup so deep pages select lightweight row keys before loading binary payload columns.
  • Stop processing cleanly when a lookup query fails instead of triggering a secondary null-pointer exception.

Problem

ClickHouse lookups that combine multiple event families append pagination directly after a multi-branch UNION ALL. The trailing LIMIT does not constrain the complete union, so a lookup configured for a few lines can return thousands of rows and flood the Minecraft client chat.

Applying an outer ORDER BY ... LIMIT ... OFFSET ... directly to the full rows fixes the result count but makes deep pages materialize large item/container payload expressions before pagination. On large histories this can exceed the ClickHouse memory limit.

Fix

For paginated ClickHouse detail lookups:

  1. Query only tbl, id, and time across the candidate sources and apply global ordering/pagination.
  2. Load complete rows only for the selected source/row IDs.

This keeps pagination global while avoiding materialization of binary payloads for discarded rows. Non-ClickHouse lookup paths are unchanged.

Verification

  • mvn verify (BUILD SUCCESS; tests are skipped by the upstream Maven configuration)
  • Reproduced the original excessive result output with a multi-family ClickHouse lookup
  • Verified the initial global-limit fix caps output to the requested page size

@CheekyNox
CheekyNox force-pushed the fix/clickhouse-union-pagination branch from 4cd6b56 to b236f2f Compare August 21, 2026 04:14
@CheekyNox
CheekyNox force-pushed the fix/clickhouse-union-pagination branch from b236f2f to 00bba76 Compare August 21, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant