Experiment with ES|QL lookup joins for event and stack queries

[ejsmith]

Summary

• keeps the experiment stacked on Upgrade Elasticsearch to 9.5.0 #2416 and on the existing /events contract
• uses GET /events?mode=stack for the organization stack list and GET /events/count?mode=stack for its chart and totals; no /stack-rollups resource is introduced
• routes ordinary event list/count queries through the same ES|QL LOOKUP JOIN whenever their filter contains stack-only criteria such as status, is_fixed, is_regressed, or is_hidden
• performs event filtering, non-deleted stack filtering, stack aggregation, ranking, and deterministic keyset selection inside Elasticsearch
• removes the 20,000-stack-id materialization path, page-number stack paging, compatibility fallback, and legacy runtime switch
• updates Svelte and Angular consumers and keeps the experiment's Elasticsearch/Kibana resources isolated on ports 9215/5615

Why pursue this

The current implementation makes stack-aware event searches into a client-side join:

1. split the filter into event and stack portions;
2. search the stack index and materialize up to 20,000 matching stack ids;
3. send that id set back to Elasticsearch in a separate event query; and
4. for the stack page, build a large terms aggregation and discard buckets until the requested page is reached.

That creates a hard 20,000-stack cardinality cliff, extra network/heap/cache work, progressively more expensive deep pages, and shard-candidate accuracy concerns when ranking aggregation buckets. The same limitation affects the ordinary events endpoint whenever it filters by stack status, so solving only the stack page would leave the core problem in place.

The lookup index lets Elasticsearch execute the relationship directly:

ordinary event list
  filtered events -> LOOKUP JOIN matching stack metadata
  -> keyset predicate on (date, event id) -> LIMIT page size + 1
  -> hydrate only the selected event ids

stack list (`mode=stack`)
  filtered events -> LOOKUP JOIN matching stack metadata
  -> STATS total/users/first/last BY stack_id
  -> keyset predicate on (sort metric, stack id) -> LIMIT page size + 1
  -> hydrate only the selected stack ids

This gives both pages one source of truth for mixed event/stack filters, eliminates the intermediate id list and its cache/inversion machinery, enforces deleted-stack exclusion in the join itself, and keeps cursor work page-sized as navigation moves deeper. It is a structural simplification rather than just a faster version of the existing aggregation workaround.

How much better

These are isolated Elasticsearch 9.5 runs on the same development machine, with two warmups and seven measured iterations in alternating order. They are synthetic query comparisons, not production capacity claims, but the grouped lookup pipeline used here is the same one measured by the benchmark.

5,000 stacks / 15,000 events / page size 25

Scenario	Former median / p95	Join median / p95	Median change
First page	44.4 / 50.9 ms	16.2 / 18.5 ms	63.5% faster
Page 100	97.1 / 122.4 ms	16.6 / 20.0 ms	82.9% faster
Stack filter	45.5 / 46.8 ms	16.1 / 18.5 ms	64.7% faster

25,000 stacks / 25,000 events / page size 25

Scenario	Former median / p95	Join median / p95	Result
First page	failed at 20,000-stack limit	21.7 / 37.3 ms	join succeeds
Page 100	failed at 20,000-stack limit	23.8 / 25.2 ms	join succeeds
Stack filter	27.2 / 37.3 ms	23.4 / 26.0 ms	14.2% faster

The most important result is the shape of the work: page 100 remains near page-1 latency, and queries continue past the old 20,000-stack ceiling.

API and behavior

• GET /events?mode=stack returns ranked stack summaries and accepts total, users, first_occurrence, or last_occurrence sort in either direction.
• ordinary /events requests automatically use the join for stack-only filters and otherwise retain the normal Foundatio repository path.
• GET /events/count?mode=stack supplies stack-page totals/chart data. Ordinary event counts with stack filters also join; the currently supported joined aggregation shapes are the event dashboard and terms:tags.
• cursor tokens bind the resolved UTC range, filter fingerprint, sort value, and stable id tie-breaker. They are live deterministic keyset cursors, not snapshots.
• direct stack-document reads remain on the existing /stacks repository/search-after path because they do not aggregate events and do not need a join.
• there is no compatibility flag, fallback query, or legacy switch.

Intentional contract changes

• stack_recent, stack_frequent, stack_new, and stack_users modes are replaced by mode=stack plus explicit sort/filter parameters.
• stack-mode and stack-filtered event lists reject page; callers use before/after with limit.
• “new stacks” is represented by an explicit first_occurrence filter rather than hidden stack_new behavior.

Tradeoffs and prerequisites

• Elasticsearch 9.5 is required.
• the canonical stack lookup index must have index.mode=lookup and exactly one primary shard; production adoption therefore needs write-throughput and shard-size monitoring.
• ordinary joined event queries currently support date/-date sorting, while stack mode supports the four grouped metrics above.
• joined count requests currently support the dashboard aggregation shape and terms:tags; unsupported free-form aggregation shapes return 400 rather than silently falling back.
• COUNT_DISTINCT remains approximate, and live cursors can observe inserts/updates between requests.
• partial ES|QL results are disabled so shard failures fail the request instead of returning incomplete rankings.

Verification

• solution build: 0 warnings, 0 errors
• event endpoint integration suite: 142 passed
• OpenAPI and endpoint-manifest snapshot suite: 4 passed
• Svelte unit suite: 612 passed across 73 files
• npm run validate: formatting/lint passed; Svelte diagnostics 0 errors, 0 warnings
• frontend production build passed
• focused Chromium runs against the isolated local Elasticsearch 9.5 app passed cursor cache reuse, next/previous paging, stack detail navigation, chart interaction, and WebSocket notification stress
• local /next/ and /api/v2/about smoke checks returned 200
• hosted Build run 32672855288 at head 7a26941f: version, frontend, full API coverage, Docker, and full Aspire/Playwright E2E jobs passed

Official references: LOOKUP JOIN, lookup prerequisites, QSTR, BUCKET, COUNT_DISTINCT, and terms aggregation ordering.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	37%	35%	286	❌
Exceptionless.Core	76%	68%	10491	✔
Exceptionless.Web	85%	69%	7901	✔
Exceptionless.AppHost	70%	65%	181	➖
Summary	79% (25816 / 32710)	68% (12120 / 17868)	18859	✔