diff --git a/e2e/demo.test.ts b/e2e/demo.test.ts index e163e252..0df05914 100644 --- a/e2e/demo.test.ts +++ b/e2e/demo.test.ts @@ -16,7 +16,9 @@ test('home page explains model-agnostic intelligence and paid Starter pricing', await expect(page.locator('.evaluation-method')).toContainText( '80 paired runs (40 code-navigation tasks' ); - await expect(page.locator('.evaluation-results')).toContainText('26% fewer'); + await expect(page.locator('.evaluation-method')).toContainText('60 paired audit tasks'); + await expect(page.locator('.evaluation-results').first()).toContainText('26% fewer'); + await expect(page.locator('.evaluation-results').nth(1)).toContainText('25% less'); await expect(page.getByRole('link', { name: 'CoIR code-retrieval benchmark ↗' })).toHaveAttribute( 'href', 'https://github.com/CoIR-team/coir' diff --git a/src/routes/+page.svelte b/src/routes/+page.svelte index 439d4740..b39c7d25 100644 --- a/src/routes/+page.svelte +++ b/src/routes/+page.svelte @@ -344,26 +344,39 @@

- Measured with real agent tasks · internal evaluation · October 20, 2026 + Measured with real agent tasks · internal evaluations · October 2026

Fewer calls. Fewer turns. Lower cost.

+

Code navigation · 40 tasks · October 20, 2026

26% fewertool calls per task, on average
25% feweragent turns per task, on average
20% lowermodel cost per task, on average
+

Code audits · 60 tasks in Go and TypeScript · October 31, 2026

+
+
26% fewertool calls per task, on average
+
25% lesstime per task, on average
+
21% lowermodel cost per task, on average
+
About the evaluation

- Claude Sonnet 5.5 with and without Context Engine: 80 paired runs (40 code-navigation tasks, - two seeds) in two repositories. Tool calls per task fell from 14.2 to 10.4 and agent turns - from 15.2 to 11.4 (both p < 0.001); model cost per task fell from $0.118 to $0.094 - (p = 0.02). Answer accuracy held level (60.5% vs 61.3%). Intervals are task-clustered - bootstrap; p-values are task-level sign-flip permutation tests. An earlier October 5, 2026 - evaluation with Claude Sonnet 4.5 measured file-and-symbol accuracy rising from 52.2% to - 64.3%. + Code navigation: Claude Sonnet 5.5 with and without Context Engine, 80 paired runs (40 + code-navigation tasks, two seeds) in two repositories. Tool calls per task fell from 14.2 to + 10.4 and agent turns from 15.2 to 11.4 (both p < 0.001); model cost per task fell from + $0.118 to $0.094 (p = 0.02). Answer accuracy held level (60.5% vs 61.3%). Intervals are + task-clustered bootstrap; p-values are task-level sign-flip permutation tests. An earlier + October 5, 2026 evaluation with Claude Sonnet 4.5 measured file-and-symbol accuracy rising + from 52.2% to 64.3%. +

+

+ Code audits: Claude Sonnet 5.5 with and without Context Engine on 60 paired audit tasks (30 in + Gitea, Go; 30 in cal.com, TypeScript), one run each. Tool calls per task fell from 14.4 to + 10.6 and agent turns from 15.4 to 11.6 (both p < 0.001); model cost per task fell from + $0.185 to $0.146 (p < 0.001) and time per task from 118 s to 88 s (p = 0.002).

With learning enabled, retrieval adapts as feedback accumulates. These measurements capture