perf(events): polling data_version against watching the WAL file

performance/README.md

@@ -6,6 +6,8 @@
 - [mail](mail/README.md): reading a mailbox into the store — Gmail import speed and limits, the sender score
   against Gmail's Spam, and re-indexing after a large rules change.
 - [store](store/README.md): the shared SQLite store — filters, migrations, write-lock contention, table sizes.
+- [events](events/README.md): how processes that share the store learn about each other's changes — polling
+  against file watching.
 - [agents](agents/README.md): what the CLIs cost the AI agent that reads their answers — output size and meeting workflows.
 - [tasks](tasks/README.md): tasks found in messages — duplicate detection.
 - [startup](startup/README.md): install size and cold start of the five CLIs, and which packages a simple

performance/events/README.md

@@ -0,0 +1,7 @@
+# Events
+
+Measurements for the events and push design: how processes that share the store learn about changes.
+
+- [change-detection](change-detection/README.md) — polling `PRAGMA data_version` against watching the `-wal` file,
+  idle and during a sync-like write load: a 250 ms poll sees every commit at 1.3 µs per check; a file event alone
+  misses the newest commit and costs 10–12 times the CPU during a sync (2026-11-01).

performance/events/change-detection/README.md

@@ -0,0 +1,84 @@
+# How a reader learns that another process wrote to the store
+
+## Why it was run
+
+The owner wants WireCat Desktop's Memory page (and the planned Tasks page) to update when an agent or another CLI
+changes a task, a memory or a fact. Every WireCat CLI writes one SQLite file in WAL mode, from several processes. A
+reader in another process needs a signal that something committed. SQLite gives two candidates that work across
+processes without a daemon:
+
+- poll `PRAGMA data_version`, which changes when another connection commits
+  ([SQLite docs](https://www.sqlite.org/pragma.html#pragma_data_version));
+- watch the folder for writes to the `-wal` file (`fs.watch`: inotify on Linux, FSEvents on macOS,
+  ReadDirectoryChangesW on Windows).
+
+The questions: how fast is each, what does it cost while a sync writes heavily, and does either miss a commit? The
+answer feeds the events design in cli-core (`docs/dev/plans/2026-11-01-events.md`,
+[cli-core#2274](https://github.com/WireCatLabs/cli-core/pull/2274)).
+
+## Setup
+
+- Machine: AMD Ryzen AI 9 HX 470 (24 threads), NVMe disk, ext4, Linux 7.0, Node 24.19.0, its built-in `node:sqlite`
+  with SQLite 3.53.3 (the driver cli-core uses under Node).
+- No cli-core code runs and no real data is used. [`bench.mjs`](bench.mjs) makes a fresh WAL-mode file per row under
+  `~/.cache/events-bench-<pid>` and removes it at the end.
+- An "events" writer process commits one row to `events` every 20–200 ms (random), stamped with the commit time.
+- "sync-like writer": a second process commits 50 rows of 4 KB every ~20 ms, as a mail or messenger sync does. About
+  every 20th commit fills 1,000 pages, so SQLite checkpoints and starts writing the `-wal` file from the beginning
+  again.
+- The reader, in the bench's own process, keeps one connection open. On each wakeup it reads `PRAGMA data_version`;
+  when it moved, it reads `events WHERE seq > last`. Latency is from the writer's commit to the row being read.
+  - `poll-N`: wake every N ms.
+  - `watch`: wake on each `fs.watch` event for the `-wal` file, plus a safety poll every 5 s.
+  - `watch-settle`: as `watch`, plus one more check 20 ms after the last file event.
+- 20 s per row. Reader CPU is the bench process's user plus system time, scaled to one minute.
+
+```sh
+nice -n 10 node bench.mjs --seconds 20
+```
+
+## Results
+
+`PRAGMA data_version` costs **1.3 µs** per call (100,000 calls on one prepared statement).
+
+| Mode | Other writers | p50 ms | p95 ms | max ms | Wakeups | Commits found only by the 5 s safety poll | Reader CPU ms/min |
+|---|---|---|---|---|---|---|---|
+| poll-250 | idle | 136 | 247 | 253 | 81 | — | 82 |
+| poll-1000 | idle | 520 | 955 | 1,006 | 20 | — | 21 |
+| watch | idle | 120 | 192 | 205 | 270 | **4** | 86 |
+| watch-settle | idle | 20 | 23 | 24 | 277 | 0 | 164 |
+| poll-250 | sync-like writer | 122 | 240 | 252 | 81 | — | 89 |
+| poll-1000 | sync-like writer | 540 | 976 | 999 | 20 | — | 24 |
+| watch | sync-like writer | 11 | 20 | 42 | **24,375** | 0 | **862** |
+| watch-settle | sync-like writer | 11 | 20 | 24 | 35,793 | 0 | 1,099 |
+
+All rows, including `poll-50`, are in [results.md](results.md).
+
+## What it showed
+
+- **Polling `data_version` saw every commit of the other process.** No row was missed in any polling mode; the
+  latency is half the interval on average and never more than one interval. That matches the SQLite docs: the value
+  changes for commits "by any other connection", "including database connections in separate processes".
+- **A file event alone is late or misses the last commit.** With an idle store, `watch` found the newest row only at
+  the *next* commit (p50 120 ms here is the gap between commits), and 4 rows were found only by the 5 s safety
+  poll. The `-wal` write is seen before `data_version` moves (inferred: the writer updates the shared-memory index
+  after it writes the log, and writes to the memory-mapped `-shm` file raise no file event). One more check 20 ms
+  later fixes it (`watch-settle`).
+- **Watching is expensive while a sync runs.** Every write to the `-wal` file wakes the reader: 24,000–36,000
+  wakeups in 20 s and 10–12 times the CPU of `poll-250`. A desktop that watches would spend 1.4–1.8% of one core
+  during a sync to save about 110 ms.
+- **The store changes far more often than anything a page shows.** With the sync-like writer, 228 of 405 `poll-50`
+  wakeups saw a commit with no new event row. A signal that only says "the store changed" would make the desktop
+  reload its pages all through a sync; a row per meaningful change avoids that.
+
+Decision it fed: the reader polls `PRAGMA data_version` every 250 ms and reads an `events` table by sequence. No
+file watching and no daemon.
+
+## Not measured
+
+- macOS and Windows. `data_version` is SQLite's own and should behave the same; file events (FSEvents,
+  ReadDirectoryChangesW) were not tried.
+- A real cli-core store and a real sync; the writers here are synthetic.
+- Many readers at once (the desktop plus several agents), and a reader whose connection outlives a `store reset`
+  that swaps the file.
+- Battery and wakeups on a laptop at rest.

performance/events/change-detection/bench.mjs

@@ -0,0 +1,175 @@
+#!/usr/bin/env node
+// How fast does a reader learn that another process committed to a WAL-mode SQLite file?
+// Compares polling PRAGMA data_version with fs.watch on the database folder (the -wal file).
+// Usage: node bench.mjs [--seconds 20] [--dir /home/...]   (the folder must be on a local disk)
+import { spawn } from "node:child_process"
+import { mkdirSync, rmSync, watch } from "node:fs"
+import { join } from "node:path"
+import { argv, cpuUsage, exit, pid, stdout } from "node:process"
+import { DatabaseSync } from "node:sqlite"
+
+const print = (line = "") => stdout.write(`${line}\n`)
+const now = () => performance.timeOrigin + performance.now()
+const arg = (name, fallback) => {
+  const at = argv.indexOf(`--${name}`)
+  return at === -1 ? fallback : argv[at + 1]
+}
+
+if (argv[2] === "writer") {
+  const [, , , path, kind, seconds] = argv
+  const db = new DatabaseSync(path)
+  db.exec("PRAGMA busy_timeout = 5000")
+  const end = Date.now() + Number(seconds) * 1000
+  const event = db.prepare("INSERT INTO events (type, at, ref) VALUES (?, ?, ?)")
+  const bulk = db.prepare("INSERT INTO filler (body) VALUES (?)")
+  const blob = "x".repeat(4096)
+  let n = 0
+  const tick = () => {
+    if (Date.now() > end) {
+      db.close()
+      exit(0)
+    }
+    if (kind === "events") {
+      db.exec("BEGIN IMMEDIATE")
+      event.run("task.updated", now(), `task:${++n}`)
+      db.exec("COMMIT")
+      setTimeout(tick, 20 + Math.random() * 180)
+    } else {
+      // A sync-like writer: 50 rows of 4 KB per transaction; about every 20th commit triggers a WAL checkpoint.
+      db.exec("BEGIN IMMEDIATE")
+      for (let i = 0; i < 50; i++) bulk.run(blob)
+      db.exec("COMMIT")
+      setTimeout(tick, 20)
+    }
+  }
+  tick()
+} else {
+  const seconds = Number(arg("seconds", "20"))
+  const root = arg("dir", join(process.env.HOME, `.cache/events-bench-${pid}`))
+  const modes = ["poll-50", "poll-250", "poll-1000", "watch", "watch-settle"]
+  const rows = []
+  const microbench = () => {
+    mkdirSync(root, { recursive: true })
+    const db = new DatabaseSync(join(root, "micro.db"))
+    db.exec("PRAGMA journal_mode = WAL")
+    const statement = db.prepare("PRAGMA data_version")
+    const count = 100_000
+    const start = performance.now()
+    for (let i = 0; i < count; i++) statement.get()
+    const perCall = ((performance.now() - start) * 1000) / count
+    db.close()
+    return perCall
+  }
+  const percentile = (values, p) => {
+    if (values.length === 0) return null
+    const sorted = [...values].sort((a, b) => a - b)
+    return sorted[Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length))]
+  }
+  const runOne = (mode, load) =>
+    new Promise((resolve) => {
+      const dir = join(root, `${mode}-${load ? "load" : "idle"}`)
+      mkdirSync(dir, { recursive: true })
+      const path = join(dir, "store.db")
+      const db = new DatabaseSync(path)
+      db.exec("PRAGMA journal_mode = WAL")
+      db.exec("CREATE TABLE IF NOT EXISTS events (seq INTEGER PRIMARY KEY AUTOINCREMENT, type TEXT, at REAL, ref TEXT)")
+      db.exec("CREATE TABLE IF NOT EXISTS filler (id INTEGER PRIMARY KEY, body TEXT)")
+      const version = db.prepare("PRAGMA data_version")
+      const since = db.prepare("SELECT seq, at FROM events WHERE seq > ? ORDER BY seq")
+      let last = 0
+      let seen = version.get().data_version
+      let wakeups = 0
+      let emptyWakeups = 0
+      let foundBySafetyNet = 0
+      const latencies = []
+      const drain = (bySafetyNet) => {
+        const current = version.get().data_version
+        if (current === seen) return
+        seen = current
+        const fresh = since.all(last)
+        if (fresh.length === 0) emptyWakeups++
+        const t = now()
+        for (const row of fresh) {
+          latencies.push(t - row.at)
+          last = row.seq
+          if (bySafetyNet) foundBySafetyNet++
+        }
+      }
+      const cpuStart = cpuUsage()
+      let timer
+      let watcher
+      let safety
+      if (mode.startsWith("poll-")) {
+        timer = setInterval(
+          () => {
+            wakeups++
+            drain(false)
+          },
+          Number(mode.slice(5)),
+        )
+      } else {
+        // Watch the folder, not the file: the -wal file is created and deleted with connections.
+        // The -wal write can land before the writer updates the shared index, so data_version may not have moved
+        // yet. "watch-settle" re-checks once, 20 ms after the last file event.
+        let settle
+        watcher = watch(dir, (_event, name) => {
+          if (name && !name.endsWith("-wal")) return
+          wakeups++
+          drain(false)
+          if (mode !== "watch-settle") return
+          clearTimeout(settle)
+          settle = setTimeout(() => drain(false), 20)
+        })
+        safety = setInterval(() => drain(true), 5000)
+      }
+      const children = [spawn(process.execPath, [import.meta.filename, "writer", path, "events", String(seconds)])]
+      if (load) children.push(spawn(process.execPath, [import.meta.filename, "writer", path, "bulk", String(seconds)]))
+      let left = children.length
+      for (const child of children) {
+        child.stderr.pipe(process.stderr)
+        child.on("exit", () => {
+          if (--left > 0) return
+          setTimeout(() => {
+            drain(true)
+            clearInterval(timer)
+            clearInterval(safety)
+            watcher?.close()
+            const cpu = cpuUsage(cpuStart)
+            const walCheckpoints = db.prepare("SELECT count(*) AS n FROM filler").get().n
+            db.close()
+            resolve({
+              mode,
+              load: load ? "sync-like writer" : "idle",
+              events: latencies.length,
+              p50: percentile(latencies, 50),
+              p95: percentile(latencies, 95),
+              max: percentile(latencies, 100),
+              wakeups,
+              emptyWakeups,
+              foundBySafetyNet,
+              readerCpuMsPerMin: ((cpu.user + cpu.system) / 1000 / seconds) * 60,
+              fillerRows: walCheckpoints,
+            })
+          }, 300)
+        })
+      }
+    })
+
+  const perCall = microbench()
+  for (const load of [false, true]) for (const mode of modes) rows.push(await runOne(mode, load))
+  const fmt = (n) => (n === null ? "—" : n.toFixed(1))
+  print(
+    `node ${process.version}, sqlite ${new DatabaseSync(":memory:").prepare("SELECT sqlite_version() AS v").get().v}`,
+  )
+  print(`PRAGMA data_version: ${perCall.toFixed(2)} µs per call (100,000 calls, prepared once)\n`)
+  print(
+    "| Mode | Other writers | Events | p50 ms | p95 ms | max ms | Wakeups | Empty wakeups | Found only by 5 s safety poll | Reader CPU ms/min |",
+  )
+  print("|---|---|---|---|---|---|---|---|---|---|")
+  for (const r of rows) {
+    print(
+      `| ${r.mode} | ${r.load} | ${r.events} | ${fmt(r.p50)} | ${fmt(r.p95)} | ${fmt(r.max)} | ${r.wakeups} | ${r.emptyWakeups} | ${r.foundBySafetyNet} | ${r.readerCpuMsPerMin.toFixed(0)} |`,
+    )
+  }
+  if (!arg("keep", null)) rmSync(root, { recursive: true, force: true })
+}

performance/events/change-detection/results.md

@@ -0,0 +1,21 @@
+# Raw output, 2026-11-01
+
+`nice -n 10 node bench.mjs --seconds 20` on the machine in the README.
+
+```text
+node v24.19.0, sqlite 3.53.3
+PRAGMA data_version: 1.28 µs per call (100,000 calls, prepared once)
+```
+
+| Mode | Other writers | Events | p50 ms | p95 ms | max ms | Wakeups | Empty wakeups | Found only by 5 s safety poll | Reader CPU ms/min |
+|---|---|---|---|---|---|---|---|---|---|
+| poll-50 | idle | 177 | 27.6 | 48.0 | 54.4 | 405 | 1 | 0 | 205 |
+| poll-250 | idle | 175 | 135.9 | 246.8 | 253.0 | 81 | 0 | 0 | 82 |
+| poll-1000 | idle | 180 | 519.5 | 955.0 | 1006.2 | 20 | 0 | 0 | 21 |
+| watch | idle | 167 | 120.4 | 191.6 | 204.8 | 270 | 0 | 4 | 86 |
+| watch-settle | idle | 172 | 20.4 | 23.0 | 24.4 | 277 | 0 | 0 | 164 |
+| poll-50 | sync-like writer | 178 | 27.8 | 47.2 | 55.2 | 405 | 228 | 0 | 183 |
+| poll-250 | sync-like writer | 183 | 122.0 | 239.8 | 252.0 | 81 | 0 | 0 | 89 |
+| poll-1000 | sync-like writer | 183 | 540.3 | 976.1 | 998.5 | 20 | 0 | 0 | 24 |
+| watch | sync-like writer | 196 | 10.9 | 20.1 | 41.5 | 24375 | 913 | 0 | 862 |
+| watch-settle | sync-like writer | 182 | 11.3 | 20.2 | 24.0 | 35793 | 975 | 0 | 1099 |