Skip to content

feat(hub): the hub watches itself and leaves a dump when stuck (#398) - #402

Merged
7schmiede merged 1 commit into
mainfrom
feat/hub-watchdog
Oct 31, 2026
Merged

7schmiede merged 1 commit into
mainfrom
feat/hub-watchdog

Conversation

@7schmiede

Copy link
Copy Markdown
Member

Refs #398. This PR captures the next hang so its cause can be found; it does not fix that cause.

  • Watchdog (crates/devstack-hub/src/watchdog.rs):
    • A plain thread, outside the tokio runtime, sends GET /hub/ping to the hub's own port every 30 s and waits 10 s for an answer.
    • Only an HTTP status line counts as an answer. A connection the OS accepted on the hub's behalf does not, because a hub with a stuck runtime still accepts connections at the kernel until its queue is full. That is also why who_answers takes such a hub for foreign, and fix(hub): a start ends a hub that holds the lock and no longer answers (#398) #399's replacement never sees it.
    • After 2 minutes without an answer, the hub does the following, in order:
      1. writes hub-stuck-<ms>.dmp next to its log (Windows, MiniDumpWriteDump with thread info and indirectly referenced memory), keeping the newest 3 dumps
      2. logs hub_stuck with the path of the dump, or the error if writing it failed
      3. starts the next hub, which waits up to 15 s for the lock
      4. exits
  • devstack_proc::write_dump: Windows only; elsewhere it answers Unsupported, and the hub still restarts.
  • windows-sys: Win32_System_Diagnostics_Debug, _Kernel and _Memory added, which MiniDumpWriteDump needs.
  • Release: the Windows build's devstack.pdb ships as devstack-x86_64-pc-windows-msvc-pdb.zip. It is attested and listed in SHA256SUMS. The installer, devstack update, winget and Homebrew keep using their archives by name. docs/releasing.md and the changelog are updated.

No review card: diagnostics and a self-restart within the hub's existing role.

Tests:

  • New unit tests: two minutes of silence count as stuck; only an HTTP status line is an answer; a port that accepts and stays silent does not answer; the newest dumps are kept; a dump of the test process is written on Windows (MDMP signature).
  • clippy and fmt are clean.
  • Not run against a really stuck hub.

🤖 Generated with Claude Code

A thread outside the async runtime asks the hub's own port for
/hub/ping every 30 s; a connection the OS took is not enough, only an
HTTP answer counts. After two minutes without one the hub writes a
minidump of itself (Windows, MiniDumpWriteDump; the newest three kept),
logs `hub_stuck`, starts the next hub and exits. The release publishes
the Windows build's PDB beside the archive, so a dump can be read.

Refs #398

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@7schmiede
7schmiede merged commit f24d7d3 into main Oct 31, 2026
26 checks passed
@7schmiede
7schmiede deleted the feat/hub-watchdog branch October 31, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant