workflow: a run whose ephemeral map is gone leaves its hidden run Bots unreapable #1414

Activity

1. Birfy commented on Oct 27, 2026

   

   Birfy

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Written-by: Triage

   Labels applied: bug, area:automation, needs-info — it was the only open workflow issue left with no type or area label, and it is parked on a decision rather than on a repro.

   Verified on main 26657c8a: plugins/workflow/src/service.js:549 is if (!run?.ephemeral) continue, so a run persisted without the map is skipped for good — by design of #1388's scope, which is why this needs its own shape rather than a change to reap(). The two shapes in the body are a design call (fields.runId on each run Bot so an orphan can be reclaimed vs. the spawn path writing the id to the run before the first Bot exists); both touch core/ if the Bot record gains a field, so whoever decides should say which, since that decides the risk tier and the test surface.

   Not agent-ready until that call is made: an agent would have to invent the record shape, and #1388's accepted behaviour ("a run without the map must not be rewritten") is exactly what an implementer would be tempted to relax. Related, not duplicates: #1387 (dispose() drops the records before the release finishes), #1390 (concurrent assign()), #1424/#1425 (fresh follow-ups from #1411).
2. Birfy

   added

   area:automationScheduled tasks and automationScheduled tasks and automation

   bugSomething isn't workingSomething isn't working

   needs-infoWaiting for the reporter or a maintainer decisionWaiting for the reporter or a maintainer decision

   on Oct 27, 2026
3. Birfy commented on Oct 27, 2026

   

   Birfy

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Root cause found for the unknown in point 1: the ephemeral map is not lost by a spawn-path race — dispose() deletes it at host shutdown while leaving the run running.

   Code (main)

   • dispose() (plugins/workflow/src/service.js:998) calls releaseAll(ids).then(() => { for (const run of runs) { delete run.ephemeral; delete run.handed } ; write() }). Nothing sets status, so every live run is persisted as live: true, status: "running" with no map.
   • reap() (:963) starts with if (!run?.ephemeral || seen.has(run)) continue, so the run written that way is skipped forever. There is no other path to end it: runningOf() (:199) counts it (live === true && status === "running"), and reset/pause are RPC-only (index.js:102/103), so nothing in the product can free the slot.

   Measured on this host, 2026-10-27T14:50Z restart

   • Store /data/dsh/bot/flybot-workflow/workflows.json (written 14:50Z): run-3fd2406e (fix(workflow): every stage chat line is clipped at 240 characters with no way to read the rest (introduced by #1456) #1460), run-0ca151a7 (label-hygiene: isolate a refused label instead of abandoning the sweep (#1418) #1422), run-2f460b96 (fix(palette): read a stored line's words in a ⌘K row, not its fence markers or trace #1395) all carry live: true, status: "running", each with nodeStatus.<node> = "running" and no ephemeral key. run-ffb574ee (fix(host): a Workflow run Bot keeps the team tools, whatever the node's tools list says #1445), started 14:50:59Z after the restart, has the full 7-node map. So: runs that were in flight at shutdown lose the map; runs started after the restart keep it.
   • bot/logs/diagnostics.jsonl at epoch 1793112215664 (≈14:43:35Z, shutdown): Bot Claim + Implement c351 not archived: DomainError: domain 'workspace' is closed, same for Bot Verify f710 — the release itself fails because the workspace domain is already closed, and dispose() drops the ids anyway.

   Consequences seen today (two user-visible ones)

   1. Capacity: runningOf() counts the dead runs, so hasRoom() (:533) answers no. Lane wf-2d79a70a (PR → 合入, maxConcurrent: 2) is at 2/2 with both slots held by dead runs; no new PR run can start at all. Lane wf-a85c4250 holds one dead run (fix(workflow): every stage chat line is clipped at 240 characters with no way to read the rest (introduced by #1456) #1460) out of 2.
   2. Replies: findEphemeral() (:241) returns null for those runs, so ownsBot() (:788) is false and the interception in index.js:139 lets a node's reply fall through to a seated Bot's chat instead of advancing the run. Observed twice: Reply from Inspect c633 at 14:50:27Z (route: rework for fix(palette): read a stored line's words in a ⌘K row, not its fence markers or trace #1395) and Reply from Enqueue 3583 at 14:52Z (PR label-hygiene: isolate a refused label instead of abandoning the sweep (#1418) #1422 enqueue/landing confirmation) both landed in the Chief chat while their runs still said inspect/enqueue = running.

   Fix proposal

   1. dispose(): end live runs before dropping ids — mark each status: "failed", set updatedAt, push the same history line the other failed transitions push. The store must be self-describing after shutdown; today it is not.
   2. reap(): also treat run.live === true && run.status === "running" && !run.ephemeral as abandoned (no ids to release — just end it and move it to history), so stores already written by older builds heal on the next load.
   3. Point 2 of the original body (fields.runId on each run Bot) is still worth it for reclaiming the orphaned records, but it is not needed to stop the lane being held.

   Acceptance (tests, plugins/workflow)

   1. Start a live run with a spawn stub, call dispose(), then build a new service on the same dir: lane.active empty, lane.history has the run with failed, and the persisted record has no ephemeral and does not say running.
   2. Hand-written legacy store (live: true, status: "running", no ephemeral): after load + reap(), runningOf() is empty (capacity free) and the run appears in history as failed.
   3. A reply from an old node Bot of such a run is dropped (ownsBot() false) and does not open a turn in a seated Bot's chat.
   4. No regression: existing workflow tests green; a reaped run that still had its map ends failed (workflow: reap() skips running runs, and it only runs once at plugin load #1388), an already ended run keeps its status, release-before-forget stays (workflow: dispose() drops the ephemeral Bot records before the release finishes #1387).

   This is ready to be re-triaged off needs-info; I am not changing labels myself.
4. Birfy commented on Oct 27, 2026

   

   Birfy

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Live repro on prod (2026-10-27, tip 952d6358)

   FlyBotDebugger measured Workflow wf-2d79a70a 「FlyBot PR → 合入」, run run-0ca151a7 (task #1422):

   • After instance restart at 14:49:55Z (upgrade boot), run is still status=running · live=true with no ephemeral / no handed.
   • nodeStatus: … review done · enqueue=running · landcheck/escalate queued; never advanced past Enqueue (PR had already merged at 14:44:39Z).
   • Matches the dispose→reap hole: unload dispose() strips ephemeral without ending the run; boot reap() skips !run.ephemeral, so the zombie persists.

   Fix direction (either):

   1. dispose() must fail/end active live runs before clearing ephemeral, or
   2. reap() must treat live && status==='running' && !ephemeral as abandoned (mark failed, clear from lane.active).

   Also related: #1424 (nodeStatus left running after a successful reap). Owner separately filed #1463 for chip/list mismatch + per-run controls (zombie inflates activeCount).
5. Birfy

   closed this as completedin #1465on Oct 27, 2026
6. HugoXK

   added a commit that references this issue on Oct 27, 2026

   End workflow zombie runs on dispose and reap (ThinkFlowLab#1465)

   ...

   This issue will close once commit 8f1ac12 is merged into the 'main' branch.

   8f1ac12