Skip to content

Hotfix: install releases carry writer locks before services restart - #3889

Merged
trissim merged 1 commit into
mainfrom
ccr-b17367e7-4095oh-install-locks
Nov 2, 2026
Merged

trissim merged 1 commit into
mainfrom
ccr-b17367e7-4095oh-install-locks

Conversation

@trissim

@trissim trissim commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Hotfix, 1 file, +12/−10 in installer/upgrade.py. No change to carries or credentials. #3861's credential carry and #3884's backup fix stay exactly as merged.

Failure (install 6e8a33d5, target 40294a2)

ServiceNotServingError: Connector round trip failed: store lock busy, held by PID 86741
  (PID 86741 = this installer: python -m agent_comms.installer.prepared)
... restoring original installation
ServiceNotServingError: Connector round trip failed: store lock busy, held by PID 86741

ExistingBusInstall.live() entered services_stopped inside staged.borrowing(writer_declarations). Both the forward stopped.start_on(...) and the restore-on-exit start_services ran ServingProbe while the installer held the carried stores' writer locks. borrowing() takes no locks when nothing is pending, so this ordering stayed latent until #3861's connector_access_carry: the first pending carry whose writer declarations include a store the connector reads, its credentials, to authenticate the probe's MCP call.

Repair

  • services_stopped is now the outer context and borrowing the inner one, so on every exit the locks release before the stopped services restart.
  • The forward path retires carry inputs (after_acceptance) right after publish, still under the locks, then closes them, clears maintenance and starts services.
  • The later retirement block is removed; owners publish exactly as before.

Evidence

  • tests/test_install_rehearsal.py tests/test_install_announcements.py tests/test_host_services.py tests/test_carry_connector_access.py: 40 passed, 4 failed and 18 errors, identical on main in this container. Every failure and error is GitVersionUnsupported (the container has Git 2.43, and Core requires 2.44), so the live-path controls are unrun here.
  • The proof is the next real install on the host. A failure there rolls back, as before.

Generated by Claude Code

live() entered services_stopped inside borrowing(), so services restarted and
the connector's ServingProbe ran while the installer held the carried stores'
writer locks. With #3861's connector_access_carry pending, the probe's MCP
round trip could not read the credentials store ("store lock busy, held by"
the installer itself): install 6e8a33d5 failed, and its restore failed the same
way. The writer locks now span carry, publish and input retirement only;
services restart after they are released, on success and on restore alike.
Retirement moves before the restart, still under the locks and still before
the rollback point's end. No carry or credential behavior changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016gZLdttfQjtizEYZGek9Gh
Copilot AI balanced review requested due to automatic review settings November 2, 2026 17:55
@trissim
trissim merged commit 5f12aa5 into main Nov 2, 2026
1 of 2 checks passed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants