Skip to content

fix(scripts): surface lerna publish outcomes and trim verify's wait - #10118

Open
vibhavgo wants to merge 1 commit into
masterfrom
WCI-2113-beta-release-hardening
Open

vibhavgo wants to merge 1 commit into
masterfrom
WCI-2113-beta-release-hardening

Conversation

@vibhavgo

@vibhavgo vibhavgo commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Surfaces the real outcome of the lerna publish steps (tee'd logs + ::warning:: annotations keyed on outcome, which continue-on-error does not mask) and uses what lerna actually published to trim Verify Publish's propagation wait — a storm-aborted wave publishes only a fraction of the tree, and waiting 15 minutes for versions that never existed was pure loss. Also caps per-package conflict polls at ~30s on mass-stall days so the settle pass cannot run packages × 5 minutes against the job timeout.

Linear: WCI-2113

Changes

  • publish.yml: both lerna steps get id: + shell: bash (opts into -eo pipefail — the default run shell has no pipefail, and a cmd | tee pipeline without it masks lerna's failure) + 2>&1 | tee /tmp/lerna-{siblings,bitgo}-publish.log; two ::warning:: steps keyed on steps.<id>.outcome == 'failure'; log paths passed to Verify; timeout-floor comment with empirical evidence (healthy 43 min, storm 78 min — do not trim timeout-minutes: 100); EUNCOMMIT on the bitgo step documented as the desirable storm-day path (document-don't-fix: fixing it would run a 20-min prepack against missing dependencies; the healthy-day Nov 3 run proves the step publishes bitgo fine).
  • verify-release.ts: parseLernaPublishedPackages + parseLernaAttemptedPackages (version-anchored regexes — excludes lerna's published N packages summary lines; lerna verb publish <name> marks every dispatched package, verified against lerna 10.0.1 output: 114 unique names in a healthy log = exactly the candidates), readLernaPublishSets (unions both logs into published + attempted sets with published ⊆ attempted; fail-open: unreadable logs or readable logs with no recognized lines — format drift — revert to waiting for every package with a one-line warning), shouldShortPollConflict (mass-invisible OR aborted-wave trigger, keyed on how many packages lerna reached), VerifyPackageOptions gains sleep/now/conflictPollTimeoutMs passthrough into publishWithRecovery, and verify() builds its wait-list from the attempted set — errored packages (TLOG 409 usually means the write landed anyway) get the wait instead of a needless bump plus dependent cascades; only never-dispatched packages skip it — and threads the 30s cap on mass-stall days.
  • Tests: 12 new (59 total passing) — parse (scoped/unscoped/summary/emoji/WARN exclusion), fail-open ×3 (missing file, unreadable, no publish lines), short-poll truth table incl. boundaries, poll-cap passthrough (asserts 1 visibility + 3 poll checks vs the default 11).

Test Plan

  • npx mocha --require tsx ./scripts/tests/generate-bitgo-shrinkwrap.test.ts ./scripts/tests/verify-release.test.ts — 57 passing; tsc --noEmit clean on changed files; workflow YAML validated.
  • Two adversarial subagent review passes over the full diff: both "correct", first pass's two P3s (summary-line capture, comment overstatement) fixed and re-verified in the second pass against live files.
  • Live: healthy-day dispatch unchanged (~43 min); storm-day dispatch expected ~63 min (from 78) with the ::warning:: visible on the run summary and the "Lerna published N of M packages" line in Verify's log confirming the trimmed wait.

@linear-code

linear-code Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

WCI-2113

@vibhavgo

vibhavgo commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@claude review always

@github-actions

This comment was marked as outdated.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread scripts/verify-release.ts Resolved
vibhavgo added a commit that referenced this pull request Nov 3, 2026
Review finding on #10118: a TLOG-409-errored package usually landed on
npm anyway (the error is a duplicate-provenance false negative), but it
got no success line, so it skipped the propagation wait and the settle
bumped it - and its dependents - needlessly. The wait list is now every
package lerna dispatched (lerna verb publish lines), with published
names unioned in as a guaranteed superset; only never-dispatched
packages from an aborted wave skip the wait. Also: readable logs with
zero recognized lines now warn once and fail open (format-drift
signal), the short-poll cap's whole-settle scope is documented, and the
lerna logs upload as an artifact when any step fails (placed after
Verify so a Verify-only failure still captures them).

Ticket: WCI-2113
@vibhavgo

This comment was marked as outdated.

@vibhavgo
vibhavgo force-pushed the WCI-2113-beta-release-hardening branch from 4f5df83 to 5b0828e Compare November 3, 2026 10:08
@vibhavgo

vibhavgo commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@claude review

@github-actions

This comment was marked as outdated.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread scripts/verify-release.ts Resolved
A storm-aborted lerna wave publishes only a fraction of the tree, but
continue-on-error paints the step green, so Verify's 15-minute wait was
spent on versions that never existed and the failure was invisible until
someone read 20k log lines.

Both lerna steps now tee their output and Verify waits only for what
lerna reached: every dispatched package (lerna verb publish), published
or errored — a TLOG 409 usually means the registry write completed
anyway, so an errored package still gets the wait instead of a needless
bump plus dependent-bump cascades — with published names unioned in as a
guaranteed superset. Only never-dispatched packages from an aborted wave
skip the wait. Fail-open: unreadable logs, or readable logs with no
recognized lines (format drift), revert to the full wait with a warning.

Failed-but-continued steps raise ::warning:: annotations keyed on
outcome (conclusion is masked by continue-on-error). Mass-stall days cap
per-package conflict polls at ~30s so the settle cannot run packages ×
5 minutes against the job cap; the bump that cap can cost is the
documented, accepted trade. The EUNCOMMIT skip on the bitgo step is
documented as the desirable storm-day path, and the lerna logs upload as
an artifact when any step fails (placed after Verify so a Verify-only
failure still captures them).

Ticket: WCI-2113
@vibhavgo
vibhavgo force-pushed the WCI-2113-beta-release-hardening branch from 5b0828e to 896a08e Compare November 3, 2026 10:31
@vibhavgo
vibhavgo marked this pull request as ready for review November 3, 2026 10:42
@vibhavgo
vibhavgo requested review from a team as code owners November 3, 2026 10:42

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@vibhavgo
vibhavgo requested a review from a team as a code owner November 3, 2026 10:42
@vibhavgo
vibhavgo requested review from davidkaplanbitgo, ranga-r2 and rishikeshdadam136 and removed request for ranga-r2 November 3, 2026 10:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant