test: add core component acceptance infrastructure - #841
Open
kvinwang wants to merge 40 commits into
Open
Conversation
kvinwang
changed the title
kvinwang
force-pushed
the
codex/core-components-test-infrastructure
branch
from
July 31, 2026 03:34
7a21a97 to
313efb9
Compare
kvinwang
changed the title
kvinwang
changed the base branch from
codex/core-components-product-fixes
to
master
July 31, 2026 03:34
This was referenced Jul 31, 2026
Use the main listener's existing /health endpoint for liveness and the authenticated Admin root for the dashboard. Do not require a product-only /health/dashboard alias invented by the acceptance fixture.
TC-GOS-OBSERVABIL-003 already asserted that a peer with no handshake reaches a forced refresh, but it drove the checker with an uneven clock (1000 1010 1191 ...). That 181s jump lets the staleness deadline and the periodic refresh land on separate ticks, which is not how the service runs: the loop sleeps 10s and HANDSHAKE_TIMEOUT equals REFRESH_INTERVAL, so the two deadlines collide every cycle and the periodic refresh clears the staleness timer 10s before it can expire. Add two scenarios on a uniform 10s clock across 45 ticks: - steady_no_handshake: a peer that never handshakes must still reach --force. Only --force rebuilds a tunnel whose config is unchanged, since gateway setup returns early otherwise. - steady_force_rate_limit: with the gateway unreachable, --force must be capped at one attempt per 180s window rather than issued every tick, which would bounce the interface and re-request certificates 45 times. Both were replayed against three checker revisions. The uneven-clock scenario cannot separate the first two; the new ones can: revision uneven no_handshake steady rate limit master 0 forced (fails) 0 44 forced #945 before review 1 forced (passes) 0 45 forced #945 after review 1 forced (passes) 2 3 forced Make the sleep stub's tick budget a parameter so a scenario can run long enough to cross several refresh windows, and require callers to supply more clock values than ticks: when the file drains the stub returns a far-future sentinel whose jump forces a refresh no real deployment would perform, which is what made the checker look correct here. case.md described the old behavior as intended ("A refresh resets the stale timer whether the command succeeds or fails"), which is the defect itself. Restate the contract: only an observed handshake clears the timer, and forced refresh is rate limited because it is the expensive path.
The checker is no longer wg-checker.sh; it is dstack-util's `gateway-checker` subcommand behind dstack-gateway-checker.service. This case installed the shell script into the guest and pinned its sha256, so it could not run at all against the new image. The old driver derived the refresh matrix (periodic interval, handshake staleness, forced-refresh rate limiting) by injecting a fake clock and fake wg/dstack-util onto PATH. That matrix is now a pure decision function with unit tests in dstack/dstack-util/src/gateway_checker.rs, so reproducing it here would only restate those tests more slowly and less reliably. Dropped. What unit tests cannot reach is the process/systemd boundary, which is what the case now covers: - an app that never enabled dstack-gateway makes the checker exit 0, so Restart=on-failure leaves it alone instead of respawning it every RestartSec on every gateway-less CVM; - a missing gateway app id and a missing gateway URL each exit with EXIT_MISCONFIGURED, read from the product source at run time rather than restated here; - the installed unit is loaded, uses Restart=on-failure, inhibits restart for exactly that code, and runs the subcommand rather than the removed script. The real namespace-isolated WireGuard topology is kept: it exercises the `wg show ... latest-handshakes` shape the checker parses and is unaffected by the rewrite. Also renames the unit in tc-gos-platform-006's graph assertions and in the source inventory and coverage map. Registry: 352 scripted / 352 promoted / 0 findings. Runner unit tests: 35 passed. validate-contracts.py findings are unchanged from the branch point.
The case tested serial.history.log: that a bounded archive kept exactly one boot delimiter when a single boot overflowed its cap. #970 removes the archive outright. The live logs are now bounded within a boot by rotation, so the previous boot survives as serial.log.1 and boot boundaries land on segment boundaries rather than on delimiters recovered from log text. Every part of the case that named the archive was therefore testing something that no longer exists. Retarget it: - The decision matrix now covers segment retention, the oldest segment being discarded, an empty log not spending a slot, and stdout/stderr rotating alongside serial. - Two properties are called out as observed rather than assumed, because both fail silently: the live file keeps its inode across a rotation (a rename would leave QEMU and the supervisor appending into an unlinked inode), and it is emptied rather than compacted (so a follower resumes instead of replaying). - Step 3 gains the upgrade path: a VM inherited across a VMM restart must not be rotated on the serial channel, because its QEMU was launched without logappend=on. stdout and stderr stay eligible, being the supervisor's and always opened in append mode. The automation stopped scraping DSTACK_SERIAL_ROW markers from the unit tests. Rows are unit-test names now. The markers only existed to feed this case, they made the production tests print for no other reason, and a silently renamed marker read as a pass rather than a failure. Matching on names that must appear in the passing set fails closed instead. The fixture rewrites cvm.log.max_bytes in place rather than appending a key after cvm.use_mrconfigid. cvm.log is a sub-table, so a key appended to the [cvm] scalar block would either sit outside the table or swallow every [cvm] key declared after it. Requires #970. The behaviour under test does not exist before it merges.
kvinwang
force-pushed
the
codex/core-components-test-infrastructure
branch
from
August 6, 2026 11:32
4beeac1 to
a085f9a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is based directly on
masterand contains only core-component acceptance test infrastructure, evidence, and product-PR accounting.dstack-testrunner, registry validation, fixtures, and component harnesses.dstack/**andos/**.Independence and separation invariant
This PR targets
master; it has no dependency on product PR #840 or any split product PR.Its changed paths are limited to:
REUSE.tomldocs/testing/**tools/dstack-test/**docs/test-plans/core-components-full/**Verification
reflects_committed_state=truegit diff --check origin/master...HEAD: passedThe remaining 8 BLOCKED cases require unavailable hardware-backed evidence or GPU capability; they are not test-framework, fixture, documentation, or product-code failures.