Repository navigation
fix: stabilize flaky integration tests - #140
Merged
Merged
Conversation
bitgorbovsky
requested changes
Oct 29, 2026
NorthArea
pushed a commit
that referenced
this pull request
Oct 29, 2026
Drop the statement semicolons from the Lua readiness probe and read the state string with result.get(1): the probe always returns the pair (ready, state). Review comments on #140.
NorthArea
changed the title
added 2 commits
October 29, 2026 16:16
The readiness probe added in 1.7.0 iterated `rs.replicas`, a field that `vshard.router.info()` does not return (vshard 0.1.25 reports `master` and `replica` per replicaset). The loop never ran, so the container was reported ready while a replicaset master was still in the vshard handshake, and the first `crud.truncate` in spring-data integration tests failed with VHANDSHAKE_NOT_COMPLETE (CI of #139). Checking the master status alone is not enough. On a stand the router state stayed `storage-002 master=unreachable replica=available` for 120 s: the router connected to storage-002-a before its storage was configured, the vshard handshake failed with "storage is not configured", and vshard closed the connection (vshard/replicaset.lua conn_vconnect_check_or_close). Only a request to that master reopens it; discovery and failover talk to the replica. The first asynchronous crud map call reopens it but does not wait for the new handshake. Requiring zero unknown buckets does not work either: the router may not know part of the buckets for a while after the masters are ready. Make a synchronous `callrw('vshard.storage.buckets_count')` to the master of every replicaset: it reopens a closed connection and waits for the handshake. Require all calls to succeed and every master to be `available`, and log call errors, replica and bucket states.
checks 287c88e ("exclude nil function name from error messages",
2026-10-28) drops the " to nil" suffix from "unexpected argument"
errors. crud 1.5.0 depends on `checks >= 3.1.0-1`, so the test cluster
picks it up on every fresh build, and testSelectWrongParameters fails
on master with "... select.lua:320: unexpected argument opts.coolOption".
Accept the message with and without the suffix.
NorthArea
force-pushed
the
northarea/vshard-storages-ready-probe
branch
2 times, most recently
from
October 29, 2026 11:33
bb7ca71 to
3bc1f6f
Compare
nickkkccc
approved these changes
Oct 29, 2026
bitgorbovsky
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two integration tests that failed in CI.
fix(testcontainers): await vshard master handshake—VshardClusterContainerwas reported ready while a replicaset master was still in the vshard handshake: the 1.7.0 probe iteratedrs.replicas, whichvshard.router.info()does not return. The first asynccrud.truncatethen failed withVHANDSHAKE_NOT_COMPLETE(CI of fix: handle failed connection attempts once #139,tests-crud-integration (3.5.0)). The probe now makes a synchronouscallrwto every master (it reopens a connection that vshard closed after a failed handshake and waits for the new one) and requires every master to beavailable.test(client): accept checks error message without function name—testSelectWrongParametersfails on every fresh build since checks287c88edropped theto nilsuffix; the test cluster does not pin checks.Known flaky tests not fixed here:
ConnectionToTarantoolTest.testSendAndReceiveWithConcurrentClose(tests-box-integration) — a race inConnectionImpl.send()/close(), fixed in fix: handle failed connection attempts once #139.CartridgeClusterContainer.setupTopologyon 2.11.8 (tests-crud-integration) —cartridge replicasets setupcause unknown.Чинит два интеграционных теста, падавших в CI.
rs.replicas, которого нет вvshard.router.info()); первыйcrud.truncateпадал сVHANDSHAKE_NOT_COMPLETE. Теперь синхронныйcallrwк каждому ведущему и все ведущиеavailable.testSelectWrongParametersпадает на каждой свежей сборке после287c88eв checks (убран хвостto nil), версия checks не закреплена.Известные нестабильные тесты, которые здесь не чинятся:
testSendAndReceiveWithConcurrentClose(box) — гонкаsend()/close()в ядре, исправлена в fix: handle failed connection attempts once #139.CartridgeClusterContainer.setupTopologyна 2.11.8 (crud) — причина пока неизвестна.I haven't forgotten about:
See also #139