Skip to content

tests: a test process drops the databases it created - #313

Merged
vsilent merged 1 commit into
devfrom
fix/tests-drop-their-databases
Oct 29, 2026
Merged

vsilent merged 1 commit into
devfrom
fix/tests-drop-their-databases

Conversation

@vsilent

@vsilent vsilent commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Problem. Every spawned test app creates its own database (about 11 MB with the migrations), and nothing drops them. A full CI run fills the runner's disk until Postgres crashes:

PANIC:  could not write to file "pg_wal/xlogtemp.2342": No space left on device

The last few hundred tests then fail with Failed to start test app (and failed to lookup address information for postgres). Which tests turn red depends only on which ran after the crash. This hits every branch, including main (9b165e7), #310, #311 and #312. Re-runs do not help. Locally stacker-test-pg had accumulated 4450 such databases, 47 GB.

Change (test code only)

  • New tests/common/test_databases.rs, included by both harnesses the same way as vault_kv_mock.rs.
  • configure_database in tests/common/mod.rs and tests/steps/common.rs registers each database right after CREATE DATABASE.
  • An atexit handler drops them all with DROP DATABASE ... WITH (FORCE) when the test process exits.
  • Why at process exit, not Drop: most apps live in a static shared by a test file, and statics are never dropped. Under nextest a process is one test; under cargo test it is one binary.
  • The drop runs on a fresh thread, because at exit the main thread's thread-locals, which tokio needs, are already gone.
  • A killed process (timeout, signal) still leaks its databases.

Tests

  • tests/test_databases_are_dropped.rs runs a child test process that spawns an app, then checks that the database is gone. It failed against the old code (database ... outlived the test process) and passes now.
  • Locally, the full cargo nextest run --tests -E 'not binary(bdd)' passed 2967 of 2967, with 0 new databases left (4450 before, 4450 after).
  • cargo test --test bdd: 334 scenarios passed, 0 new databases left.
  • clippy and rustfmt report nothing in the changed files.

Once this is merged, #310, #312 and the others need a re-run (or a merge of dev) to go green.

🤖 Generated with Claude Code

Every spawned test app creates its own database (~11 MB with the
migrations) and nothing dropped them. A full CI run filled the runner's
disk until Postgres panicked ("could not write to file pg_wal/...: No
space left on device") and the last few hundred tests failed with
"Failed to start test app", on every branch.

configure_database (integration and BDD harnesses) now registers each
database it creates, and an atexit handler drops them all with
DROP DATABASE ... WITH (FORCE) when the process exits. Most apps live in
a static shared by a test file, so a Rust Drop would never run; process
exit is the end of each test under nextest and of each binary under
cargo test. The drop runs on a fresh thread: at exit the main thread's
thread-locals, which tokio needs, are gone.

Regression test: tests/test_databases_are_dropped.rs runs a child test
process that spawns an app, then checks its database is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vsilent
vsilent merged commit 222667a into dev Oct 29, 2026
11 checks passed
@vsilent
vsilent deleted the fix/tests-drop-their-databases branch October 29, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants