Skip to content

test: Add failing test for set_config() blocking concurrent get() - #499

Open
aviadr1 wants to merge 1 commit into
launchdarkly:mainfrom
aviadr1:test/set-config-blocks-get
Open

test: Add failing test for set_config() blocking concurrent get()#499
aviadr1 wants to merge 1 commit into
launchdarkly:mainfrom
aviadr1:test/set-config-blocks-get

Conversation

@aviadr1

@aviadr1 aviadr1 commented Aug 15, 2026

Copy link
Copy Markdown

Failing test for #496. Test only — no fix.

Found while investigating #493, a production incident where LDClient.close() hung and left worker pods alive for hours to days. This issue is what makes that class of hang so damaging under the singleton API. I have not observed this specific path in our own production — we do not call set_config() after startup — but the coupling is real and reproduces reliably.

What the test pins

set_config() holds the global write lock across both the construction of the replacement client and old_client.close() (ldclient/__init__.py:38-47). ReadWriteLock.lock() holds the underlying mutex for the whole write section and rlock() needs that same mutex, so every concurrent ldclient.get() blocks — and get() is on the hot path of every flag evaluation in an application using the singleton API.

test_set_config_does_not_block_concurrent_get drives a 3 second close() and measures a concurrent get():

AssertionError: ldclient.get() blocked for 2.8s while set_config() was closing the previous client

Before a shutdown timeout existed, an unbounded close() here would block every get() in the process indefinitely. Even bounded, the window is start_wait (default 5s) plus the shutdown timeout, so a reconfigure can stall all evaluating threads for roughly ten seconds.

Why xfail

Marked @pytest.mark.xfail(strict=True) so CI stays green while the defect is documented; being strict, it fails loudly once fixed so the marker gets removed. To see the real failure:

uv run pytest ldclient/testing/test_ldclient_singleton.py -k set_config_does_not_block --runxfail

Why no fix

Moving construction and close() outside the lock is the obvious direction, but doing it without introducing a lost-update race between concurrent set_config() callers is a design decision I would rather leave with you.


Note

Overview
Test-only change for issue #496: documents a singleton API defect where set_config() keeps the global write lock while it builds the replacement client and runs old_client.close(), so concurrent ldclient.get() (read lock) stalls on the same mutex.

Adds test_set_config_does_not_block_concurrent_get, which patches LDClient.close to sleep 3s during reconfigure and asserts a concurrent get() finishes in under half that time. The test fails today (~2.8s block) and is marked @pytest.mark.xfail(strict=True) so CI stays green until a fix lands.

No library changes; the PR pins the invariant that reconfigure should not couple slow shutdown to the flag-evaluation hot path.

Reviewed by Cursor Bugbot for commit ce3e272. Bugbot is set up for automated code reviews on this repo. Configure here.

set_config() holds the global write lock (ldclient/__init__.py) across both the
construction of the replacement client and old_client.close(). ReadWriteLock
holds the underlying mutex for the whole write section, so every concurrent
ldclient.get() - which takes a read lock, and which sits on the hot path of
every flag evaluation - blocks until both finish.

The test drives a 3 second close() and measures how long a concurrent get()
blocks; it currently reports 2.8s. Marked xfail strict so it fails loudly once
the behaviour is fixed and the marker can be removed.

This couples a network-dependent shutdown to a lock that every evaluating
thread needs, so a slow or stalled close() stalls the whole application rather
than just the thread that called set_config().

No fix is proposed here; this only pins the contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant