Repository navigation
fix(verl): report failed rollout deletion instead of treating it as success - #656
Open
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Open
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Conversation
…uccess httpx.Client.delete does not raise for 4xx/5xx, so the existing failure branch never ran. Raise for status inside the try so a rejected DELETE is reported through the same best-effort path as delete_model().
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The focused fix preserves best-effort cleanup and includes a targeted regression test, with no unresolved review findings.
No open issues 🎉
What changed in this PR
This PR makes failed rollout deletion visible while preserving the rollout manager’s best-effort cleanup behavior.
Changes:
- Checks the DELETE response status so HTTP failures reach the existing error-reporting branch.
- Adds a regression test verifying that a mocked HTTP 500 response produces a deletion-failure message.
| File | Description |
|---|---|
tests/verl/test_agl_rollout_manager.py |
Adds regression coverage for reporting failed rollout deletion. |
agentlightning/verl/agl_rollout_manager.py |
Validates the DELETE response inside the existing exception handler. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AglRolloutManagerBase._delete_rollout()treats a failed HTTP DELETE as success:httpx.Client.deletedoes not raise for 4xx/5xx, and the method never checks the response. A 500 leaves the completed rollout and its events on the server for the rest of the training run, while the caller advances as if cleanup succeeded.The fix calls
raise_for_status()inside the existingtry, so the failure goes through the method's existingfailed to delete rollout ...branch. Deletion remains best-effort, matchingdelete_model()in the same class, but a rejected DELETE is no longer silent.Test:
tests/verl/test_agl_rollout_manager.py::test_delete_rollout_reports_http_errorsuses anhttpx.MockTransportreturning 500; it fails on05c7fe1because no failure is printed and passes here. The server/controller/client and rollout-manager suites are 88 passed;ruff check/format --checkare clean.Details
The failure is surfaced but not retried, and a 3xx response is not treated as an error by
raise_for_status(); both match the class's documented best-effort cleanup behavior.