Skip to content

fix: allow concurrent streamable http requests - #1186

Draft
nickcoai wants to merge 4 commits into
mainfrom
agent/configurable-streamable-http-parallelism
Draft

fix: allow concurrent streamable http requests#1186
nickcoai wants to merge 4 commits into
mainfrom
agent/configurable-streamable-http-parallelism

Conversation

@nickcoai

@nickcoai nickcoai commented Aug 18, 2026

Copy link
Copy Markdown

Summary

Allow ordinary streamable http POSTs to run concurrently. Previously, one slow application/json response blocked unrelated requests.

StreamableHttpClientTransportConfig::max_concurrent_requests defaults to 16. Set it to 1 to keep ordinary POSTs serial; 0 is treated as 1. Open sse response streams do not count against this limit. Initialization and protocol-version changes remain ordered with ordinary POSTs.

Cancellation and recovery

Cancellation and replies to server requests use a separate bounded queue with one extra POST slot. Each control POST has a five-second timeout after it starts. Queued and active ordinary POSTs can be cancelled without waiting for a free slot.

For an open legacy response stream, cancellation stops reading but keeps the stream alive until the cancellation send finishes or is dropped. This lets custom http adapters handle cancellation using their existing stream state. Ordinary legacy cancellation notifications still reach the adapter; modern http cancellation remains local.

When recovery is enabled, ordinary POSTs that report SessionExpired in the same session share one recovery attempt. Wait up to session_recovery_timeout for old POSTs, then stop any that remain. Those interrupted POSTs return SessionRecoveryTimeout and are not retried: the server may have processed them. The replacement handshake has a separate timeout of the same length; both default to five seconds.

Retry only ordinary POSTs that returned SessionExpired, at most once each. Control POSTs are not retried. A request that already completed on the common response stream is not retried. Old queued controls cannot change replacement-session state, and a cancelled or dropped version-change request does not block later requests.

The shared worker's control queue is opt-in; server scheduling is unchanged. Callers still decide which tools may run together and which need approval. Oauth refresh coordination remains a separate follow-up.

Validation

Passed locally:

  • Client-only concurrency target: 18 tests, including early responses, recovery, cancellation, and custom-adapter compatibility.
  • Worker unit tests: 3 tests for abandoned cancellation sends, request-ID reuse, and exact cancellation-ID matching.
  • cargo +1.96 test --offline --all-features
  • Selected transport targets with the complete non-local feature set from justfile: 34 tests covering concurrency, subscriptions, expired sessions, disconnects, connection reuse, and shutdown.
  • cargo +1.96 clippy --offline --all-targets --all-features -- -D warnings
  • cargo +nightly fmt --all -- --check
  • git diff --check

Downstream custom-adapter integration tests also passed without adapter source changes.

@github-actions github-actions Bot added T-documentation Documentation improvements T-dependencies Dependencies related changes T-test Testing related changes T-config Configuration file changes T-core Core library changes T-transport Transport layer changes labels Aug 18, 2026
@nickcoai nickcoai changed the title fix: make streamable http client parallelism configurable fix: allow concurrent streamable http requests Aug 18, 2026
@nickcoai
nickcoai force-pushed the agent/configurable-streamable-http-parallelism branch from fac20e9 to fd439b3 Compare August 18, 2026 15:55
Keep cancellation and replies available while old session POSTs finish.
Bound the wait for old POSTs and the replacement initialization handshake.
Do not retry interrupted POSTs because the server may have processed them.

Add regressions for recovery, queued cancellation, control timeouts, and
server replies needed by active requests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T-config Configuration file changes T-core Core library changes T-dependencies Dependencies related changes T-documentation Documentation improvements T-test Testing related changes T-transport Transport layer changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant