[fylorn]

Summary

A provider can now set the slowest generation its streams may keep up. A stream that falls below it is cut with a 503 so the client can try again, and the route is charged a failure.

• Columns (added on startup in db/schema.sql): providers.min_speed_tokens_per_sec INT NULL (1–1000, NULL = off) and providers.min_speed_window_secs INT NOT NULL DEFAULT 20 (5–120), both with CHECK constraints.
• Admin API: POST /api/admin/providers and PATCH /api/admin/providers/{id} take min_speed_tokens_per_sec and min_speed_window_secs. On PATCH, a missing field is left unchanged and null turns the check off. Values out of range get a 400. The provider responses include both fields.
• Web: the edit-provider dialog gets one row, "Stop when generation is slower than [ ] tokens/s for [20] s", checked as you type. A blank speed means off. en/zh keys are in parity.
• Gateway (proxy/min_speed.rs, wired into the stream pump):
  • Judging starts at the first token and runs once a second, after one window has passed.
  • Speed = tokens in the last window / (window − pauses of 5 s or more). If the pauses take up more than half the window, that second is skipped. Nothing is judged after the upstream's stop reason.
  • Tokens count answer text, reasoning text and tool arguments, read from the upstream's own format: an ASCII byte is a quarter token, any other character is one token. For Gemini, its running output count is used when it is higher.
  • When a stream is too slow:
    • The client gets tw_dialect::convert::error_frame(client, 503, …): overloaded_error for Anthropic, response.failed for Responses, UNAVAILABLE for Gemini, a Chat error chunk for Chat.
    • The upstream connection is dropped and the outcome is UpstreamTooSlow / 503, which counts against the route's breaker.
    • The session's affinity to that route is cleared.
    • Usage so far is estimated, the same way as any stream cut short.
    • The log row gets tokens_per_sec, the warn log line carries the numbers, and gateway_provider_too_slow_total{provider} is incremented.
  • Whole answers and Responses WebSocket turns are not measured.
• CHANGELOG [Unreleased] entry.

Tests

• Unit tests:
  • judging rules: before the first token, before a full window, under the minimum, at the minimum, long and short pauses, a pause that runs into the window, a window mostly paused, after the stop reason
  • token counting: reasoning, tool arguments and signatures; Gemini's running count
  • row parsing
  • error status, tag and message
  • admin validation and PATCH null-vs-absent handling
  • dialog round-trip and range check
• Integration tests in crates/test-support/tests/gateway_min_speed.rs, run against paced SSE upstreams:
  • a slow Chat stream is cut (error frame, upstream hung up, one route failure)
  • an Anthropic client gets overloaded_error
  • a stream at speed runs to its end
  • a long pause does not cut the stream
  • with no minimum set, a slow stream is left alone
  • the console sets, keeps and clears the values, and refuses ones out of range
• make precommit passes. The new integration file and the related gateway, streaming, retry, console and OpenAPI suites pass locally.

🤖 Generated with Claude Code