Repository navigation
hive/openai-server: held to OpenAI's spec and to Open Responses, checked in every test - #104
Merged
Merged
Conversation
…ked in every test The specs ride in spec/, pinned: OpenAI's OpenAPI cut to what the hive serves, and Open Responses whole; `bun spec/fetch.ts` pulls them again. The end-to-end client checks every request it sends, and every answer and stream event it hears, against both, so a test cannot pass on a shape no client would send or take. The seven HTTP cases of the Open Responses compliance suite run among them, as the suite writes them. What the specs found, and how it is now: - a deleted response says `object: "response.deleted"`; - chat leaves `system_fingerprint` out rather than sending null; - a response carries every field both specs require, the request's settings or their defaults, `completed_at` once complete, and the tokens a cache was written with; - an echoed function tool carries `description`, `parameters` and `strict`; - `response.failed` says an error code from the spec's list, its own code kept as `type`; - a call the server ran that failed says so as the spec shapes it, an `mcp_tool_execution_error`, in a response and on a chat message, and is read back as its words; - `item_reference` and `compaction` input items, whose words are kept elsewhere, are refused rather than silently dropped. `mcp_call` items stay as OpenAI names them, since OpenAI's clients read them; Open Responses would want a prefixed name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
wryme turned out to read a field of the Responses stream (
output_item) that no server sends, and its tests passed because their fixtures agreed with the bug. The same could be true of hum: openai-server's tests checked it against what we thought OpenAI sends, not against the spec. This holds the hive to both specs it speaks, and makes the tests check every shape against them.Specs
bun spec/fetch.tspulls both again at their pinned commits.Checked in every test
test/spec.ts(ajv) validates against either spec. The e2e client checks every request it sends against the route's request schema and every answer, error body and stream event it hears against both specs (each event by itstype). The seven HTTP cases of the Open Responses compliance suite run as the suite writes them, with its validators.What the specs found
object: "response.deleted""response""response.deleted"system_fingerprintis a string, optionalnulleverywhereaccess_programs,cache_write_tokens(OpenAI) andcompleted_at,background,service_tier,top_logprobs, the penalties,max_tool_calls,safety_identifier,prompt_cache_key, numerictemperature/top_p(Open Responses)description,parameters,strictresponse.failederror.codeis one of a fixed listtypemcp_call.erroris an object (mcp_tool_execution_error, …)mcp_calls; read back as its wordsitem_reference/compactioninput itemsKept on purpose
mcp_callitems keep OpenAI's name, since OpenAI's clients read them; Open Responses would want a prefixed type, so its check sets them aside.reasoning_contentandmcp_callson chat messages,X-Hum-Sid./responses/compact.Testing
Official compliance runner
The Open Responses runner (
bin/compliance-test.ts@ 8907bc3), run unchanged against this branch under the e2e harness (real humd, scripted upstream,gpt-lab): every case for what hum serves passes, 8 of 8: Basic Text Response, Streaming Response, System Prompt, Multi-turn Conversation, Image Input, Tool Calling, Assistant Message Phase, Response Output Phase Schema. The other 9 need what hum does not serve: the WebSocket transport (6) and/responses/compact(3).