Skip to content

Add OpenApiMockResponsePlugin to mock APIs from OpenAPI specs - #2036

Open
waldekmastykarz wants to merge 7 commits into
dotnet:mainfrom
waldekmastykarz:openapi-mock-slm
Open

waldekmastykarz wants to merge 7 commits into
dotnet:mainfrom
waldekmastykarz:openapi-mock-slm

Conversation

@waldekmastykarz

Copy link
Copy Markdown
Collaborator

Closes #531

Adds OpenApiMockResponsePlugin, which mocks an API based on its OpenAPI spec.

Configuration

{
  "plugins": [
    {
      "name": "OpenApiMockResponsePlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/DevProxy.Plugins.dll",
      "configSection": "openApiMock"
    }
  ],
  "openApiMock": {
    "specFile": "openapi.yaml"
  }
}

specFile can be a local path (relative to the config file, watched for changes) or an http(s) URL. JSON and YAML specs are supported.

Behavior

  • Requests are matched against spec paths (path parameters act as wildcards) and operations. Watched requests that don't match the spec pass through to the real API.
  • The response status is the lowest 2xx response defined for the operation, then default, otherwise 200.
  • The response body comes from the first available source:
    1. Examples defined in the spec (media type example/examples)
    2. The configured language model (global languageModel setting), prompted with the request, the operation description, and a simplified schema so values are realistic (for example, givenName and mail)
    3. Values generated from the schema: example → default → enum → format-aware value → type-based value. $ref, allOf, oneOf, and anyOf are followed, arrays get one item, and recursion is cut off.
  • Language model output is validated as JSON and normalized to the expected type (array or object). Invalid output falls back to the schema-based values.
  • Generated bodies are cached in memory per operation (method + path template), so subsequent requests don't call the model again. The cache is cleared when the spec changes or the proxy restarts.
  • Adds CORS headers when the request has an Origin header and merges RateLimitingPlugin headers.

Other changes

Testing

Tested manually with a detached proxy against a sample spec:

  • With Ollama (llama3.2): realistic objects and arrays; repeated requests are served from the cache (~11s for the first request, ~0.1s after that)
  • With no language model: schema-based fallback; spec examples are used as-is; 201/204 responses work; text/plain works; unmatched URLs pass through
  • Spec loaded from a URL; editing a local spec reloads it and clears the cache

…otnet#531

Generates responses from spec examples, the configured language model,
or schema-based fallback values, and caches them per operation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical cache-isolation and path/server-matching defects can serve stale or incorrect mock bodies or prevent valid OpenAPI requests from being mocked.

9 open issues
What changed in this PR

Adds OpenApiMockResponsePlugin to Dev Proxy, enabling API mocking from OpenAPI specifications with optional language-model-generated responses.

Changes:

  • Loads local or remote JSON/YAML specs and matches requests to API operations.
  • Generates responses from examples, a language model, or schema-based sample values.
  • Caches generated bodies per operation and adds local-spec reload support, CORS headers, and rate-limit header integration.
File Description
schemas/​v4.1.0/​openapimockresponseplugin.schema.json Defines configuration validation for specFile.
DevProxy/​prompts/​openapi_mock_response.prompty Prompts the language model to generate realistic JSON responses.
DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs Generates schema-based samples and simplified model input.
DevProxy.Plugins/​Mocking/​OpenApiMockResponseSpecLoader.cs Connects local spec loading and file watching to the plugin.
DevProxy.Plugins/​Mocking/​OpenApiMockResponsePlugin.cs Implements spec loading, request matching, response generation, and caching.

Review effort: Balanced

- Use ordinal cache keys and publish spec + cache as one snapshot
- Prefer concrete paths over templated ones when matching requests
- Return JSON null for null-only schemas
- Accept scalar language model responses and validate scalar types
- Keep generated numbers within minimum/maximum without overflow
- Don't cache failed generations

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Schema composition handling drops required nested and enclosing properties and ignores later non-object constraints, allowing invalid fallback responses to be generated and cached.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Preserve enclosing schema constraints when selecting a branch

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:92

Selecting a oneOf or anyOf branch discards the enclosing schema's properties and constraints. For example, an object with a required id property and a branch defining name generates only {"name":"string"} when the language model is unavailable, omitting the required id. Generate from the combination of the enclosing schema and the selected branch rather than the branch alone.

Medium severity Combine allOf constraints when generating non-object samples

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:122

For non-object allOf schemas, retaining the first generated value ignores constraints in later branches. With allOf: [{type: integer}, {type: integer, minimum: 1}], the branches generate 0 and 1, but this returns 0, which violates the response schema and is then cached. Combine the branches' constraints before generating a scalar or array sample instead of selecting the first branch's value.

Medium severity Recursively merge overlapping objects in allOf schemas

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:251

When two allOf branches define required fields under the same object property—for example, profile.givenName and profile.mail—this assignment replaces the entire first profile object. The fallback then returns only mail and caches a response missing givenName; the same helper also loses nested properties in the language-model schema description. Recursively merge overlapping objects instead of replacing them.

9 resolved since last review

Review effort: Balanced

- Merge enclosing properties with the selected oneOf/anyOf branch
- Combine allOf constraints for non-object values
- Recursively merge nested objects and required properties

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved schema-generation, validation, and serialization issues can produce and cache invalid mock responses, including violated allOf constraints, incorrect null handling, and JSON bodies mislabeled as XML.

1 open issue
6 previously missed issues

In code that hasn't changed since last review

Medium severity Serialize mock responses according to the selected media type

DevProxy.Plugins/​Mocking/​OpenApiMockResponsePlugin.cs:335

For an operation whose only response media type is application/xml and whose schema is an object, GetMediaType selects XML but this serializes the sample as JSON. The client receives a JSON body labeled as XML and cannot parse the mocked response. Serialize according to the selected media type, or pass through unsupported encodings instead of returning a malformed mock.

Medium severity Reject non-null model output for null-only schemas

DevProxy.Plugins/​Mocking/​OpenApiMockResponsePlugin.cs:406

A null-only schema reaches this default branch, so any non-null JSON returned by the language model is accepted and cached. This bypasses the corrected schema fallback that emits the JSON value null. Reject model output for the "null" expected type so it falls back to that valid body.

Medium severity Merge allOf constraints before generating shared property values

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:127

Generating each allOf branch independently and then merging the samples can violate earlier branches. For example, two object branches defining count as an integer with minimum: 1 and maximum: 10 generate {"count":1} and {"count":0}; this merge returns {"count":0}, which violates the combined schema and is cached as the mock response. Combine constraints for shared properties before generating their values rather than overwriting previously generated scalar values.

Medium severity Intersect enum values across allOf branches

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:166

Keeping only the first enum does not satisfy allOf: branches enum: [a, b] and enum: [b, c] generate a, although only b is permitted. Intersect enum constraints using JSON value equality, handle empty intersections explicitly, and ensure selected examples or defaults satisfy the intersection.

Medium severity Preserve null values as valid array items

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:218

A null sample can be a valid array item, not just a recursion cutoff. For type: array, items: { type: "null" }, and minItems: 1, this returns [] instead of [null], producing a response outside the schema. Preserve the item when its schema explicitly has the null type.

Medium severity Preserve stricter constraints when combining allOf schemas

DevProxy.Plugins/​Mocking/​OpenApiSchemaSampleGenerator.cs:356

The simplified schema treats allOf constraints as last-write-wins: integer branches with minimum: 10 followed by minimum: 1 describe a minimum of 1 instead of 10. Model responses satisfying only that weaker bound pass the top-level type check and are cached. Preserve the conjunction when describing allOf, including constraints on shared properties, rather than overwriting constraints in branch order.

Review effort: Balanced

- Pass through media types that can't be generated, like XML, unless
  the spec defines a string example
- Skip the language model for null-only schemas
- Resolve nested allOf when combining constraints
- Keep null array items for null-only item schemas
- Keep the stricter minimum/maximum when describing allOf schemas

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@waldekmastykarz

Copy link
Copy Markdown
Collaborator Author

Responses to the issues listed in the review summaries (they weren't posted as threads):

Fixed:

  • oneOf/anyOf now keep the enclosing schema's properties (3b676de).
  • Non-object allOf combines constraints from all branches, including nested allOf (3b676de, cb0e4c2).
  • Overlapping objects in allOf are merged recursively, and required lists are combined (3b676de).
  • Non-JSON media types: when the spec has no string example (for example, an XML schema without an example), requests now pass through instead of getting a JSON body labeled as XML (cb0e4c2).
  • Null-only schemas skip the language model and return null (cb0e4c2).
  • items: { type: 'null' } generates [null] (cb0e4c2).
  • The schema description sent to the model keeps the stricter minimum/maximum across allOf, including on shared properties (cb0e4c2).

Not changing:

  • Merge allOf constraints for shared property values and intersect enums across allOf: the schema-based fallback is meant to produce basic, best-effort values for when no language model is available. Fully solving allOf constraints (combining per-property constraints, intersecting enums, checking examples and defaults against the result) means building a JSON Schema constraint solver, which is out of scope here. These are corner cases, and users can always add explicit examples to the spec, which take precedence.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed to assess omitted findings involving lost schema constraints and response-format mismatches that can produce invalid mocked responses.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

- Generate text/* bodies only from string schemas or string examples,
  pass other text responses through
- Include length, pattern, multipleOf and array constraints in the
  schema sent to the language model
- Generate minItems items and honor multipleOf in fallback samples
- Add tests

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@waldekmastykarz

Copy link
Copy Markdown
Collaborator Author

Follow-up on the latest review summary (it had no inline threads):

Fixed in this push:

  • text/* response format: a text/* body is now generated only from a string schema or a string example. Other text responses, like an object schema under text/plain, pass through. This matches the existing handling of non-JSON media types without an example.
  • Constraints in the prompt: the schema sent to the language model now includes minLength/maxLength, pattern, multipleOf, minItems/maxItems and uniqueItems, and the prompt asks the model to respect them. Validating model output against the full schema will be tracked in a separate issue.
  • Fallback generator: it now generates minItems items (capped by maxItems) and honors multipleOf. pattern stays best-effort.
  • Added OpenApiMockResponsePluginTests covering the above.

Not changing:

  • default-only responses return 200: default has no status code, so 200 is a reasonable, predictable choice for a mock. Responses with an explicit 2xx status code still use that code.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The generator drops schema constraints, produces duplicate items in unique arrays, and can stall requests with billions of increments, while AI completion and fallback behavior remain untested.

4 open issues

Review effort: Balanced

…ApiMockResponsePlugin

- Keep multipleOf, minItems, maxItems, uniqueItems and pattern when combining allOf
- Compute integer-compatible multiples directly instead of searching
- Generate distinct items for uniqueItems arrays (best-effort)
- Test language model output handling and caching with a fake client

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The unique-items array generator inserts an item when maxItems is zero, violating the schema’s upper bound and requiring a guard and regression test.

1 open issue
4 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because enum handling and allOf merging can weaken schema constraints, allowing invalid generated responses to be cached.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@waldekmastykarz

Copy link
Copy Markdown
Collaborator Author

Re: the latest review summary ("enum handling and allOf merging can weaken schema constraints"). This repeats findings we already pushed back on in an earlier round. The schema-based generator is a best-effort fallback, used only when there's no spec example and no language model. Full enum intersection and per-property constraint merging across allOf are out of scope for it. Validating generated output against the schema is tracked as a separate follow-up. No further changes for this one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mock API by pointing to Open API spec

2 participants