Profile variation submissions leak baseline profile configuration #1613

Please do not report security vulnerabilities here. Follow SECURITY.md instead.
Redact secrets and sensitive data from examples, logs, and screenshots.

Summary

Submitting a run with more than one profile can fail because configuration copied from the baseline profile is sent as top-level run configuration and then applied to every profile variation.

The observed failure involves lifecycle resources: the Portal copies the baseline profile's resource bindings into the request-level resources field. During API fan-out, the API resolves each profile independently but also merges the shared request-level resources into every variation. If another profile controls different resource bindings or parameter values, validation rejects the submission with an error similar to:

Profile "<profile-id>" controls these fields. Either omit them or match the profile values.

This violates the intended separation of concerns. Clients should submit the shared run definition and profile references; the API should create a fresh request for each profile and apply that profile/version without configuration leaking from the baseline or another variation.

Affected areas: API profile fan-out, Portal run submission, and CLI/API contract consistency.

Steps to reproduce

1. Create a baseline profile with a lifecycle resource and profile-managed parameter values.
2. Create a second profile with different resource bindings or different managed values for the same resource.
3. In the Portal, create a new run, select the first profile as the base, and add the second profile as a variation.
4. Submit the run.
5. Observe that the API rejects the comparative submission because baseline resource values are treated as shared run-level overrides and conflict with the second profile.

The same class of failure can be triggered by any API or CLI client that submits profile-derived configuration as shared top-level fields during profile variation fan-out.

Expected behavior

In profile-variation mode, the API should:

1. Treat the scenario, criteria, gates, occurrences, codebase, and other explicitly shared inputs as the common run definition.
2. Create a fresh request instance for the baseline and for every variation.
3. Resolve all profile-managed configuration from that request's selected profile/version only.
4. Prevent profile-managed configuration from leaking between variations, regardless of whether submission originates from the Portal, CLI, or direct API usage.
5. Reject ambiguous profile-managed top-level fields in variation mode with a clear validation error, rather than merging baseline-derived values into every profile.

Actual behavior

The Portal materializes the baseline profile's resources into the top-level run payload. The API accepts that field in variation mode and applies it while resolving every profile. A variation whose profile controls different values is rejected before any comparative runs are created.

The API already rejects several profile-controlled top-level fields in variation mode (model, mcpServers, skills, and extensions), but resources are still accepted and merged. The complete variation-mode contract should be audited so profile application remains entirely API-owned and consistent across clients.

Environment and logs

• Hosted Scope Portal (/runs/new)
• Browser submission with a baseline profile plus one additional profile variation
• Consistently reproduced when the profiles have conflicting managed resource configuration
• API error: Profile "<profile-id>" controls these fields. Either omit them or match the profile values.
• No secrets, profile identifiers, or run identifiers included