Make AI extension prompts deterministic in no-prompt mode - #9720
Open
Glenn Harper (glharper) wants to merge 2 commits into
Open
Make AI extension prompts deterministic in no-prompt mode#9720Glenn Harper (glharper) wants to merge 2 commits into
Glenn Harper (glharper) wants to merge 2 commits into
Conversation
Add explicit non-interactive inputs and safe defaults across the AI extensions, including destructive-operation consent and a maintained prompt contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 758532eb-50d8-4e25-b4c6-ff12e051162e
Glenn Harper (glharper)
requested review from
JeffreyCA,
Rick Winter (RickWinter),
Marina He (hemarina),
Hui Miao (huimiu),
Zhijie Huang (hund030),
Wei Meng (m5i-work),
Richard Park (richardpark-msft),
tg-msft,
John Miller (therealjohn),
Travis Angevine (trangevi),
JianW (v1212) and
Victor Vazquez (vhvb1989)
as code owners
August 24, 2026 22:05
|
Azure Pipelines: Successfully started running 2 pipeline(s). 19 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds deterministic, documented non-interactive behavior across the AI extensions.
Changes:
- Adds headless manifest, eval, and ACR inputs.
- Requires explicit consent for destructive or overwrite operations.
- Documents non-interactive prompt equivalents and adds regression tests.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
azure.ai.projects/README.md |
Links the shared reference. |
foundry_provisioning_provider.go |
Requires force for headless teardown. |
foundry_provisioning_provider_destroy_confirm_test.go |
Tests teardown consent. |
azure.ai.agents/README.md |
Links automation guidance. |
parameters.go |
Reads manifest parameters from the environment. |
parameters_test.go |
Tests headless parameter resolution. |
exterrors/codes.go |
Adds delete-consent error code. |
init.go |
Adds ACR selection and overwrite help. |
init_test.go |
Tests ACR flag registration. |
init_from_code.go |
Honors force during overwrite confirmation. |
init_from_code_test.go |
Tests overwrite consent. |
init_foundry_resources_helpers.go |
Stabilizes and persists ACR selection. |
init_foundry_resources_helpers_test.go |
Tests ACR selection behavior. |
init_foundry_project_setup.go |
Propagates the ACR selector. |
init_adopt.go |
Propagates the selector during adoption. |
eval.go |
Centralizes eval context flags. |
eval_update.go |
Adds deterministic eval context. |
eval_test.go |
Tests shared eval flags. |
eval_show.go |
Adds deterministic eval context. |
eval_run.go |
Adds deterministic eval context. |
eval_list.go |
Adds deterministic eval context. |
eval_generate.go |
Reuses shared context flags. |
eval_generate_test.go |
Updates shared flag test setup. |
delete.go |
Requires force for headless deletion. |
delete_test.go |
Tests deletion consent. |
ai-non-interactive.md |
Documents prompt alternatives. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // Prompt for value | ||
| isRequired := property.Required != nil && *property.Required | ||
|
|
||
| if envValue, found := os.LookupEnv(property.Name); found { |
Contributor
There was a problem hiding this comment.
Fixed in 25c051ce3. Manifest parameters now read only from the dedicated AZD_AI_AGENT_MANIFEST_PARAMETER_<NAME> channel, with regression coverage proving an ambient GITHUB_TOKEN is ignored.
Comment on lines
+1878
to
+1879
| cmd.Flags().StringVar(&flags.acrConnection, "acr-connection", "", | ||
| "Foundry Azure Container Registry connection name to use for an existing project") |
Contributor
There was a problem hiding this comment.
Fixed in 25c051ce3. --acr-connection now blocks early reuse, with coverage in TestAgentDefiningFlagsSet.
Comment on lines
1932
to
+1934
| cmd.Flags().BoolVar(&flags.force, "force", false, | ||
| "Overwrite an input manifest that already lives inside the generated src tree without prompting. "+ | ||
| "Required together with --no-prompt when init would otherwise need confirmation.") | ||
| "Overwrite existing agent definitions or an input manifest inside the generated src tree without prompting. "+ | ||
| "Required together with --no-prompt when init would otherwise need overwrite confirmation.") |
Contributor
There was a problem hiding this comment.
Fixed in 25c051ce3. --force now blocks early reuse so overwrite handling is reachable, with a reuse-predicate regression case.
github-actions
Bot
added
area/extensions
Co-authored-by: glharper <64209257+glharper@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_test.go:83
- These checks only prove that flags exist and that a helper copies values; they never exercise
run,list,show, orupdateagainst no-prompt context resolution. The linked issue requires regressions for each finding, so a command could stop forwardingnoPromptor the explicit endpoint while these tests still pass. Add command/context tests that assert no Prompt RPC occurs, a missing endpoint fails outside a project, and explicit context reaches resolution.
require.NotNil(t, cmd.Flags().Lookup("agent"))
endpointFlag := cmd.Flags().Lookup("project-endpoint")
require.NotNil(t, endpointFlag)
assert.Equal(t, "p", endpointFlag.Shorthand)
|
|
||
| // Prompt for value | ||
| isRequired := property.Required != nil && *property.Required | ||
| parameterEnvVar := manifestParameterEnvPrefix + property.Name |
Comment on lines
+166
to
+170
| if len(enumValues) > 0 && !slices.Contains(enumValues, envValue) { | ||
| return nil, fmt.Errorf( | ||
| "environment variable %s has invalid value %q for parameter '%s'; expected one of %v", | ||
| parameterEnvVar, | ||
| envValue, |
Contributor
azure.ai.projects PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9720/azure-ai-projects.zip"
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--acr-connectionand persisted registry state--forcefor init overwritesValidation
go test ./... -count=1(azure.ai.agents)go test ./... -count=1(azure.ai.projects)golangci-lint run --concurrency 1 ./...(both extensions)mage preflightstatic checks passed; local core unit/playback completion was blocked by missing Docker and restrictive PowerShell execution policyFixes #9570