diff --git a/.claude/templates/coding_prompt.template.md b/.claude/templates/coding_prompt.template.md index 832eb5996..c0508cd3b 100644 --- a/.claude/templates/coding_prompt.template.md +++ b/.claude/templates/coding_prompt.template.md @@ -86,6 +86,13 @@ Implement the chosen feature thoroughly: **CRITICAL:** You MUST verify features through the actual UI. +**No browser UI?** For CLI tools, scripts, libraries, and API-only services, verify through bash +instead of the browser: plan the expected exit codes, output, and side effects first, run the +program with simulated user input (stdin, arguments, input files), and check stdout, stderr, the +exit code, and produced files against that plan -- including at least one invalid input. Save the +scenario as `tests/feature__test.sh` (run with `bash tests/feature__test.sh`) so testing +agents can re-run it for regression checks. + Use `playwright-cli` for browser automation: - Open the browser: `playwright-cli open http://localhost:PORT` diff --git a/.claude/templates/testing_prompt.template.md b/.claude/templates/testing_prompt.template.md index ee6a08fa2..8606bcad1 100644 --- a/.claude/templates/testing_prompt.template.md +++ b/.claude/templates/testing_prompt.template.md @@ -9,7 +9,7 @@ You are assigned to test the following features: {{TESTING_FEATURE_IDS}} ### Workflow for EACH feature: 1. Call `feature_get_by_id` with the feature ID 2. Read the feature's verification steps -3. Test the feature in the browser +3. Test the feature the way a real user would (browser for web UIs, bash for CLIs/scripts/libraries) 4. Call `feature_mark_passing` or `feature_mark_failing` 5. Move to the next feature @@ -25,7 +25,16 @@ Use the feature_get_by_id tool with feature_id= ### STEP 2: VERIFY THE FEATURE -**CRITICAL:** You MUST verify the feature through the actual UI using browser automation. +**CRITICAL:** You MUST verify the feature by actually running the application the way a real user would. Reading the code is NOT verification. + +First, decide how this feature is exercised. Check `app_spec.txt`, the feature's verification steps, and the project layout: + +- **Web UI** (pages, forms, buttons, anything opened in a browser) -> follow **2A: Browser Automation** +- **No browser UI** (CLI tool, bash/Python/Node script, data pipeline, library/SDK, background worker, API-only service) -> follow **2B: Command-Line Verification** + +If a feature has both (e.g. a CLI that also serves a dashboard), verify each part with the matching method. + +#### 2A: Browser Automation (Web UI features) For the feature returned: 1. Read and understand the feature's verification steps @@ -58,6 +67,59 @@ For the feature returned: **Note:** Screenshots and snapshots save to files. Read the file to see the content. + +#### 2B: Command-Line Verification (CLI / script / library / non-UI features) + +Do NOT open a browser for these features. Exercise the program through bash, exactly as a user or calling program would, and judge the result only from observable evidence: **exit code, stdout, stderr, and the files/data it produces.** + +1. **Plan the success criteria BEFORE running anything.** Turn each verification step into concrete, checkable test cases: + - The exact command and arguments to run + - The simulated user input (stdin answers, arguments, environment variables, input files) + - The expected exit code (`0` on success, non-zero for documented error cases) + - The expected output (exact text, key lines, or a pattern) and the stream it goes to (stdout vs stderr) + - The expected side effects (files created/modified, database rows, generated output contents) + - At least one **invalid or edge-case input** (empty input, malformed data, missing file, bad argument) and the expected error behavior + +2. **Create input fixtures that mimic real user data** with the file Write tool, e.g. `tests/fixtures/feature__input.txt` and, where useful, `tests/fixtures/feature__expected.txt`. + +3. **Write a scenario script** (with the Write tool) such as `tests/feature__test.sh` that runs each planned test case and checks it. Keep it self-contained and repeatable: + ```bash + #!/usr/bin/env bash + # Feature #: + set -u + FAIL=0 + WORK=$(mktemp -d) # isolate side effects between runs + check() { if eval "$2"; then echo "PASS: $1"; else echo "FAIL: $1"; FAIL=1; fi; } + + # Case 1: valid input supplied on stdin + printf 'alice\n42\ny\n' | timeout 30 ./app --out "$WORK/result.txt" > "$WORK/out" 2> "$WORK/err"; CODE=$? + check "exits 0" '[ "$CODE" -eq 0 ]' + check "prints greeting" 'grep -q "Hello, alice" "$WORK/out"' + check "no errors on stderr" '[ ! -s "$WORK/err" ]' + check "writes result file" 'diff -u tests/fixtures/feature__expected.txt "$WORK/result.txt"' + + # Case 2: invalid input fails cleanly + timeout 30 ./app --out "$WORK/result.txt" < /dev/null > "$WORK/out" 2> "$WORK/err"; CODE=$? + check "rejects empty input" '[ "$CODE" -ne 0 ]' + check "explains the error" 'grep -qi "input" "$WORK/err"' + + rm -rf "$WORK" + exit $FAIL + ``` + Adapt the command to the project (`python main.py`, `node cli.js`, `cargo run --`, a library driver script that imports the module and calls its public API, etc.). Wrap anything that could hang waiting for input in `timeout`. For a long-running worker or service, start it in the background inside the script, exercise it (send a request, drop a file in a watched folder, enqueue a job), check its logs, then stop it. + +4. **Run it and read ALL of the output:** `bash tests/feature__test.sh`. Do not rely on the exit code alone -- read the PASS/FAIL lines and look at the captured stdout/stderr for any test that fails. + +5. **Judge the result strictly.** The feature FAILS if any of these happen: + - An exit code, output, or side effect differs from what you planned + - A traceback, stack trace, panic, or unexpected warning appears on the success path + - An invalid input crashes the program instead of failing with a clear message and a non-zero exit code + - The program hides failures (exits `0` after an error, prints nothing, or only logs somewhere invisible). Missing error reporting is itself a bug: when you fix the feature, add clear stderr messages and correct exit codes to the program + +6. **Keep the evidence.** Commit the scenario script and fixtures so later regression runs can re-use them, and also run the project's own test suite (pytest, `npm test`, `go test`, etc.) if it has one. + +"It ran without crashing" is NOT a pass. A feature only passes when every planned criterion is met. + ### STEP 3: HANDLE RESULTS #### If the feature PASSES: @@ -74,18 +136,19 @@ A regression has been introduced. You MUST fix it: ``` 2. **Investigate the root cause:** - - Check console errors - - Review network requests + - Web UI: check console errors and review network requests + - CLI / non-UI: read the captured stderr, tracebacks, and exit codes from your scenario script - Examine recent git commits that might have caused the regression 3. **Fix the regression:** - Make the necessary code changes - - Test your fix using browser automation + - Test your fix with the same method you used in Step 2 (browser automation or your scenario script) - Ensure the feature works correctly again 4. **Verify the fix:** - Run through all verification steps again - - Take screenshots and read them to confirm the fix + - Web UI: take screenshots and read them to confirm the fix + - CLI / non-UI: re-run `bash tests/feature__test.sh` and confirm every case prints PASS 5. **Mark as passing after fix:** ``` @@ -99,7 +162,7 @@ A regression has been introduced. You MUST fix it: - [Describe what was broken] - [Describe the fix] - - Verified with browser automation" + - Verified with browser automation / scenario script" ``` --- @@ -124,6 +187,13 @@ Use `playwright-cli` commands for browser interaction. Key commands: - `playwright-cli console` - Check for JS errors - `playwright-cli close` - Close browser (always do this when done) +### Command-Line Verification (Bash) +For CLIs, scripts, libraries, and other non-UI features: +- Write a scenario script with the Write tool (e.g. `tests/feature__test.sh`) plus input/expected-output fixtures +- Run it with `bash tests/feature__test.sh` +- In the script, feed simulated user input via stdin (`printf '...' | ./app`, `./app < input.txt`), arguments, and environment variables +- Capture stdout, stderr, and the exit code separately and check each against your planned criteria + --- ## IMPORTANT REMINDERS @@ -131,15 +201,14 @@ Use `playwright-cli` commands for browser interaction. Key commands: **Your Goal:** Test each assigned feature thoroughly. Verify it still works, and fix any regression found. Process ALL features in your list before ending your session. **Quality Bar:** -- Zero console errors - All verification steps pass -- Visual appearance correct -- API calls succeed +- Web UI: zero console errors, visual appearance correct, API calls succeed +- CLI / non-UI: expected exit codes, expected stdout/stderr, correct side effects, clean failure on invalid input **If you find a regression:** 1. Mark the feature as failing immediately 2. Fix the issue -3. Verify the fix with browser automation +3. Verify the fix with browser automation (web UI) or your scenario script (CLI / non-UI) 4. Mark as passing only after thorough verification 5. Commit the fix diff --git a/CLAUDE.md b/CLAUDE.md index 6f59910be..e2d9c111f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -120,6 +120,7 @@ python test_security_integration.py # Integration tests (9 tests) python -m pytest test_client.py # Client tests (20 tests) python -m pytest test_dependency_resolver.py # Dependency resolver tests (12 tests) python -m pytest test_rate_limit_utils.py # Rate limit tests (22 tests) +python -m pytest test_prompts.py # Prompt template + migration tests (7 tests) ``` ### React UI diff --git a/prompts.py b/prompts.py index 6031080d2..ae5ceccf1 100644 --- a/prompts.py +++ b/prompts.py @@ -17,7 +17,7 @@ TEMPLATES_DIR = Path(__file__).parent / ".claude" / "templates" # Migration version — bump when adding new migration steps -CURRENT_MIGRATION_VERSION = 1 +CURRENT_MIGRATION_VERSION = 2 def get_project_prompts_dir(project_dir: Path) -> Path: @@ -821,6 +821,73 @@ def _migrate_v0_to_v1(project_dir: Path) -> list[str]: return migrated +# Phrases from the v1 testing prompt (browser-only) and their v2 replacements +# (browser or command-line verification). Only exact matches are replaced, so +# user-customized wording is left untouched. +_V2_TESTING_PHRASES = ( + ( + "3. Test the feature in the browser\n", + "3. Test the feature the way a real user would (browser for web UIs, bash for CLIs/scripts/libraries)\n", + ), + ( + "**Quality Bar:**\n- Zero console errors\n- All verification steps pass\n" + "- Visual appearance correct\n- API calls succeed\n", + "**Quality Bar:**\n- All verification steps pass\n" + "- Web UI: zero console errors, visual appearance correct, API calls succeed\n" + "- CLI / non-UI: expected exit codes, expected stdout/stderr, correct side effects, " + "clean failure on invalid input\n", + ), + ( + "3. Verify the fix with browser automation\n", + "3. Verify the fix with browser automation (web UI) or your scenario script (CLI / non-UI)\n", + ), +) + + +def _get_template_section(name: str, start_pattern: str, end_pattern: str) -> str | None: + """Return the section of a base template from start_pattern up to (not including) end_pattern.""" + template_path = TEMPLATES_DIR / f"{name}.template.md" + try: + template = template_path.read_text(encoding="utf-8") + except (OSError, PermissionError): + return None + match = re.search(f"{start_pattern}.*?(?={end_pattern})", template, flags=re.DOTALL) + return match.group(0) if match else None + + +def _migrate_testing_prompt_to_v2(content: str) -> str: + """Add command-line verification guidance for non-web projects to a testing prompt. + + The STEP 2 section is only replaced when it is the unmodified v1 (browser-only) + text, so project-specific customizations are preserved. + """ + new_step2 = _get_template_section("testing_prompt", r"### STEP 2: VERIFY THE FEATURE", r"### STEP 3:") + if new_step2 and _CLI_TESTING_STEP2 in content: + content = content.replace(_CLI_TESTING_STEP2, new_step2, 1) + + for old_phrase, new_phrase in _V2_TESTING_PHRASES: + content = content.replace(old_phrase, new_phrase, 1) + + return content + + +def _migrate_v1_to_v2(project_dir: Path) -> list[str]: + """Migrate testing_prompt.md to support CLI / non-web verification (issue #136).""" + migrated: list[str] = [] + testing_prompt_path = get_project_prompts_dir(project_dir) / "testing_prompt.md" + if not testing_prompt_path.exists(): + return migrated + try: + content = testing_prompt_path.read_text(encoding="utf-8") + updated = _migrate_testing_prompt_to_v2(content) + if updated != content: + testing_prompt_path.write_text(updated, encoding="utf-8") + migrated.append("Updated testing_prompt.md with command-line verification for non-web projects") + except (OSError, PermissionError) as e: + print(f" Warning: Could not update testing_prompt.md: {e}") + return migrated + + def migrate_project_to_current(project_dir: Path) -> list[str]: """Migrate an existing project to the current AutoForge version. @@ -836,7 +903,8 @@ def migrate_project_to_current(project_dir: Path) -> list[str]: if current < 1: migrated.extend(_migrate_v0_to_v1(project_dir)) - # Future: if current < 2: migrated.extend(_migrate_v1_to_v2(project_dir)) + if current < 2: + migrated.extend(_migrate_v1_to_v2(project_dir)) _set_migration_version(project_dir, CURRENT_MIGRATION_VERSION) return migrated diff --git a/test_prompts.py b/test_prompts.py new file mode 100644 index 000000000..7ff1ed0ec --- /dev/null +++ b/test_prompts.py @@ -0,0 +1,111 @@ +""" +Prompt Tests +============ + +Tests for testing-agent prompt content and the v1 -> v2 prompt migration +that adds command-line verification for non-web projects (issue #136). + +Run with: python -m pytest test_prompts.py -v +""" + +import tempfile +import unittest +from pathlib import Path + +import prompts +from prompts import ( + _CLI_TESTING_STEP2, + CURRENT_MIGRATION_VERSION, + _migrate_testing_prompt_to_v2, + get_testing_prompt, + migrate_project_to_current, +) + +# Minimal reproduction of a project testing_prompt.md scaffolded from the v1 template +V1_TESTING_PROMPT = ( + "## YOUR ROLE - TESTING AGENT\n\n" + "### Workflow for EACH feature:\n" + "1. Call `feature_get_by_id` with the feature ID\n" + "2. Read the feature's verification steps\n" + "3. Test the feature in the browser\n" + "4. Call `feature_mark_passing` or `feature_mark_failing`\n\n" + "---\n\n" + + _CLI_TESTING_STEP2 + + "### STEP 3: HANDLE RESULTS\n\n" + "**Quality Bar:**\n- Zero console errors\n- All verification steps pass\n" + "- Visual appearance correct\n- API calls succeed\n\n" + "**If you find a regression:**\n" + "3. Verify the fix with browser automation\n" +) + + +class TestTestingPromptTemplate(unittest.TestCase): + def test_template_covers_browser_and_cli_verification(self): + prompt = get_testing_prompt(testing_feature_ids=[1, 2]) + self.assertIn("1, 2", prompt) + self.assertIn("#### 2A: Browser Automation", prompt) + self.assertIn("#### 2B: Command-Line Verification", prompt) + # Key steps requested in issue #136 + self.assertIn("Plan the success criteria BEFORE running anything", prompt) + self.assertIn("simulated user input", prompt) + self.assertIn("exit code", prompt) + self.assertIn("bash tests/feature__test.sh", prompt) + + def test_template_step2_is_bounded_by_step3(self): + section = prompts._get_template_section( + "testing_prompt", r"### STEP 2: VERIFY THE FEATURE", r"### STEP 3:" + ) + self.assertIsNotNone(section) + assert section is not None + self.assertIn("#### 2B:", section) + self.assertNotIn("### STEP 3:", section) + + +class TestTestingPromptMigrationV2(unittest.TestCase): + def test_migrates_unmodified_v1_prompt(self): + updated = _migrate_testing_prompt_to_v2(V1_TESTING_PROMPT) + self.assertNotIn(_CLI_TESTING_STEP2, updated) + self.assertIn("#### 2B: Command-Line Verification", updated) + self.assertIn("### STEP 3: HANDLE RESULTS", updated) + self.assertNotIn("3. Test the feature in the browser\n", updated) + self.assertIn("CLI / non-UI: expected exit codes", updated) + self.assertIn("or your scenario script (CLI / non-UI)", updated) + + def test_migration_is_idempotent(self): + once = _migrate_testing_prompt_to_v2(V1_TESTING_PROMPT) + self.assertEqual(once, _migrate_testing_prompt_to_v2(once)) + + def test_current_template_is_unchanged_by_migration(self): + template = (prompts.TEMPLATES_DIR / "testing_prompt.template.md").read_text(encoding="utf-8") + self.assertEqual(template, _migrate_testing_prompt_to_v2(template)) + + def test_customized_step2_is_preserved(self): + custom = V1_TESTING_PROMPT.replace( + "**CRITICAL:** You MUST verify the feature through the actual UI using browser automation.", + "**CRITICAL:** Run `make e2e` to verify.", + ) + updated = _migrate_testing_prompt_to_v2(custom) + self.assertIn("Run `make e2e` to verify.", updated) + self.assertNotIn("#### 2B:", updated) + + def test_migrate_project_to_current_updates_testing_prompt(self): + with tempfile.TemporaryDirectory() as tmp: + project_dir = Path(tmp) + prompts_dir = project_dir / ".autoforge" / "prompts" + prompts_dir.mkdir(parents=True) + (project_dir / ".autoforge" / ".migration_version").write_text("1") + testing_prompt = prompts_dir / "testing_prompt.md" + testing_prompt.write_text(V1_TESTING_PROMPT, encoding="utf-8") + + migrated = migrate_project_to_current(project_dir) + + self.assertEqual(len(migrated), 1) + self.assertIn("#### 2B:", testing_prompt.read_text(encoding="utf-8")) + version = (project_dir / ".autoforge" / ".migration_version").read_text() + self.assertEqual(version, str(CURRENT_MIGRATION_VERSION)) + # Second run is a no-op + self.assertEqual(migrate_project_to_current(project_dir), []) + + +if __name__ == "__main__": + unittest.main()