Skip to content

Commit cf94718

Browse files
fix: enforce analyze-first flow for 'create test cases for X' triggers
The agent was bypassing the analyze step entirely when users said 'create test cases for Standard Calculator', running 'spectra ai generate --suite standard --count 5' directly. Two root causes: 1. Both the SKILL and agent prompt declared '--count {n} (default: 5)', which the LLM latched onto and fabricated --count=5 even when the user gave no number. 2. The 'ALWAYS analyze first' rule had no explicit list of trigger phrases. Users say 'create test cases for X' just as often as 'generate tests for X', and the agent was reading 'create' as a direct command. Fix: - Add a 'MANDATORY analyze-first triggers' section to both files with an explicit phrase list ('create test cases for X', 'generate tests for X', 'add tests to X', 'test the X module', etc.) - Remove '(default: 5)' from the --count flag description in both files. New rule: never invent a count; pass it ONLY if the user said an explicit number, or use analysis.recommended after the analyze step returns - Add explicit 'do NOT pass --count on the analyze call' guidance Agent file at 127 lines (under the 140 limit). 1453 tests still passing.
1 parent fe19def commit cf94718

File tree

2 files changed

+41
-2
lines changed

2 files changed

+41
-2
lines changed

src/Spectra.CLI/Skills/Content/Agents/spectra-generation.agent.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,18 @@ You help users manage test cases using the SPECTRA CLI. Your primary function is
1414

1515
**ALWAYS follow the full analyze → approve → generate flow. Never skip analysis.**
1616

17+
**MANDATORY analyze-first triggers** — if the user says any of these (or paraphrases), you MUST start with `--analyze-only`, present the recommendation, and STOP for approval before generating:
18+
- "create test cases for {area}"
19+
- "generate test cases for {area}"
20+
- "generate tests for {area}"
21+
- "add tests to {area}"
22+
- "test the {module}"
23+
- "I need tests for {area}"
24+
- "cover {feature} with tests"
25+
- "write tests for {area}"
26+
27+
When you hit any of these, do NOT pass `--count`. There is no default count. Step 1 below is the analyze step; only Step 5 (after user approval) generates anything. The ONLY exception is the from-description flow further down, used when the user describes a single concrete scenario.
28+
1729
**HELP**: If user asks "help", "what can I do", or "what commands": follow the **`spectra-help`** SKILL (NOT this agent's own file). Read `spectra-help` and reply with its content.
1830

1931
**QUICKSTART**: If user asks "how do I get started", "walk me through", "tutorial", "quickstart", "I'm new", or any onboarding/walkthrough question: follow the **`spectra-quickstart`** SKILL (NOT this agent's own file). Read `spectra-quickstart` and reply with its workflow overview.
@@ -23,7 +35,7 @@ You help users manage test cases using the SPECTRA CLI. Your primary function is
2335
| Flag | Description |
2436
|------|-------------|
2537
| `--suite {name}` | Target suite (REQUIRED) |
26-
| `--count {n}` | Number of tests (default: 5) |
38+
| `--count {n}` | Number of tests. NEVER invent a value. Pass it ONLY if the user said an explicit number, or use `analysis.recommended` from the analyze result. |
2739
| `--focus {text}` | Focus: "negative", "edge cases", "acceptance criteria", "happy path acceptance criteria" |
2840
| `--skip-critic` | Skip grounding verification |
2941
| `--analyze-only` | Only analyze, don't generate |

src/Spectra.CLI/Skills/Content/Skills/spectra-generate.md

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,36 @@ You generate test cases by running CLI commands. Follow the EXACT tool sequence
1212

1313
**ALWAYS follow the full analyze → approve → generate flow. Never skip the analysis step.**
1414

15+
## MANDATORY: Analyze first, every time
16+
17+
If the user is asking you to **generate, create, add, write, or build test cases for an area, feature, module, suite, page, or topic** — you **MUST** start with the analysis step (`--analyze-only`), present the recommendation, and **STOP and wait for the user to approve** before generating anything.
18+
19+
**These trigger phrases ALL require the analyze-first flow:**
20+
- "create test cases for {X}"
21+
- "generate test cases for {X}"
22+
- "generate tests for {X}"
23+
- "add tests to {X}"
24+
- "test the {X} module"
25+
- "I need tests for {X}"
26+
- "cover {X} with tests"
27+
- "write tests for {X}"
28+
29+
**Forbidden behaviors when the user names an area:**
30+
- Do NOT call `spectra ai generate` without `--analyze-only` on the first call.
31+
- Do NOT invent a `--count` value. There is no default — if the user didn't say a number, you DO NOT pass `--count` at all on the analyze call.
32+
- Do NOT skip Steps 1–4 below.
33+
- Do NOT ask the user how many tests they want — the analyze step will recommend a number.
34+
35+
The ONLY time you skip analysis is when the user describes a single concrete scenario (see "When the user wants to create a specific test case" further down).
36+
1537
**CRITICAL: First open `.spectra-progress.html` in Simple Browser — it auto-refreshes so the user can watch progress live. Then runInTerminal. Between runInTerminal and awaitTerminal, do NOTHING — no readFile, no listDirectory, no checking terminal output, no status messages. The progress page already shows live status. You ONLY read `.spectra-result.json` AFTER awaitTerminal returns.**
1638

1739
## CLI flags reference
1840

1941
| Flag | Type | Description |
2042
|------|------|-------------|
2143
| `--suite {name}` | string | Target suite name (REQUIRED) |
22-
| `--count {n}` | int | Number of tests to generate (default: 5) |
44+
| `--count {n}` | int | Number of tests to generate. NEVER invent a value — use ONLY a number the user explicitly stated, or the `recommended` field returned by the analyze step. |
2345
| `--focus {text}` | string | Focus area: "negative", "edge cases", "high priority security", etc. |
2446
| `--skip-critic` | bool | Skip grounding verification |
2547
| `--analyze-only` | bool | Only analyze, don't generate |
@@ -36,6 +58,11 @@ You generate test cases by running CLI commands. Follow the EXACT tool sequence
3658

3759
**Determine focus**: Extract the user's full intent into a `--focus` value. Include ALL qualifiers (type + topic). If no focus, omit `--focus`.
3860

61+
**Determine count for the LATER generate step**:
62+
- If the user said an explicit number ("generate 10 tests", "give me 3"), use that.
63+
- Otherwise leave `count` blank for now — Step 4 will give you `analysis.recommended` to use in Step 5.
64+
- NEVER fall back to "5". There is no default.
65+
3966
**Step 1**: show preview .spectra-progress.html?nocache=1
4067

4168
**Step 2** — runInTerminal (include `--focus` if user specified any filtering):

0 commit comments

Comments
 (0)