|
| 1 | +# Implementation Plan: From-Description Chat Flow & Doc-Aware Manual Tests |
| 2 | + |
| 3 | +**Branch**: `033-from-description-chat-flow` | **Date**: 2026-04-10 | **Spec**: [spec.md](./spec.md) |
| 4 | + |
| 5 | +## Summary |
| 6 | + |
| 7 | +Three-part feature: (1) update `spectra-generate` SKILL with a dedicated single-test "from-description" flow and an intent routing table, (2) update the `spectra-generation` agent prompt with explicit intent-classification rules, (3) enhance `UserDescribedGenerator` to load relevant docs and acceptance criteria as best-effort formatting context — populating `source_refs` and `criteria` on the resulting test while keeping `grounding.verdict: manual`. |
| 8 | + |
| 9 | +## Technical Context |
| 10 | + |
| 11 | +**Language/Version**: C# 12, .NET 8 |
| 12 | +**Primary Dependencies**: Spectra.CLI (existing), Spectra.Core (TestCase, GroundingMetadata, AcceptanceCriterion) |
| 13 | +**Storage**: File-based — embedded SKILL/agent `.md` resources in `Spectra.CLI`; SHA-256 hashes computed at install time |
| 14 | +**Testing**: xUnit (`Spectra.CLI.Tests`) |
| 15 | +**Target Platform**: Cross-platform .NET CLI |
| 16 | +**Project Type**: CLI (single project) |
| 17 | +**Constraints**: Best-effort doc/criteria loading must not block or fail the command. Doc context capped at 3 docs × 8000 chars. |
| 18 | +**Scale/Scope**: ~3 source files modified, 1 SKILL md file, 1 agent md file, ~10 new tests, 9 doc files updated. |
| 19 | + |
| 20 | +## Constitution Check |
| 21 | + |
| 22 | +No constitution file. Standard CLAUDE.md guidelines apply: no unnecessary refactors, only test the changed paths, prefer small focused changes. |
| 23 | + |
| 24 | +## Project Structure |
| 25 | + |
| 26 | +### Documentation (this feature) |
| 27 | + |
| 28 | +```text |
| 29 | +specs/033-from-description-chat-flow/ |
| 30 | +├── spec.md |
| 31 | +├── plan.md |
| 32 | +├── tasks.md |
| 33 | +├── checklists/ |
| 34 | +│ └── requirements.md |
| 35 | +└── (no contracts/ — internal CLI feature, no API) |
| 36 | +``` |
| 37 | + |
| 38 | +### Source Code (touched paths) |
| 39 | + |
| 40 | +```text |
| 41 | +src/Spectra.CLI/ |
| 42 | +├── Commands/Generate/ |
| 43 | +│ ├── UserDescribedGenerator.cs # MODIFIED: add documentContext + criteriaContext params, refactor prompt builder for testability |
| 44 | +│ └── GenerateHandler.cs # MODIFIED: ExecuteFromDescriptionAsync loads doc + criteria context, populates source_refs |
| 45 | +└── Skills/Content/ |
| 46 | + ├── Skills/spectra-generate.md # MODIFIED: add "create a specific test case" section + routing table |
| 47 | + └── Agents/spectra-generation.agent.md # MODIFIED: add Test Creation Intent Routing section |
| 48 | +
|
| 49 | +tests/Spectra.CLI.Tests/ |
| 50 | +├── Commands/Generate/ |
| 51 | +│ └── UserDescribedGeneratorTests.cs # NEW: prompt-building tests |
| 52 | +└── Skills/ |
| 53 | + └── GenerateSkillContentTests.cs # NEW: SKILL/agent content assertions |
| 54 | +``` |
| 55 | + |
| 56 | +## Phases |
| 57 | + |
| 58 | +### Phase 0 — Research / discovery |
| 59 | + |
| 60 | +No external research needed. All primitives exist: |
| 61 | +- `SourceDocumentLoader.LoadAllAsync(basePath, maxDocuments, maxContentLengthPerDoc, ct)` already supports caps. |
| 62 | +- `LoadCriteriaContextAsync` (private static in `GenerateHandler`, line 1943) is the criteria primitive — promote to internal/static-helper-callable from the from-description branch. |
| 63 | +- `SkillContent` / `AgentContent` already load embedded resources via `SkillResourceLoader`. SHA-256 hashes are computed at install time, not stored — so editing the `.md` resources is sufficient; no manifest table to regenerate. |
| 64 | + |
| 65 | +### Phase 1 — SKILL & agent content (no code changes beyond .md files) |
| 66 | + |
| 67 | +1. Add new section to `Skills/Content/Skills/spectra-generate.md`: |
| 68 | + - Heading: `## When the user wants to create a specific test case` |
| 69 | + - Numbered Step 1..5 sequence (open progress page → runInTerminal → awaitTerminal → readFile → present). |
| 70 | + - Command line: `spectra ai generate --suite {suite} --from-description "{description}" --context "{context}" --no-interaction --output-format json --verbosity quiet`. |
| 71 | + - Explicit "Do NOT run analysis. Do NOT ask how many tests. Always 1 test." line. |
| 72 | + - Routing table mapping intent signal → flow. |
| 73 | + |
| 74 | +2. Add new section to `Skills/Content/Agents/spectra-generation.agent.md`: |
| 75 | + - Heading: `## Test Creation Intent Routing`. |
| 76 | + - Three intent classes (Intent 1: explore area → `--focus`, Intent 2: specific test → `--from-description`, Intent 3: from suggestions → `--from-suggestions`) with examples and actions. |
| 77 | + - Ambiguous-intent rule: topic-vs-scenario; never ask about count. |
| 78 | + |
| 79 | +3. Verify SkillContent/AgentContent dictionaries still resolve (smoke test in build). |
| 80 | + |
| 81 | +### Phase 2 — Doc-aware `--from-description` (CLI code) |
| 82 | + |
| 83 | +1. **`UserDescribedGenerator.cs`** — refactor: |
| 84 | + - Add public `static string BuildPrompt(string description, string? context, string suite, IReadOnlyCollection<string> existingIds, string? documentContext, string? criteriaContext)` method that returns the AI prompt string. This makes prompt construction testable without invoking AI. |
| 85 | + - Add optional parameters `string? documentContext = null`, `string? criteriaContext = null`, and `IReadOnlyList<string>? sourceRefPaths = null` to `GenerateAsync(...)`. |
| 86 | + - When `documentContext` is non-null: insert "## Reference Documentation (for formatting context only)" section in the prompt. |
| 87 | + - When `criteriaContext` is non-null: insert "## Related Acceptance Criteria" section. |
| 88 | + - When `sourceRefPaths` is non-null: populate the returned `TestCase.SourceRefs` from those paths instead of `[]`. |
| 89 | + - Keep AI's `criteria` output (already populated by `agent.GenerateTestsAsync`) flowing into `TestCase.Criteria`. |
| 90 | + - Keep `grounding.verdict = Manual` unconditionally. |
| 91 | + |
| 92 | +2. **`GenerateHandler.cs`** — modify `ExecuteFromDescriptionAsync`: |
| 93 | + - Promote `LoadCriteriaContextAsync` from `private static` to allow reuse, OR call directly (it is already in the same class). |
| 94 | + - After loading config, before calling `generator.GenerateAsync`, perform best-effort load: |
| 95 | + ```csharp |
| 96 | + string? docContext = null; |
| 97 | + IReadOnlyList<string> docPaths = []; |
| 98 | + try |
| 99 | + { |
| 100 | + var loader = new SourceDocumentLoader(config.Source); |
| 101 | + var allDocs = await loader.LoadAllAsync(currentDir, maxDocuments: null, maxContentLengthPerDoc: 8000, ct); |
| 102 | + var matching = allDocs |
| 103 | + .Where(d => MatchesSuite(d, suite)) |
| 104 | + .Take(3) |
| 105 | + .ToList(); |
| 106 | + if (matching.Count > 0) |
| 107 | + { |
| 108 | + docContext = FormatDocContext(matching); |
| 109 | + docPaths = matching.Select(d => d.Path).ToList(); |
| 110 | + } |
| 111 | + } |
| 112 | + catch { /* best-effort */ } |
| 113 | + |
| 114 | + string? criteriaContext = null; |
| 115 | + try { criteriaContext = await LoadCriteriaContextAsync(currentDir, suite, config, ct); } |
| 116 | + catch { /* best-effort */ } |
| 117 | + ``` |
| 118 | + - `MatchesSuite` is a small private helper: case-insensitive contains on `doc.Path` filename or `doc.Title`. |
| 119 | + - `FormatDocContext` produces a delimited string of `## {title}\n{content}\n`. |
| 120 | + - Pass `docContext`, `criteriaContext`, `docPaths` to `generator.GenerateAsync`. |
| 121 | + |
| 122 | +3. **JSON result** — no shape change. `source_refs` and `criteria` are persisted via `TestFileWriter`, which already writes them. No `GenerateResult` schema change needed. |
| 123 | + |
| 124 | +### Phase 3 — Tests |
| 125 | + |
| 126 | +1. **`UserDescribedGeneratorTests`** (new): |
| 127 | + - `BuildPrompt_WithoutContext_DoesNotIncludeRefSection` |
| 128 | + - `BuildPrompt_WithDocContext_IncludesRefDocumentationSection` |
| 129 | + - `BuildPrompt_WithCriteriaContext_IncludesAcceptanceCriteriaSection` |
| 130 | + - `BuildPrompt_WithBothContexts_IncludesBoth` |
| 131 | + - `BuildPrompt_IncludesUserDescriptionAsSourceOfTruth` |
| 132 | + |
| 133 | +2. **`GenerateSkillContentTests`** (new): |
| 134 | + - `GenerateSkill_HasFromDescriptionSection` — asserts `SkillContent.Generate.Contains("create a specific test case")`. |
| 135 | + - `GenerateSkill_HasIntentRoutingTable` — asserts the table headers ("User intent", "Signal", "Flow") all present. |
| 136 | + - `GenerateSkill_FromDescriptionUsesCorrectFlags` — asserts `--from-description` line contains `--no-interaction` and `--output-format json` and `--verbosity quiet`. |
| 137 | + - `GenerationAgent_HasIntentRoutingSection` — asserts agent content contains "Test Creation Intent Routing" + "--from-description" + "--focus". |
| 138 | + - `GenerationAgent_RoutesToFromDescriptionForSpecificTest` — asserts agent content includes the example "Add a test for". |
| 139 | + - `GenerationAgent_DoesNotAskAboutCountInRoutingRules` — asserts the "do NOT ask clarifying questions about count" instruction exists. |
| 140 | + |
| 141 | +3. **Integration tests** — deferred. The from-description path invokes AgentFactory which requires real AI. Coverage of FR-008..FR-014 is via the prompt-building unit tests + manual smoke; no integration test will be added in this spec to keep the test suite isolated from network/AI dependencies. |
| 142 | + |
| 143 | +### Phase 4 — Documentation updates |
| 144 | + |
| 145 | +Update the 9 doc files listed in the spec: |
| 146 | +- `CLAUDE.md` — add 033 to Recent Changes. |
| 147 | +- `PROJECT-KNOWLEDGE.md` — add 033 implemented entry. |
| 148 | +- `README.md` — add "create a specific test" example near Quick Start. |
| 149 | +- `docs/getting-started.md` — add from-description example. |
| 150 | +- `docs/cli-reference.md` — note `--from-description` doc/criteria context. |
| 151 | +- `docs/skills-integration.md` — describe new from-description flow + intent routing. |
| 152 | +- `docs/test-format.md` — note `source_refs`/`criteria` may be populated for manual tests. |
| 153 | +- `docs/cli-vs-chat-generation.md` — update Dimension 8. |
| 154 | +- `docs/coverage.md` — note manual tests can now contribute to coverage. |
| 155 | + |
| 156 | +(If any of these files do not exist, skip that line item — they are optional polish.) |
| 157 | + |
| 158 | +## Risks & Mitigations |
| 159 | + |
| 160 | +| Risk | Mitigation | |
| 161 | +|------|------------| |
| 162 | +| Doc loading slows from-description noticeably | Cap at 3 docs × 8000 chars; load synchronously inside best-effort try block; no timeout needed since file I/O is bounded. | |
| 163 | +| AI emits criteria IDs that don't exist | Acceptable — coverage analyzer will simply not match them. The criteria context tells the AI which IDs are valid, so this should be rare. | |
| 164 | +| SKILL .md changes break existing skill content tests | Search existing tests for hardcoded SKILL strings before edit; update them in the same change. | |
| 165 | +| Refactoring `BuildPrompt` to static breaks existing call site | The existing `GenerateAsync` will still build the prompt internally (calling `BuildPrompt`), so call sites are unchanged. | |
0 commit comments