Merge branch '044-coverage-aware-analysis' into main

Author: angelovstanton

CLAUDE.md

@@ -130,3 +130,10 @@ Generated in JSON, Markdown, HTML. Features: test titles from `_index.json`, hum
 
 <!-- MANUAL ADDITIONS START -->
 <!-- MANUAL ADDITIONS END -->
+
+## Active Technologies
+- C# 12, .NET 8+ + GitHub Copilot SDK, System.CommandLine, Spectre.Console, System.Text.Json, YamlDotNet (044-coverage-aware-analysis)
+- File-based (Markdown/YAML/JSON in `test-cases/`, `docs/criteria/`, `_index.json`, `_criteria_index.yaml`) (044-coverage-aware-analysis)
+
+## Recent Changes
+- 044-coverage-aware-analysis: Added C# 12, .NET 8+ + GitHub Copilot SDK, System.CommandLine, Spectre.Console, System.Text.Json, YamlDotNet

Directory.Build.props

@@ -16,7 +16,7 @@
     <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
 
     <!-- Versioning -->
-    <Version>1.49.0</Version>
+    <Version>1.50.0</Version>
     <Authors>Spectra Contributors</Authors>
     <Company>Spectra</Company>
     <Product>Spectra</Product>

docs/cli-reference.md

@@ -134,7 +134,9 @@ techniques (Equivalence Partitioning, Boundary Value Analysis, Decision
 Table, State Transition, Error Guessing, Use Case). The analysis output
 includes both a category breakdown (`happy_path`, `boundary`, `negative`, …)
 and a technique breakdown (`BVA`, `EP`, `DT`, `ST`, `EG`, `UC`) — see spec
-037. Supports multiple modes.
+037. The analysis step is **coverage-aware** (spec 044): for existing suites,
+it considers test coverage from `_index.json`, acceptance criteria, and
+doc sections to recommend only gap tests. Supports multiple modes.
 
 **Interactive Session** — four-phase guided session:
 

docs/coverage.md

@@ -170,6 +170,20 @@ Scan patterns are templates where `{id}` is replaced with the test ID regex. Exa
 
 If `scan_patterns` is empty, SPECTRA falls back to the legacy `attribute_patterns` regex list.
 
+## Coverage-Aware Generation
+
+When running `spectra ai generate`, the analysis step is coverage-aware for existing suites. Before identifying testable behaviors, the analyzer builds a coverage snapshot from:
+
+- **`_index.json`**: Existing test titles, criteria links, and source refs
+- **`.criteria.yaml` files**: All acceptance criteria, cross-referenced against tests
+- **`docs/_index.md`**: Documentation sections, cross-referenced against test source refs
+
+The AI receives this coverage context and only recommends tests for genuine gaps — uncovered criteria and undocumented sections. For a mature suite with 231 tests covering 38/41 criteria, the analysis recommends ~8 new tests (the actual gap) instead of 139.
+
+For suites with more than 500 tests, the analyzer switches to summary mode to conserve prompt tokens: only coverage statistics and uncovered items are sent, not the full title list.
+
+New suites with no `_index.json` or criteria files work exactly as before — the coverage context is simply omitted.
+
 ## Coverage Configuration
 
 Full coverage settings in `spectra.config.json`:

docs/customization.md

@@ -109,6 +109,12 @@ When analyzing documentation, pay special attention to:
 - Rate limiting and abuse prevention on public APIs
 ```
 
+**Coverage-aware analysis**: The `behavior-analysis.md` template includes a
+`{{coverage_context}}` placeholder (spec 044). When an existing suite has
+coverage data, this resolves to a markdown block listing covered/uncovered
+criteria and doc sections, directing the AI to focus on gaps. If your custom
+template omits this placeholder, coverage context is simply not injected.
+
 **Safe updates**: `spectra update-skills` refreshes unmodified templates to
 the latest version. Modified templates are preserved. Use
 `spectra prompts reset {template}` to restore a template to default.

specs/044-coverage-aware-analysis/checklists/requirements.md

@@ -0,0 +1,35 @@
+# Specification Quality Checklist: Coverage-Aware Behavior Analysis
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-04-13
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Spec is ready for `/speckit.plan`.
+- The spec references file formats (`_index.json`, `_criteria_index.yaml`) by name since they are domain concepts (not implementation details) — these are the user-facing data files the feature operates on.

specs/044-coverage-aware-analysis/contracts/coverage-context-contract.md

@@ -0,0 +1,49 @@
+# Contract: Coverage Context in Analysis Output
+
+**Date**: 2026-04-13 | **Feature**: 044-coverage-aware-analysis
+
+## JSON Output Contract (--output-format json)
+
+When `spectra ai generate --analyze-only --output-format json` runs on a suite with existing tests, the `analysis` object in the result JSON includes coverage fields:
+
+### New fields in `analysis` object
+
+```json
+{
+  "status": "success",
+  "analysis": {
+    "total_behaviors": 8,
+    "already_covered": 233,
+    "recommended": 8,
+    "breakdown": { "boundary": 2, "negative": 2, "happy_path": 3, "edge_case": 1 },
+    "technique_breakdown": { "BVA": 3, "EP": 2, "EG": 1, "UC": 2 },
+    "existing_test_count": 231,
+    "total_criteria": 41,
+    "covered_criteria": 38,
+    "uncovered_criteria": 3,
+    "uncovered_criteria_ids": ["AC-039", "AC-040", "AC-041"]
+  }
+}
+```
+
+### Backward compatibility
+
+- All new fields have defaults: `existing_test_count: 0`, `total_criteria: 0`, `covered_criteria: 0`, `uncovered_criteria: 0`, `uncovered_criteria_ids: []`
+- Old consumers that don't read these fields are unaffected
+- When no coverage data is available (new suite), all new fields are 0/empty
+
+## Prompt Template Contract
+
+### New placeholder: `{{coverage_context}}`
+
+Added to `behavior-analysis.md` template. Resolves to a markdown block or empty string.
+
+**Full mode** (tests <= 500): Includes covered criteria IDs, uncovered criteria details, covered/uncovered source refs, and truncated test titles.
+
+**Summary mode** (tests > 500): Includes only coverage statistics and uncovered items. No title list.
+
+**Empty** (no coverage data): Placeholder resolves to empty string. Template behavior is identical to pre-feature.
+
+### User customization
+
+Users can override `behavior-analysis.md` via `.spectra/prompts/behavior-analysis.md`. If their template omits `{{coverage_context}}`, coverage data is simply not injected — no error, no crash.

specs/044-coverage-aware-analysis/data-model.md

@@ -0,0 +1,87 @@
+# Data Model: Coverage-Aware Behavior Analysis
+
+**Date**: 2026-04-13 | **Feature**: 044-coverage-aware-analysis
+
+## New Entities
+
+### CoverageSnapshot
+
+Aggregated view of what's already tested in a suite. Built from three independent data sources.
+
+| Field | Type | Source | Description |
+|-------|------|--------|-------------|
+| ExistingTestCount | int | `_index.json` | Total test cases in the suite |
+| ExistingTestTitles | string[] | `_index.json` | Test titles for dedup (truncated to 80 chars) |
+| CoveredCriteriaIds | HashSet\<string\> | `_index.json` tests → criteria fields | Criteria IDs with at least one linked test |
+| UncoveredCriteria | UncoveredCriterion[] | `_criteria_index.yaml` - CoveredCriteriaIds | Criteria with zero linked tests |
+| CoveredSourceRefs | HashSet\<string\> | `_index.json` tests → source_refs fields | Doc sections with at least one linked test |
+| UncoveredSourceRefs | string[] | `docs/_index.md` - CoveredSourceRefs | Doc sections with no linked tests |
+| TotalCriteriaCount | int | `_criteria_index.yaml` | Total criteria across all sources |
+| Mode | Full \| Summary | Computed | Full if tests <= 500, Summary otherwise |
+
+**Lifecycle**: Created per analysis run. Not persisted — derived from committed files.
+
+**Validation**:
+- Empty/missing data sources produce zero values (not errors)
+- CoveredCriteriaIds uses case-insensitive comparison
+- CoveredSourceRefs uses case-insensitive comparison
+
+### UncoveredCriterion
+
+A single acceptance criterion that has no linked tests.
+
+| Field | Type | Source | Description |
+|-------|------|--------|-------------|
+| Id | string | `.criteria.yaml` | Criterion identifier (e.g., "AC-001") |
+| Text | string | `.criteria.yaml` | Full criterion text |
+| Source | string? | `.criteria.yaml` | Source document path |
+| Priority | string | `.criteria.yaml` | Priority level (high/medium/low) |
+
+## Modified Entities
+
+### GenerateAnalysis (in GenerateResult.cs)
+
+New fields added alongside existing ones:
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| ExistingTestCount | int | 0 | Total tests in suite (from snapshot) |
+| TotalCriteria | int | 0 | Total acceptance criteria count |
+| CoveredCriteria | int | 0 | Criteria with linked tests |
+| UncoveredCriteria | int | 0 | Criteria without linked tests |
+| UncoveredCriteriaIds | string[] | [] | IDs of uncovered criteria |
+
+**Backward compatibility**: All new fields default to 0 or empty. Old JSON consumers ignore unknown fields. Old JSON without new fields deserializes with defaults.
+
+### BehaviorAnalysisResult (unchanged schema)
+
+No structural changes. The `AlreadyCovered` field now uses the snapshot count (when available) instead of title-similarity heuristic. `RecommendedCount` derives from `TotalBehaviors - AlreadyCovered` as before.
+
+## Data Flow
+
+```
+_index.json ──────┐
+                   │
+.criteria.yaml ────┤──→ CoverageSnapshotBuilder.BuildAsync() ──→ CoverageSnapshot
+                   │
+docs/_index.md ────┘
+                                    │
+                                    ▼
+                   CoverageContextFormatter.Format(snapshot) ──→ string (markdown block)
+                                    │
+                                    ▼
+                   PlaceholderResolver.Resolve(template, {coverage_context: block})
+                                    │
+                                    ▼
+                   BehaviorAnalyzer.AnalyzeAsync() ──→ BehaviorAnalysisResult
+                                    │
+                                    ▼
+                   GenerateAnalysis (enriched with snapshot stats)
+```
+
+## Relationships to Existing Models
+
+- **TestIndexEntry** (`_index.json`): Read `Criteria` (string[]) and `SourceRefs` (string[]) fields
+- **AcceptanceCriterion** (`.criteria.yaml`): Read `Id`, `Text`, `Source`/`SourceDoc`, `Priority` fields
+- **DocumentIndexEntry** (`docs/_index.md`): Read `Path` and `Sections[].Heading` for source ref matching
+- **CriteriaIndex** (`_criteria_index.yaml`): Read `Sources[]` to locate `.criteria.yaml` files

specs/044-coverage-aware-analysis/plan.md

@@ -0,0 +1,100 @@
+# Implementation Plan: Coverage-Aware Behavior Analysis
+
+**Branch**: `044-coverage-aware-analysis` | **Date**: 2026-04-13 | **Spec**: [spec.md](spec.md)
+**Input**: Feature specification from `/specs/044-coverage-aware-analysis/spec.md`
+
+## Summary
+
+The behavior analysis step in `spectra ai generate` currently treats documentation as a blank slate, identifying all testable behaviors and applying a weak title-similarity heuristic for deduplication. For mature suites this produces wildly inaccurate recommendations (139 new tests when 8 are actually needed). This plan adds a coverage snapshot that feeds existing test index, criteria coverage, and doc section coverage into the analysis prompt so the AI only recommends tests for genuine gaps.
+
+## Technical Context
+
+**Language/Version**: C# 12, .NET 8+  
+**Primary Dependencies**: GitHub Copilot SDK, System.CommandLine, Spectre.Console, System.Text.Json, YamlDotNet  
+**Storage**: File-based (Markdown/YAML/JSON in `test-cases/`, `docs/criteria/`, `_index.json`, `_criteria_index.yaml`)  
+**Testing**: xUnit (~1279 tests across 3 test projects)  
+**Target Platform**: Windows/macOS/Linux CLI  
+**Project Type**: CLI tool with MCP server  
+**Performance Goals**: Coverage snapshot computation < 2s for 500 tests  
+**Constraints**: Prompt token overhead < 5,000 tokens; switch to summary mode > 500 tests  
+**Scale/Scope**: Suites up to 500+ tests, 50+ criteria, 20+ doc sections
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+| Principle | Status | Notes |
+|-----------|--------|-------|
+| I. GitHub as Source of Truth | PASS | All data read from committed files (`_index.json`, `.criteria.yaml`, `_index.md`) |
+| II. Deterministic Execution | PASS | Same coverage data + same docs = same snapshot; no randomness in snapshot building |
+| III. Orchestrator-Agnostic Design | PASS | Coverage context is injected into the prompt template, not tied to a specific LLM |
+| IV. CLI-First Interface | PASS | Feature is CLI-only; no new UI required; existing flags preserved |
+| V. Simplicity (YAGNI) | PASS | One new model + one builder service + prompt template changes. No new abstractions beyond what's needed. Reuses existing `IndexWriter`, `CriteriaFileReader`, `DocumentIndexReader` |
+
+**Quality Gates**: All existing gates preserved. No new validation rules needed.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/044-coverage-aware-analysis/
+├── plan.md              # This file
+├── research.md          # Phase 0 output
+├── data-model.md        # Phase 1 output
+├── contracts/           # Phase 1 output
+│   └── coverage-context-contract.md
+├── quickstart.md        # Phase 1 output
+└── tasks.md             # Phase 2 output (/speckit.tasks)
+```
+
+### Source Code (repository root)
+
+```text
+src/
+  Spectra.CLI/
+    Agent/
+      Analysis/
+        BehaviorAnalysisResult.cs      # Existing
+        IdentifiedBehavior.cs          # Existing
+        CoverageSnapshot.cs            # NEW: model + UncoveredCriterion record
+        CoverageSnapshotBuilder.cs     # NEW: builds snapshot from index/criteria/docs
+        CoverageContextFormatter.cs    # NEW: formats snapshot for prompt injection
+      Copilot/
+        BehaviorAnalyzer.cs            # MODIFIED: accept snapshot, inject context
+    Commands/
+      Generate/
+        GenerateHandler.cs             # MODIFIED: build snapshot before analysis
+    Output/
+        AnalysisPresenter.cs           # MODIFIED: show coverage summary
+    Progress/
+        ProgressPageWriter.cs          # MODIFIED: coverage snapshot in HTML
+    Prompts/
+      Content/
+        behavior-analysis.md           # MODIFIED: add {{coverage_context}} placeholder
+    Results/
+        GenerateResult.cs              # MODIFIED: add coverage fields to GenerateAnalysis
+
+  Spectra.Core/
+    # No changes — read existing data via existing readers
+
+tests/
+  Spectra.CLI.Tests/
+    Agent/
+      Analysis/
+        CoverageSnapshotBuilderTests.cs  # NEW: 8 tests
+        CoverageContextFormatterTests.cs # NEW: 5 tests
+    Commands/
+      Generate/
+        GenerateHandlerCoverageTests.cs  # NEW: 2 integration tests
+    Output/
+        AnalysisPresenterCoverageTests.cs # NEW: 3 tests
+    Results/
+        GenerateAnalysisCoverageTests.cs  # NEW: 3 tests
+```
+
+**Structure Decision**: All new files go within the existing `Agent/Analysis/` directory (model + builder + formatter). No new project or directory structure needed — follows existing patterns exactly.
+
+## Complexity Tracking
+
+> No violations — all gates pass. No complexity justification needed.

specs/044-coverage-aware-analysis/quickstart.md

@@ -0,0 +1,46 @@
+# Quickstart: Coverage-Aware Behavior Analysis
+
+**Date**: 2026-04-13 | **Feature**: 044-coverage-aware-analysis
+
+## What changed
+
+The `spectra ai generate` analysis step now considers existing test coverage before recommending new tests. For mature suites, this means accurate gap-only recommendations instead of inflated counts.
+
+## How to verify
+
+### 1. Run analysis on an existing suite
+
+```bash
+spectra ai generate --suite my-suite --analyze-only
+```
+
+**Expected**: Output shows a coverage snapshot with criteria/doc coverage ratios and recommends only tests for genuine gaps.
+
+### 2. Check JSON output
+
+```bash
+spectra ai generate --suite my-suite --analyze-only --output-format json
+```
+
+**Expected**: JSON includes `existing_test_count`, `total_criteria`, `covered_criteria`, `uncovered_criteria`, `uncovered_criteria_ids` fields.
+
+### 3. Verify graceful degradation
+
+```bash
+spectra ai generate --suite new-suite --analyze-only
+```
+
+**Expected**: New suite with no `_index.json` produces standard analysis (no coverage section, no errors).
+
+## Key files to review
+
+| File | Change |
+|------|--------|
+| `src/Spectra.CLI/Agent/Analysis/CoverageSnapshot.cs` | New model |
+| `src/Spectra.CLI/Agent/Analysis/CoverageSnapshotBuilder.cs` | New builder |
+| `src/Spectra.CLI/Agent/Analysis/CoverageContextFormatter.cs` | New formatter |
+| `src/Spectra.CLI/Agent/Copilot/BehaviorAnalyzer.cs` | Accepts snapshot, injects context |
+| `src/Spectra.CLI/Commands/Generate/GenerateHandler.cs` | Builds snapshot before analysis |
+| `src/Spectra.CLI/Prompts/Content/behavior-analysis.md` | New `{{coverage_context}}` placeholder |
+| `src/Spectra.CLI/Output/AnalysisPresenter.cs` | Coverage summary display |
+| `src/Spectra.CLI/Results/GenerateResult.cs` | New fields on GenerateAnalysis |