Skip to content

Commit 0da74c6

Browse files
docs: update PROJECT-KNOWLEDGE, README, and user docs for specs 037/038/039
PROJECT-KNOWLEDGE.md: - Add specs 037 (ISTQB techniques), 038 (Testimize integration), 039 (unified critic provider list) to the Completed Feature Specs table - Add testimize section to the spectra.config.json schema example - Update Test Counts to 1551 (491 Core + 709 CLI + 351 MCP) README.md: - AI Test Generation feature blurb mentions ISTQB techniques and the category + technique breakdown - Ecosystem section gains a callout for the optional Testimize MCP integration with install instructions - Documentation index gains a Testimize Integration link docs/cli-reference.md: - spectra ai generate description mentions systematic ISTQB techniques - New "Testimize Commands (Optional)" section documenting spectra testimize check (human + JSON output, install hint, no MCP process when disabled) docs/coverage.md: - Note about boundary coverage gains from spec 037 ISTQB techniques - Note about optional spec 038 Testimize precision docs/generation-profiles.md: - Three-layer customization clarification table: profile (style), prompt template (reasoning, ISTQB), Testimize (test data values) docs/skills-integration.md: - Document the technique_breakdown field in analysis JSON output - Document the spectra testimize check JSON shape
1 parent 08cbb97 commit 0da74c6

File tree

6 files changed

+113
-5
lines changed

6 files changed

+113
-5
lines changed

PROJECT-KNOWLEDGE.md

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -291,10 +291,23 @@ Three-section unified coverage with distinct semantics:
291291
"criteria_file": "docs/criteria/_criteria_index.yaml",
292292
"criteria_dir": "docs/criteria",
293293
"automation_dirs": ["tests"]
294+
},
295+
"testimize": {
296+
"enabled": false,
297+
"mode": "exploratory",
298+
"strategy": "HybridArtificialBeeColony",
299+
"mcp": { "command": "testimize-mcp", "args": ["--mcp"] }
294300
}
295301
}
296302
```
297303

304+
> **Spec 039**: critic providers now use the same five names as the generator
305+
> (`github-models`, `azure-openai`, `azure-anthropic`, `openai`, `anthropic`).
306+
> Legacy `github` is a soft alias; legacy `google` is rejected.
307+
>
308+
> **Spec 038**: optional `testimize` section enables external Testimize MCP
309+
> integration for algorithmic test data optimization (off by default).
310+
298311
## Code Conventions
299312

300313
- **C# 12, .NET 8+**
@@ -310,6 +323,9 @@ Three-section unified coverage with distinct semantics:
310323

311324
| # | Feature | Key Changes |
312325
|---|---------|-------------|
326+
| 039 | Unified Critic Provider List | Critic provider validation now matches the generator provider list (`github-models`, `azure-openai`, `azure-anthropic`, `openai`, `anthropic`). Legacy `github` is a soft alias with deprecation warning; legacy `google` is hard-rejected. New `ResolveProvider` helper in `CriticFactory`. Enables Azure-only billing setups. |
327+
| 038 | Testimize Integration | Optional MCP integration with the external `Testimize.MCP.Server` global tool for algorithmic test data optimization (BVA, EP, pairwise, ABC). New `testimize` config section (disabled by default), `TestimizeMcpClient` (process lifecycle, JSON-RPC, 30s/5s timeouts, idempotent disposal), two new conditionally-registered AI tools (`GenerateTestData`, `AnalyzeFieldSpec`), `{{#if testimize_enabled}}` blocks in behavior-analysis and test-generation templates, new `spectra testimize check` CLI command, full graceful degradation. No NuGet dependency. |
328+
| 037 | ISTQB Test Design Techniques | All five built-in prompt templates rewritten to teach the AI six ISTQB techniques (EP, BVA, DT, ST, EG, UC). New `IdentifiedBehavior.Technique` field, `BehaviorAnalysisResult.TechniqueBreakdown` map, `AcceptanceCriterion.TechniqueHint` field. Analysis output includes `technique_breakdown` alongside `breakdown`. Terminal and progress page render a Technique Breakdown section in fixed display order. Distribution guideline caps any single category at 40%. Existing user-edited templates preserved; opt in via `spectra prompts reset --all`. |
313329
| 033 | From-Description Chat Flow | Dedicated `--from-description` SKILL section, agent intent routing (focus vs from-description vs from-suggestions), doc-aware manual tests with populated `source_refs` and `criteria` (verdict stays manual) |
314330
| 029 | spectra-update SKILL (10th) | Agent delegation, documentation sync, version 1.35.0 |
315331
| 028 | Coverage & Criteria Pipeline | Fixed criteria propagation in parser, wired criteria into generation pipeline, always write criteria: [] |
@@ -332,11 +348,12 @@ Three-section unified coverage with distinct semantics:
332348
| 008 | Grounding Verification | Dual-model critic, grounded/partial/hallucinated verdicts |
333349
| 006 | Conversational Generation | Direct/interactive modes, test updates, classification |
334350

335-
## Test Counts (as of 2026-04-08)
351+
## Test Counts (as of 2026-04-11)
336352

337353
| Project | Tests |
338354
|---------|-------|
339-
| Spectra.Core.Tests | 462 |
340-
| Spectra.CLI.Tests | 466 |
355+
| Spectra.Core.Tests | 491 |
356+
| Spectra.CLI.Tests | 709 |
341357
| Spectra.MCP.Tests | 351 |
358+
| **Total** | **1551** |
342359
| **Total** | **1279** |

README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ SPECTRA reads your product documentation, generates comprehensive test suites, a
6161
Generate test cases from product documentation through an iterative session.
6262
The AI analyzes your docs, identifies testable behaviors, generates structured test cases, suggests additional tests for uncovered areas, and lets you describe undocumented behaviors — all in one continuous flow.
6363

64+
Behavior analysis applies six **ISTQB test design techniques** systematically (Equivalence Partitioning, Boundary Value Analysis, Decision Table, State Transition, Error Guessing, Use Case) so the produced suites have meaningful boundary and negative coverage instead of happy-path-only scenarios. Analysis output exposes both a category breakdown and a per-technique breakdown.
65+
6466
```bash
6567
spectra ai generate # Interactive session
6668
spectra ai generate checkout # Direct mode
@@ -195,6 +197,14 @@ SPECTRA is part of the [Automate The Planet](https://www.automatetheplanet.com/)
195197

196198
**BELLATRIX** automates test execution. **Testimize** optimizes test case selection. **SPECTRA** generates and maintains the test cases themselves — closing the loop between documentation and quality assurance.
197199

200+
> **Optional Testimize integration**: SPECTRA can call the
201+
> [Testimize.MCP.Server](https://github.com/AutomateThePlanet/Testimize) global
202+
> tool to replace AI-approximated boundary values with mathematically optimal
203+
> ones (BVA / EP / pairwise / ABC). Disabled by default; enable with
204+
> `dotnet tool install --global Testimize.MCP.Server` and set
205+
> `testimize.enabled` to `true` in `spectra.config.json`. See
206+
> [Testimize Integration](docs/testimize-integration.md).
207+
198208
## Documentation
199209

200210
| Guide | Description |
@@ -210,6 +220,7 @@ SPECTRA is part of the [Automate The Planet](https://www.automatetheplanet.com/)
210220
| [Skills Integration](docs/skills-integration.md) | Copilot Chat SKILLs and agent prompts |
211221
| [Usage (Copilot Chat)](docs/usage.md) | Workflow-by-workflow guide for driving SPECTRA via Copilot Chat |
212222
| [Customization](docs/customization.md) | Every customization knob — config, profiles, prompts, palettes |
223+
| [Testimize Integration](docs/testimize-integration.md) | Optional algorithmic test data optimization (BVA / EP / pairwise / ABC) |
213224
| [Execution Agent](docs/execution-agent/overview.md) | MCP tools and AI-driven test execution |
214225
| [Architecture](docs/architecture/overview.md) | System design and key decisions |
215226
| [Development Guide](docs/DEVELOPMENT.md) | Building, testing, and running locally |

docs/cli-reference.md

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,12 @@ See [Document Index](document-index.md) for full details.
129129

130130
### `spectra ai generate`
131131

132-
Generate test cases from documentation. Supports multiple modes.
132+
Generate test cases from documentation using systematic ISTQB test design
133+
techniques (Equivalence Partitioning, Boundary Value Analysis, Decision
134+
Table, State Transition, Error Guessing, Use Case). The analysis output
135+
includes both a category breakdown (`happy_path`, `boundary`, `negative`, …)
136+
and a technique breakdown (`BVA`, `EP`, `DT`, `ST`, `EG`, `UC`) — see spec
137+
037. Supports multiple modes.
133138

134139
**Interactive Session** — four-phase guided session:
135140

@@ -283,6 +288,30 @@ See [Generation Profiles](generation-profiles.md) for details.
283288

284289
---
285290

291+
## Testimize Commands (Optional)
292+
293+
### `spectra testimize check`
294+
295+
Reports the status of the optional Testimize integration (spec 038): whether
296+
it is enabled in config, whether the `Testimize.MCP.Server` global tool is
297+
installed, whether the server passes a health probe, the configured mode and
298+
strategy, and whether the optional `testimizeSettings.json` is present.
299+
300+
```bash
301+
spectra testimize check
302+
spectra testimize check --output-format json
303+
```
304+
305+
When Testimize is disabled (the default), the command does NOT start the MCP
306+
process — it just reports `enabled: false`. When enabled but not installed,
307+
the output includes a one-line `dotnet tool install --global Testimize.MCP.Server`
308+
instruction. JSON output always contains the fields `enabled`, `installed`,
309+
`healthy`, plus `mode`, `strategy`, and (when not installed) `install_command`.
310+
311+
See [Testimize Integration](testimize-integration.md) for the full guide.
312+
313+
---
314+
286315
## Execution Commands
287316

288317
### `spectra-mcp` (separate tool)

docs/coverage.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,18 @@ SPECTRA produces a unified coverage report with three sections:
2222
| **Acceptance Criteria** | Which criteria are covered | `criteria` field in test frontmatter + `_criteria_index.yaml` |
2323
| **Automation** | Which tests have automation code | `automated_by` field in test frontmatter + code scanning |
2424

25+
> **Spec 037 — boundary coverage from ISTQB techniques**: Test generation now
26+
> applies six ISTQB test design techniques systematically (EP, BVA, DT, ST, EG,
27+
> UC). Suites generated after spec 037 typically have 50%+ more tests in the
28+
> `boundary` and `negative` categories than pre-037 suites on the same docs.
29+
> The analysis output exposes this via a `technique_breakdown` map alongside
30+
> the existing category `breakdown`.
31+
>
32+
> **Spec 038 — algorithmic precision (optional)**: When the optional
33+
> [Testimize integration](testimize-integration.md) is enabled, the AI replaces
34+
> approximated boundary values with mathematically optimal ones from
35+
> Testimize's BVA / EP / pairwise / ABC algorithms. Disabled by default.
36+
2537
## Run Coverage Analysis
2638

2739
```bash

docs/generation-profiles.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,18 @@ Related: [CLI Reference](cli-reference.md) | [Test Format](test-format.md) | [Co
1414

1515
## Overview
1616

17-
Profiles control test generation style: detail level, negative scenario count, step formatting, domain-specific considerations, and more. They're optional but recommended for consistent output.
17+
Profiles control test generation **style**: detail level, negative scenario count, step formatting, domain-specific considerations, and more. They're optional but recommended for consistent output.
18+
19+
> **Profiles vs prompt templates vs Testimize** — three orthogonal customization layers:
20+
>
21+
> | Layer | What it controls | Where it lives | Default |
22+
> |-------|-----------------|---------------|---------|
23+
> | **Profile** | Output format and style (step granularity, priority defaults, domain hints) | `profiles/_default.yaml` | Built-in default |
24+
> | **Prompt template** | Reasoning strategy (which ISTQB techniques to apply, which categories to use, how to structure analysis) — see [spec 037](../specs/037-istqb-test-techniques/spec.md) | `.spectra/prompts/*.md` | ISTQB-enhanced built-ins |
25+
> | **Testimize** (optional) | Test data values (boundary numbers, equivalence class examples, pairwise combinations) — see [spec 038](testimize-integration.md) | `testimize` section in `spectra.config.json` | Disabled |
26+
>
27+
> A typical setup uses the default profile, the default ISTQB prompts, and
28+
> leaves Testimize disabled. Power users mix and match independently.
1829
1930
## Create a Profile
2031

docs/skills-integration.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,34 @@ spectra <command> --output-format json --verbosity quiet
6767
- `--output-format json`: Structured JSON on stdout (no colors, spinners, or ANSI codes)
6868
- `--verbosity quiet`: Only the final result (no progress indicators)
6969

70+
#### Analysis JSON includes a `technique_breakdown` (spec 037)
71+
72+
The `analysis` subobject in `.spectra-result.json` from `spectra ai generate
73+
--analyze-only` exposes both the existing category `breakdown` and a new
74+
`technique_breakdown` map. Keys are short ISTQB codes (`BVA`, `EP`, `DT`,
75+
`ST`, `EG`, `UC`); values are counts. Always present (`{}` when empty)
76+
so SKILL parsers can rely on the field existing.
77+
78+
```json
79+
{
80+
"analysis": {
81+
"total_behaviors": 141,
82+
"breakdown": { "happy_path": 42, "boundary": 38, "negative": 24, "edge_case": 18 },
83+
"technique_breakdown": { "BVA": 38, "EP": 24, "UC": 32, "EG": 15, "DT": 18, "ST": 14 }
84+
}
85+
}
86+
```
87+
88+
The `spectra-generate` SKILL renders both breakdowns to the user when
89+
presenting the analyze recommendation.
90+
91+
#### `spectra testimize check` JSON (spec 038)
92+
93+
When the optional Testimize integration is in use, `spectra testimize check
94+
--output-format json` returns a `TestimizeCheckResult` object with the
95+
required fields `enabled`, `installed`, `healthy` plus `mode`, `strategy`,
96+
and (when not installed) `install_command`. See [Testimize Integration](testimize-integration.md).
97+
7098
### Generation Session in SKILLs
7199

72100
The generate SKILL supports the full session flow:

0 commit comments

Comments
 (0)