Skip to content

Commit 622e32e

Browse files
docs(spec 040): configuration audit — Testimize rewrite, missing sections, accuracy fixes
- testimize-integration.md: full rewrite for v1.48.3 in-process NuGet flow (was MCP child process). New: how-it-works pipeline, abc_settings sub-table, single-field vs multi-field paths, debug log line reference, health check output update. Removed all MCP references (mcp.command, mcp.args, global tool install). - configuration.md: added analysis section (categories schema + defaults), testimize cross-reference section, ai.fallback_strategy property, coverage.requirements_file + criteria_import sub-properties. - customization.md: expanded coverage settings with requirements_file, criteria_import sub-object, report_orphans/broken_links/mismatches flags. - README.md: updated Testimize callout from "install global MCP tool" to "bundled NuGet dependency, no separate install needed". - PROJECT-KNOWLEDGE.md: removed testimize.mcp from example config, updated spec 038 note for v1.48.3 in-process flow, fixed ai.debug_log_enabled → debug.enabled reference.
1 parent 444a23e commit 622e32e

File tree

5 files changed

+228
-54
lines changed

5 files changed

+228
-54
lines changed

PROJECT-KNOWLEDGE.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -299,8 +299,7 @@ Three-section unified coverage with distinct semantics:
299299
"testimize": {
300300
"enabled": false,
301301
"mode": "exploratory",
302-
"strategy": "HybridArtificialBeeColony",
303-
"mcp": { "command": "testimize-mcp", "args": ["--mcp"] }
302+
"strategy": "HybridArtificialBeeColony"
304303
}
305304
}
306305
```
@@ -309,8 +308,9 @@ Three-section unified coverage with distinct semantics:
309308
> - `ai.analysis_timeout_minutes` (default 2) — behavior analysis timeout
310309
> - `ai.generation_timeout_minutes` (default 5) — per-batch generation timeout
311310
> - `ai.generation_batch_size` (default 30) — tests per AI call
312-
> - `ai.debug_log_enabled` (default true) — writes per-call timing diagnostics
313-
> to `.spectra-debug.log` in the project root
311+
> - `debug.enabled` (default false) — writes per-call timing diagnostics
312+
> to `.spectra-debug.log` in the project root. Use `--verbosity diagnostic`
313+
> for one-shot override without changing config.
314314
>
315315
> Slower / reasoning models (DeepSeek-V3, large Azure deployments) typically
316316
> need: analysis 5–10 min, generation 15–20 min, batches of 6–10.
@@ -325,8 +325,13 @@ Three-section unified coverage with distinct semantics:
325325
> (`github-models`, `azure-openai`, `azure-anthropic`, `openai`, `anthropic`).
326326
> Legacy `github` is a soft alias; legacy `google` is rejected.
327327
>
328-
> **Spec 038**: optional `testimize` section enables external Testimize MCP
329-
> integration for algorithmic test data optimization (off by default).
328+
> **Spec 038 / v1.48.3**: optional `testimize` section enables in-process
329+
> Testimize integration for algorithmic test data optimization (BVA, EP,
330+
> pairwise, ABC). The library runs in-process as a bundled NuGet dependency
331+
> — no MCP child process, no separate tool installation. When enabled,
332+
> `TestimizeRunner` calls `TestimizeEngine.Configure(...).Generate()` once
333+
> per suite and embeds the result in the generation prompt as authoritative
334+
> facts. Off by default.
330335

331336
## Code Conventions
332337

README.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -197,13 +197,14 @@ SPECTRA is part of the [Automate The Planet](https://www.automatetheplanet.com/)
197197

198198
**BELLATRIX** automates test execution. **Testimize** optimizes test case selection. **SPECTRA** generates and maintains the test cases themselves — closing the loop between documentation and quality assurance.
199199

200-
> **Optional Testimize integration**: SPECTRA can call the
201-
> [Testimize.MCP.Server](https://github.com/AutomateThePlanet/Testimize) global
202-
> tool to replace AI-approximated boundary values with mathematically optimal
203-
> ones (BVA / EP / pairwise / ABC). Disabled by default; enable with
204-
> `dotnet tool install --global Testimize.MCP.Server` and set
205-
> `testimize.enabled` to `true` in `spectra.config.json`. See
206-
> [Testimize Integration](docs/testimize-integration.md).
200+
> **Optional Testimize integration**: SPECTRA ships with the
201+
> [Testimize](https://github.com/AutomateThePlanet/Testimize) library bundled
202+
> as an in-process NuGet dependency (no separate installation needed). When
203+
> enabled, it replaces AI-approximated boundary values with mathematically
204+
> precise ones — exact min-1/min/max/max+1 BVA, security-focused invalid
205+
> patterns (XSS, SQLi), and optimized multi-field pairwise/ABC combinations.
206+
> Disabled by default; set `testimize.enabled` to `true` in
207+
> `spectra.config.json`. See [Testimize Integration](docs/testimize-integration.md).
207208
208209
## Documentation
209210

docs/configuration.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,12 @@ SPECTRA is configured via `spectra.config.json` at the repository root. Run `spe
109109

110110
All providers are accessed through the GitHub Copilot SDK.
111111

112+
### `ai.fallback_strategy`
113+
114+
| Property | Type | Default | Description |
115+
|----------|------|---------|-------------|
116+
| `fallback_strategy` | string | `"auto"` | How to handle provider failures. `"auto"` tries the next enabled provider in priority order. When all providers fail, generation exits with an error. |
117+
112118
### `ai.generation_timeout_minutes`, `ai.generation_batch_size`
113119

114120
Per-batch tuning for `spectra ai generate`. These knobs are needed for slower
@@ -312,6 +318,36 @@ and re-run.
312318
| `duplicate_threshold` | double | `0.6` | Similarity threshold for duplicate detection |
313319
| `categories` | string[] | `["happy_path", "negative", "boundary", "integration"]` | Test categories to generate |
314320

321+
## `analysis` — Behavior Analysis Categories
322+
323+
Controls how the AI categorizes testable behaviors during the analysis phase
324+
(`spectra ai generate` runs this automatically before generation). See
325+
[Customization > Behavior Categories](customization.md#3-behavior-categories)
326+
for domain-specific examples.
327+
328+
| Property | Type | Default | Description |
329+
|----------|------|---------|-------------|
330+
| `categories` | object[] | (6 built-in) | Behavior classification taxonomy injected into the analysis prompt. |
331+
| `categories[].name` | string || Category identifier (used in tags and breakdown). |
332+
| `categories[].description` | string || One-line description injected into the AI prompt to guide classification. |
333+
334+
Default categories:
335+
336+
```json
337+
{
338+
"analysis": {
339+
"categories": [
340+
{ "name": "happy_path", "description": "Core happy-path functionality" },
341+
{ "name": "negative", "description": "Invalid input, unauthorized access, bad state" },
342+
{ "name": "edge_case", "description": "Unusual but valid scenarios" },
343+
{ "name": "boundary", "description": "Min/max values, exact limits, off-by-one" },
344+
{ "name": "error_handling", "description": "System failures, timeouts, downstream errors" },
345+
{ "name": "security", "description": "Auth checks, input sanitization, data exposure" }
346+
]
347+
}
348+
}
349+
```
350+
315351
## `update` — Test Update Settings
316352

317353
| Property | Type | Default | Description |
@@ -375,9 +411,14 @@ See [Coverage](coverage.md) for how these settings are used.
375411
| `file_patterns` | string[] | `["*.cs", "*.ts", "*.js", "*.py", "*.java"]` | Glob patterns for automation files (legacy) |
376412
| `attribute_patterns` | string[] | (framework defaults) | Regex patterns for test attributes (legacy, use `scan_patterns` instead) |
377413
| `test_id_pattern` | string | `"TC-\\d{3,}"` | Regex for matching test IDs |
414+
| `requirements_file` | string | `"docs/requirements/_requirements.yaml"` | Path to requirements YAML index for requirements coverage analysis |
378415
| `report_orphans` | bool | `true` | Report automation files not linked to any test |
379416
| `report_broken_links` | bool | `true` | Report tests referencing non-existent files |
380417
| `report_mismatches` | bool | `true` | Report inconsistencies between index and files |
418+
| `criteria_import.default_source_type` | string | `"manual"` | Default source type for imported acceptance criteria |
419+
| `criteria_import.auto_split` | bool | `true` | Automatically split compound criteria into atomic ones during import |
420+
| `criteria_import.normalize_rfc2119` | bool | `true` | Normalize RFC 2119 keywords (MUST, SHALL, SHOULD) in imported criteria |
421+
| `criteria_import.id_prefix` | string | `"AC"` | Prefix for auto-generated acceptance criteria IDs |
381422

382423
## `selections` — Saved Test Selections
383424

@@ -465,6 +506,22 @@ When configured, the execution agent uses the specified Copilot Space to answer
465506
| `validate_on_load` | bool | `true` | Validate profile structure on load |
466507
| `include_in_validation` | bool | `true` | Include profile validation in `spectra validate` |
467508

509+
## `testimize` — Testimize Integration (Optional)
510+
511+
Disabled by default. When enabled, SPECTRA runs the Testimize engine
512+
in-process during test generation to produce algorithmically precise
513+
boundary values, equivalence partitions, and multi-field combinations.
514+
See [Testimize Integration](testimize-integration.md) for the full setup
515+
guide, how-it-works flow, and ABC tuning reference.
516+
517+
| Property | Type | Default | Description |
518+
|----------|------|---------|-------------|
519+
| `enabled` | bool | `false` | Master switch. |
520+
| `mode` | string | `"exploratory"` | Maps to Testimize's `TestCaseCategory`: `"exploratory"` → All, `"valid"` → Valid only, `"validation"` → Invalid only. |
521+
| `strategy` | string | `"HybridArtificialBeeColony"` | Algorithm for multi-field suites: `Pairwise`, `OptimizedPairwise`, `Combinatorial`, `OptimizedCombinatorial`, `HybridArtificialBeeColony`. |
522+
| `settings_file` | string? | `null` | Path to a `testimizeSettings.json` with per-type equivalence classes and ABC tuning. |
523+
| `abc_settings` | object? | `null` | Optional ABC algorithm tuning. See [Testimize Integration > abc_settings](testimize-integration.md#abc_settings-optional-tuning). |
524+
468525
## `debug` — Debug Logging (Spec 040)
469526

470527
Controls the append-only `.spectra-debug.log` diagnostic file. **Disabled by

docs/customization.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,33 @@ match, and where acceptance criteria live.
183183
- `scan_patterns` — glob patterns for automation code files.
184184
- `criteria_dir` — where per-document `.criteria.yaml` files are stored.
185185
- `criteria_file` — path to the master criteria index.
186+
- `requirements_file` — path to the requirements YAML index (default:
187+
`"docs/requirements/_requirements.yaml"`). Used by requirements coverage
188+
analysis.
189+
- `report_orphans` / `report_broken_links` / `report_mismatches` — boolean
190+
flags (all default `true`) controlling which coverage report sections are
191+
generated. Set to `false` to suppress noisy sections in large repos.
192+
193+
**Criteria import tuning** (`coverage.criteria_import`):
194+
195+
```json
196+
{
197+
"coverage": {
198+
"criteria_import": {
199+
"default_source_type": "manual",
200+
"auto_split": true,
201+
"normalize_rfc2119": true,
202+
"id_prefix": "AC"
203+
}
204+
}
205+
}
206+
```
207+
208+
- `auto_split` — when importing compound acceptance criteria from CSV/YAML,
209+
automatically split them into atomic criteria.
210+
- `normalize_rfc2119` — normalize RFC 2119 keywords (MUST, SHALL, SHOULD) to
211+
a consistent form during import.
212+
- `id_prefix` — prefix for auto-generated acceptance criteria IDs (default: `"AC"`).
186213

187214
---
188215

0 commit comments

Comments
 (0)