Skip to content

Commit efb6518

Browse files
angelovstantonangelovstanton
committed
feat: visible default profile format & customization guide (031)
Ship profiles/_default.yaml as the visible source of the JSON schema the AI uses for test generation, and CUSTOMIZATION.md as a one-stop reference for every SPECTRA customization surface. - New ProfileFormatLoader resolves {{profile_format}} from profiles/_default.yaml on disk, falling back to an embedded default on missing/malformed files. - GenerationAgent.BuildFullPrompt now sources the JSON schema via the loader instead of a hardcoded constant. - spectra init creates both files and registers their hashes in .spectra/skills-manifest.json. - spectra update-skills tracks both files and preserves user edits. - 9 new tests (6 ProfileFormatLoader + 3 InitHandler), 1426 total passing.
1 parent d576713 commit efb6518

File tree

12 files changed

+904
-22
lines changed

12 files changed

+904
-22
lines changed

‎specs/031-profile-format-file/plan.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Implementation Plan: 031 Profile Format File & Customization Guide
2+
3+
## Architecture
4+
5+
Two new embedded resources, one new loader class, and additions to three existing files.
6+
7+
### New embedded resources
8+
9+
```
10+
src/Spectra.CLI/Skills/Content/
11+
├── Profiles/_default.yaml ← built-in default profile
12+
└── Docs/CUSTOMIZATION.md ← customization guide
13+
```
14+
15+
Both registered in `Spectra.CLI.csproj` via `<EmbeddedResource>`.
16+
17+
### New code
18+
19+
- `src/Spectra.CLI/Profile/ProfileFormatLoader.cs`
20+
- Static helper `LoadFormat(string workingDirectory)` returns the JSON schema string used for `{{profile_format}}`.
21+
- Resolution order:
22+
1. `profiles/_default.yaml` on disk → parse YAML, extract `format` field.
23+
2. Embedded resource `Spectra.CLI.Skills.Content.Profiles._default.yaml` → parse, extract `format`.
24+
- On parse failure or missing `format`, log warning and fall back to embedded.
25+
- Static helpers `LoadEmbeddedDefaultYaml()` and `LoadEmbeddedCustomizationGuide()` return raw text for use by init/update-skills.
26+
27+
### Modified code
28+
29+
- `src/Spectra.CLI/Agent/Copilot/GenerationAgent.cs`
30+
- `BuildFullPrompt` accepts an optional `profileFormat` parameter (string). When non-null, it replaces the hardcoded `jsonExample`. The legacy fallback path and template-loader path both use it.
31+
- `GenerateTestsAsync` calls `ProfileFormatLoader.LoadFormat(_basePath)` and passes the result.
32+
33+
- `src/Spectra.CLI/Commands/Init/InitHandler.cs`
34+
- New step `CreateDefaultProfileAsync` writes `profiles/_default.yaml`.
35+
- New step `CreateCustomizationGuideAsync` writes `CUSTOMIZATION.md` at the project root.
36+
- Both register their hashes in the skills manifest so `update-skills` can track them.
37+
38+
- `src/Spectra.CLI/Commands/UpdateSkills/UpdateSkillsHandler.cs`
39+
- Include `profiles/_default.yaml` and `CUSTOMIZATION.md` in the managed-file enumeration.
40+
41+
- `src/Spectra.CLI/Spectra.CLI.csproj`
42+
- Add `<EmbeddedResource Include="Skills\Content\Profiles\*.yaml" />` and `<EmbeddedResource Include="Skills\Content\Docs\*.md" />`.
43+
44+
## Tests
45+
46+
Added to `tests/Spectra.CLI.Tests/`:
47+
48+
- `Profile/ProfileFormatLoaderTests.cs`
49+
- `LoadFormat_ReturnsEmbeddedDefault_WhenNoFileExists`
50+
- `LoadFormat_ReturnsFileContent_WhenDefaultFileExists`
51+
- `LoadFormat_FallsBackToEmbedded_WhenFileIsMalformed`
52+
- `LoadFormat_FallsBackToEmbedded_WhenFormatFieldMissing`
53+
- Extend `Commands/InitCommandTests.cs`
54+
- `HandleAsync_CreatesDefaultProfile`
55+
- `HandleAsync_CreatesCustomizationGuide`
56+
- Verify both files appear in skills manifest.
57+
58+
## Risks & decisions
59+
60+
- The existing `ProfileConfig` in `Spectra.Core` is unrelated (it governs repository/suite markdown profiles). We deliberately do NOT add an `ai.profile` config field — the spec resolution narrows to `profiles/_default.yaml` on disk + embedded fallback. Named profiles via config are out of scope for this addendum and can be added later without breaking changes.
61+
- YAML parsing uses YamlDotNet (already a transitive dependency via `Spectra.Core`).

‎specs/031-profile-format-file/spec.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Feature Specification: Default Profile Visibility & Customization Guide
2+
3+
**Feature Branch**: `031-profile-format-file`
4+
**Created**: 2026-04-10
5+
**Status**: Draft
6+
**Depends on**: 030-prompt-templates, 004-test-generation-profile
7+
8+
## Problem
9+
10+
The `test-generation.md` prompt template references a `{{profile_format}}` placeholder that is silently filled by a hardcoded JSON schema baked into `GenerationAgent.BuildFullPrompt`. Users have no visibility into what the AI is asked to produce, no starting point to customize the output schema, and no single document explaining all the customization surfaces SPECTRA exposes.
11+
12+
## User Scenarios & Testing
13+
14+
### User Story 1 - Discoverable default format (Priority: P1)
15+
16+
Test architects want to see — and edit — the JSON schema the AI uses for test generation, without grepping source code.
17+
18+
**Why this priority**: This is the foundational visibility gap that drives the rest of the feature.
19+
20+
**Independent Test**: After running `spectra init`, the file `profiles/_default.yaml` exists, contains a `format` field with the JSON schema, and includes inline documentation comments. Editing the `format` field changes what the AI receives on the next `spectra ai generate` run.
21+
22+
**Acceptance Scenarios**:
23+
24+
1. **Given** a fresh project, **When** the user runs `spectra init`, **Then** `profiles/_default.yaml` exists with a documented `format` field.
25+
2. **Given** a project with `profiles/_default.yaml` modified by the user, **When** generation runs, **Then** the AI prompt's `{{profile_format}}` placeholder resolves to the contents of the user's `format` field.
26+
3. **Given** `profiles/_default.yaml` is missing, **When** generation runs, **Then** the system uses the embedded built-in default and does not fail.
27+
28+
### User Story 2 - One-stop customization guide (Priority: P1)
29+
30+
Users want a single curated reference document explaining every place SPECTRA can be customized: profiles, prompt templates, behavior categories, config, branding, SKILLs, and agents.
31+
32+
**Why this priority**: Customization surfaces already exist; without one map, they are effectively undiscoverable.
33+
34+
**Independent Test**: After `spectra init`, `CUSTOMIZATION.md` exists at the project root and lists all customization surfaces with file paths and worked examples.
35+
36+
**Acceptance Scenarios**:
37+
38+
1. **Given** a fresh project, **When** the user runs `spectra init`, **Then** `CUSTOMIZATION.md` exists at the project root.
39+
2. **Given** an existing project where the user has not edited `CUSTOMIZATION.md`, **When** they run `spectra update-skills`, **Then** the file is refreshed to the latest version.
40+
3. **Given** an existing project where the user has edited `CUSTOMIZATION.md`, **When** they run `spectra update-skills`, **Then** the file is left untouched.
41+
42+
### User Story 3 - Safe upgrade path (Priority: P2)
43+
44+
Users want both new files tracked by the same hash-based update mechanism as SKILL files and prompt templates, so upgrades never silently overwrite their customizations.
45+
46+
**Acceptance Scenarios**:
47+
48+
1. **Given** `profiles/_default.yaml` matches the current built-in hash, **When** `spectra update-skills` runs, **Then** the file is updated to the latest built-in.
49+
2. **Given** the user has modified `profiles/_default.yaml`, **When** `spectra update-skills` runs, **Then** the file is preserved and reported as user-modified.
50+
51+
### Edge Cases
52+
53+
- `profiles/_default.yaml` exists but has malformed YAML → fall back to built-in embedded default and log a warning.
54+
- `profiles/_default.yaml` exists but has no `format` field → fall back to built-in embedded default.
55+
- Project does not have a `profiles/` directory at generation time → use built-in embedded default.
56+
57+
## Requirements
58+
59+
### Functional Requirements
60+
61+
- **FR-001**: System MUST ship a built-in default profile YAML as an embedded resource that contains a `format` field whose value is the JSON schema sent to the AI as `{{profile_format}}`.
62+
- **FR-002**: `spectra init` MUST create `profiles/_default.yaml` from the embedded default unless the file already exists (without `--force`).
63+
- **FR-003**: When resolving `{{profile_format}}`, the system MUST check `profiles/_default.yaml` on disk before falling back to the embedded default.
64+
- **FR-004**: System MUST ship a built-in `CUSTOMIZATION.md` as an embedded resource covering all user-facing customization surfaces.
65+
- **FR-005**: `spectra init` MUST create `CUSTOMIZATION.md` at the project root from the embedded default unless the file already exists.
66+
- **FR-006**: Both `profiles/_default.yaml` and `CUSTOMIZATION.md` MUST be tracked in `.spectra/skills-manifest.json` so `spectra update-skills` can detect user modifications and preserve them.
67+
- **FR-007**: If `profiles/_default.yaml` is malformed or missing the `format` field, the system MUST gracefully fall back to the embedded built-in default and never crash test generation.
68+
69+
### Key Entities
70+
71+
- **ProfileFormatDocument**: A YAML document with a top-level `format` field (string containing the JSON schema sent to the AI) and an optional `fields` section documenting each output field for human readers.
72+
73+
## Success Criteria
74+
75+
- **SC-001**: A user can change the JSON schema sent to the AI for test generation by editing exactly one file (`profiles/_default.yaml`) without writing any code.
76+
- **SC-002**: A new user can locate every customization surface in SPECTRA from a single document at the project root in under 60 seconds.
77+
- **SC-003**: 100% of `dotnet test` runs (including new tests added by this feature) pass after implementation.
78+
- **SC-004**: `spectra update-skills` never overwrites a user-modified `profiles/_default.yaml` or `CUSTOMIZATION.md`.

‎specs/031-profile-format-file/tasks.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Tasks: 031 Profile Format File & Customization Guide
2+
3+
## T1 — Add embedded default profile YAML
4+
- Create `src/Spectra.CLI/Skills/Content/Profiles/_default.yaml` with documented `format` field + `fields` reference section.
5+
- Update `Spectra.CLI.csproj` to include `Skills\Content\Profiles\*.yaml` as embedded resource.
6+
7+
## T2 — Add embedded customization guide
8+
- Create `src/Spectra.CLI/Skills/Content/Docs/CUSTOMIZATION.md`.
9+
- Update `Spectra.CLI.csproj` to include `Skills\Content\Docs\*.md` as embedded resource.
10+
11+
## T3 — Implement ProfileFormatLoader
12+
- New file `src/Spectra.CLI/Profile/ProfileFormatLoader.cs`.
13+
- Methods: `LoadFormat(string workingDirectory)`, `LoadEmbeddedDefaultYaml()`, `LoadEmbeddedCustomizationGuide()`.
14+
- 3-tier fallback with malformed-file safety.
15+
16+
## T4 — Wire ProfileFormatLoader into GenerationAgent
17+
- Update `BuildFullPrompt` to accept optional `profileFormat` string and use it in both code paths.
18+
- Update `GenerateTestsAsync` to load and pass it.
19+
20+
## T5 — Init handler creates new files
21+
- Add `CreateDefaultProfileAsync` and `CreateCustomizationGuideAsync` to `InitHandler`.
22+
- Both register hashes in `.spectra/skills-manifest.json`.
23+
24+
## T6 — update-skills tracks new files
25+
- Update `UpdateSkillsHandler` to include both new files in its managed list.
26+
27+
## T7 — Tests
28+
- Add `tests/Spectra.CLI.Tests/Profile/ProfileFormatLoaderTests.cs` (4 tests).
29+
- Add init tests for new files in `Commands/InitCommandTests.cs` (2 tests).
30+
31+
## T8 — Verify
32+
- Run `dotnet test` and confirm all tests pass.

‎src/Spectra.CLI/Agent/Copilot/GenerationAgent.cs‎

Lines changed: 13 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
using GitHub.Copilot.SDK;
55
using Microsoft.Extensions.AI;
66
using Spectra.CLI.Agent.Tools;
7+
using Spectra.CLI.Profile;
78
using Spectra.CLI.Prompts;
89
using Spectra.Core.Index;
910
using Spectra.Core.Models;
@@ -133,8 +134,11 @@ public async Task<GenerationResult> GenerateTestsAsync(
133134
}
134135
});
135136

136-
// Build the combined prompt with system instructions and user request
137-
var fullPrompt = BuildFullPrompt(prompt, requestedCount, criteriaContext);
137+
// Build the combined prompt with system instructions and user request.
138+
// The profile format (JSON schema sent to the AI) is resolved from
139+
// profiles/_default.yaml on disk if present, else the embedded default.
140+
var profileFormat = ProfileFormatLoader.LoadFormat(_basePath);
141+
var fullPrompt = BuildFullPrompt(prompt, requestedCount, criteriaContext, profileFormat: profileFormat);
138142

139143
// Send and wait for the complete response
140144
_onStatus?.Invoke("Starting AI generation...");
@@ -246,27 +250,14 @@ private void SaveDebugResponse(string responseText)
246250
}
247251

248252
internal static string BuildFullPrompt(string userPrompt, int requestedCount, string? criteriaContext = null,
249-
PromptTemplateLoader? templateLoader = null)
253+
PromptTemplateLoader? templateLoader = null, string? profileFormat = null)
250254
{
251-
var jsonExample = """
252-
[
253-
{
254-
"id": "TC-XXX",
255-
"title": "Descriptive title based on documentation",
256-
"priority": "high|medium|low",
257-
"tags": ["tag1", "tag2"],
258-
"component": "component-name",
259-
"preconditions": "Setup requirements",
260-
"steps": ["Step 1", "Step 2", "Step 3"],
261-
"expected_result": "Specific outcome from documentation",
262-
"test_data": "Test data if needed",
263-
"source_refs": ["docs/file.md#Section-Name"],
264-
"scenario_from_doc": "Quote or paraphrase the documented behavior",
265-
"estimated_duration": "5m",
266-
"criteria": ["AC-ID-1", "AC-ID-2"]
267-
}
268-
]
269-
""";
255+
// The JSON output schema sent to the AI. Prefer the caller-supplied
256+
// profileFormat (resolved from profiles/_default.yaml on disk or the
257+
// embedded default by ProfileFormatLoader). When null (legacy callers
258+
// and unit tests), fall back to the embedded default to keep behavior
259+
// identical.
260+
var jsonExample = profileFormat ?? ProfileFormatLoader.LoadFormat(Directory.GetCurrentDirectory());
270261

271262
if (templateLoader is not null)
272263
{

‎src/Spectra.CLI/Commands/Init/InitHandler.cs‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System.Text.Json;
44
using Spectra.CLI.Infrastructure;
55
using Spectra.CLI.Output;
6+
using Spectra.CLI.Profile;
67
using Spectra.CLI.Source;
78
using Spectra.Core.Config;
89
using Spectra.Core.Models.Config;
@@ -79,6 +80,9 @@ public async Task<int> HandleAsync(bool force, bool skipSkills = false, Cancella
7980
// Create prompt templates
8081
await CreatePromptTemplatesAsync(force, ct);
8182

83+
// Create default profile and customization guide
84+
await CreateDefaultProfileAndCustomizationAsync(force, ct);
85+
8286
// Create dashboard deployment workflow
8387
await CreateDeployWorkflowAsync(ct);
8488

@@ -589,6 +593,48 @@ private async Task CreatePromptTemplatesAsync(bool force, CancellationToken ct)
589593
await manifestStore.SaveAsync(manifest, ct);
590594
}
591595

596+
private async Task CreateDefaultProfileAndCustomizationAsync(bool force, CancellationToken ct)
597+
{
598+
var manifestStore = new Skills.SkillsManifestStore(_workingDirectory);
599+
var manifest = await manifestStore.LoadAsync(ct);
600+
601+
// profiles/_default.yaml
602+
var profilesDir = Path.Combine(_workingDirectory, "profiles");
603+
Directory.CreateDirectory(profilesDir);
604+
var profilePath = Path.Combine(profilesDir, "_default.yaml");
605+
var profileContent = ProfileFormatLoader.LoadEmbeddedDefaultYaml();
606+
var profileRelative = Path.Combine("profiles", "_default.yaml");
607+
608+
if (!File.Exists(profilePath) || force)
609+
{
610+
await File.WriteAllTextAsync(profilePath, profileContent, ct);
611+
_logger.LogInformation("Created default profile: {Path}", profileRelative);
612+
}
613+
else
614+
{
615+
_logger.LogDebug("Default profile already exists, skipping: {Path}", profilePath);
616+
}
617+
manifest.Files[profileRelative] = Infrastructure.FileHasher.ComputeHash(profileContent);
618+
619+
// CUSTOMIZATION.md (project root)
620+
var customizationPath = Path.Combine(_workingDirectory, "CUSTOMIZATION.md");
621+
var customizationContent = ProfileFormatLoader.LoadEmbeddedCustomizationGuide();
622+
const string customizationRelative = "CUSTOMIZATION.md";
623+
624+
if (!File.Exists(customizationPath) || force)
625+
{
626+
await File.WriteAllTextAsync(customizationPath, customizationContent, ct);
627+
_logger.LogInformation("Created customization guide: {Path}", customizationRelative);
628+
}
629+
else
630+
{
631+
_logger.LogDebug("Customization guide already exists, skipping: {Path}", customizationPath);
632+
}
633+
manifest.Files[customizationRelative] = Infrastructure.FileHasher.ComputeHash(customizationContent);
634+
635+
await manifestStore.SaveAsync(manifest, ct);
636+
}
637+
592638
private async Task CreateVsCodeMcpConfigAsync(CancellationToken ct)
593639
{
594640
var mcpConfigPath = Path.Combine(_workingDirectory, VsCodeMcpPath);

‎src/Spectra.CLI/Commands/UpdateSkills/UpdateSkillsHandler.cs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
using Spectra.CLI.Infrastructure;
22
using Spectra.CLI.Output;
3+
using Spectra.CLI.Profile;
34
using Spectra.CLI.Prompts;
45
using Spectra.CLI.Skills;
56

@@ -62,6 +63,20 @@ public async Task<int> ExecuteAsync(CancellationToken ct = default)
6263
await ProcessFileAsync(templatePath, content, manifest, updated, unchanged, skipped, ct);
6364
}
6465

66+
// Process default profile (profiles/_default.yaml)
67+
var defaultProfilePath = Path.Combine(currentDir, "profiles", "_default.yaml");
68+
await ProcessFileAsync(
69+
defaultProfilePath,
70+
ProfileFormatLoader.LoadEmbeddedDefaultYaml(),
71+
manifest, updated, unchanged, skipped, ct);
72+
73+
// Process customization guide (CUSTOMIZATION.md at project root)
74+
var customizationPath = Path.Combine(currentDir, "CUSTOMIZATION.md");
75+
await ProcessFileAsync(
76+
customizationPath,
77+
ProfileFormatLoader.LoadEmbeddedCustomizationGuide(),
78+
manifest, updated, unchanged, skipped, ct);
79+
6580
// Save updated manifest
6681
await manifestStore.SaveAsync(manifest, ct);
6782

0 commit comments

Comments
 (0)
⚡