Merge pull request #41 from agent-ecosystem/feat-allow-dirs

Feat: Add support for explicitly allowing specified directories

Author: dacharyc

README.md

@@ -33,6 +33,8 @@ Spec compliance is table stakes. `skill-validator` goes further: it checks that
 - [Examples](#examples)
 - [What it checks & why](#what-it-checks)
   - [Structure validation](#structure-validation-validate-structure)
+    - [Flat skill layouts](#flat-skill-layouts)
+    - [Allowing non-standard directories](#allowing-non-standard-directories)
   - [Link validation](#link-validation-validate-links)
   - [Content analysis](#content-analysis-analyze-content)
   - [Contamination analysis](#contamination-analysis-analyze-contamination)
@@ -185,9 +187,18 @@ skill-validator validate structure --skip-orphans <path>
 skill-validator validate structure --strict <path>
 skill-validator validate structure --allow-extra-frontmatter <path>
 skill-validator validate structure --allow-flat-layouts <path>
+skill-validator validate structure --allow-dirs=evals,testing <path>
 ```
 
-Checks spec compliance: directory structure, frontmatter fields, token limits, skill ratio, code fence integrity, internal link validity, and orphan file detection. Use `--skip-orphans` to suppress warnings about unreferenced files in `scripts/`, `references/`, and `assets/`. Use `--strict` to treat warnings as errors (exit 1 instead of 2). Use `--allow-extra-frontmatter` to suppress warnings for frontmatter fields not defined in the spec (e.g. `user-invokable`). Standard frontmatter fields are still fully validated. Use `--allow-flat-layouts` to allow supplemental files alongside SKILL.md at the skill root without warnings (see [Flat skill layouts](#flat-skill-layouts)).
+Checks spec compliance: directory structure, frontmatter fields, token limits, skill ratio, code fence integrity, internal link validity, and orphan file detection.
+
+| Flag | Effect |
+|---|---|
+| `--strict` | Treat warnings as errors (exit 1 instead of 2) |
+| `--skip-orphans` | Suppress warnings about unreferenced files in `scripts/`, `references/`, and `assets/` |
+| `--allow-extra-frontmatter` | Suppress warnings for non-spec frontmatter fields (e.g. `user-invokable`). Standard fields are still fully validated |
+| `--allow-flat-layouts` | Allow files at the skill root without warnings (see [Flat skill layouts](#flat-skill-layouts)) |
+| `--allow-dirs=evals,testing` | Accept specific non-standard directories without warnings (see [Allowing non-standard directories](#allowing-non-standard-directories)) |
 
 ```
 Validating skill: my-skill/
@@ -284,9 +295,21 @@ skill-validator check --skip-orphans <path>
 skill-validator check --strict <path>
 skill-validator check --allow-extra-frontmatter <path>
 skill-validator check --allow-flat-layouts <path>
+skill-validator check --allow-dirs=evals,testing <path>
 ```
 
-Runs all checks (structure + links + content + contamination). Use `--only` or `--skip` to select specific check groups. The flags are mutually exclusive. Use `--per-file` to see per-file reference analysis alongside the aggregate. Use `--skip-orphans` to suppress orphan file warnings in the structure check. Use `--strict` to treat warnings as errors (exit 1 instead of 2). Use `--allow-extra-frontmatter` to suppress warnings for non-spec frontmatter fields. Use `--allow-flat-layouts` to allow supplemental files at the skill root without warnings (see [Flat skill layouts](#flat-skill-layouts)).
+Runs all checks (structure + links + content + contamination).
+
+| Flag | Effect |
+|---|---|
+| `--only` | Comma-separated list of check groups to run (mutually exclusive with `--skip`) |
+| `--skip` | Comma-separated list of check groups to skip (mutually exclusive with `--only`) |
+| `--per-file` | Show per-file reference analysis alongside the aggregate |
+| `--strict` | Treat warnings as errors (exit 1 instead of 2) |
+| `--skip-orphans` | Suppress orphan file warnings in the structure check |
+| `--allow-extra-frontmatter` | Suppress warnings for non-spec frontmatter fields |
+| `--allow-flat-layouts` | Allow files at the skill root without warnings (see [Flat skill layouts](#flat-skill-layouts)) |
+| `--allow-dirs=evals,testing` | Accept specific non-standard directories without warnings (see [Allowing non-standard directories](#allowing-non-standard-directories)) |
 
 Valid check groups: `structure`, `links`, `content`, `contamination`.
 
@@ -709,6 +732,27 @@ skill-validator check --allow-flat-layouts my-skill/
 > [!NOTE]
 > The standard directory structure remains the recommended approach for maximum portability across agent platforms. Use `--allow-flat-layouts` when a flat layout better fits your workflow, with the understanding that some platforms may not discover files outside the recognized directories.
 
+**Allowing non-standard directories**
+
+The spec defines three recognized directories (`scripts/`, `references/`, `assets/`). Any other directory at the skill root produces a warning. This relates to cross-platform skill file loading considerations described in [agent-ecosystem/agent-skill-implementation](https://github.com/agent-ecosystem/agent-skill-implementation).
+
+Some development workflows use additional directories that may produce unexpected behavior across agent platforms. For example, the [evaluating-skills guide](https://agentskills.io/skill-creation/evaluating-skills) recommends an `evals/` directory for evaluation test cases, and teams may keep integration test fixtures in a `testing/` directory. If you are not distributing cross-platform skills and want to suppress warnings for specific directories that you know your preferred agent platform supports, use the `--allow-dirs` flag to suppress warnings for specific directories by name:
+
+```
+skill-validator validate structure --allow-dirs=evals my-skill/
+skill-validator check --allow-dirs=evals,testing my-skill/
+```
+
+The flag accepts a comma-separated list or can be repeated (`--allow-dirs=evals --allow-dirs=testing`). Allowed directories differ from recognized directories in two ways:
+
+1. **Exempt from deep-nesting checks**: The validator can't know the expected internal structure of arbitrary directories, so subdirectories like `evals/files/` won't trigger nesting warnings.
+2. **Skipped for orphan detection**: Since the validator doesn't know how these directories are used, it skips orphan file checks for them and emits an informational note instead.
+
+Directories not in the allow list still produce the standard warning with file counts and suggestions. If an allowed directory name matches a recognized directory (e.g., `--allow-dirs=scripts`), it's silently accepted with no change in behavior.
+
+> [!NOTE]
+> Allowing a directory suppresses validator warnings but does not change how agent platforms handle the directory. Files in non-standard directories may not be discovered during skill activation, or may load into agent context unexpectedly. If you're distributing skills across platforms, consider whether those files belong in `references/` or `assets/` instead.
+
 ### Link validation (`validate links`)
 
 - Checks external (HTTP/HTTPS) links only -- internal (relative) links are validated by `validate structure`

cmd/check.go

@@ -20,6 +20,7 @@ var (
 	strictCheck                bool
 	checkAllowExtraFrontmatter bool
 	checkAllowFlatLayouts      bool
+	checkAllowDirs             []string
 )
 
 var checkCmd = &cobra.Command{
@@ -41,6 +42,8 @@ func init() {
 		"suppress warnings for non-spec frontmatter fields")
 	checkCmd.Flags().BoolVar(&checkAllowFlatLayouts, "allow-flat-layouts", false,
 		"allow files at the skill root without warnings and treat them as standard content for token counting")
+	checkCmd.Flags().StringSliceVar(&checkAllowDirs, "allow-dirs", nil,
+		"comma-separated list of directory names to accept without warnings (e.g. --allow-dirs=evals,testing)")
 	rootCmd.AddCommand(checkCmd)
 }
 
@@ -72,6 +75,7 @@ func runCheck(cmd *cobra.Command, args []string) error {
 			SkipOrphans:           checkSkipOrphans,
 			AllowExtraFrontmatter: checkAllowExtraFrontmatter,
 			AllowFlatLayouts:      checkAllowFlatLayouts,
+			AllowDirs:             checkAllowDirs,
 		},
 	}
 	eopts := exitOpts{strict: strictCheck}

cmd/cmd_test.go

@@ -551,6 +551,121 @@ func TestValidateCommand_FlatSkill_OrphanDetection(t *testing.T) {
 	}
 }
 
+func TestValidateCommand_AllowedDirsSkill_WithoutFlag(t *testing.T) {
+	dir := fixtureDir(t, "allowed-dirs-skill")
+
+	r := structure.Validate(dir, structure.Options{})
+	// Without --allow-dirs, evals/ and testing/ should produce warnings
+	hasEvalsWarning := false
+	hasTestingWarning := false
+	for _, res := range r.Results {
+		if res.Level == types.Warning && strings.Contains(res.Message, "unknown directory: evals/") {
+			hasEvalsWarning = true
+		}
+		if res.Level == types.Warning && strings.Contains(res.Message, "unknown directory: testing/") {
+			hasTestingWarning = true
+		}
+	}
+	if !hasEvalsWarning {
+		t.Error("expected warning for evals/ without --allow-dirs")
+	}
+	if !hasTestingWarning {
+		t.Error("expected warning for testing/ without --allow-dirs")
+	}
+}
+
+func TestValidateCommand_AllowedDirsSkill_WithFlag(t *testing.T) {
+	dir := fixtureDir(t, "allowed-dirs-skill")
+
+	r := structure.Validate(dir, structure.Options{AllowDirs: []string{"evals", "testing"}})
+
+	// Should pass with no errors
+	if r.Errors != 0 {
+		t.Errorf("expected 0 errors, got %d", r.Errors)
+		for _, res := range r.Results {
+			if res.Level == types.Error {
+				t.Logf("  error: %s: %s", res.Category, res.Message)
+			}
+		}
+	}
+
+	// No warnings about evals/ or testing/
+	for _, res := range r.Results {
+		if res.Level == types.Warning &&
+			(strings.Contains(res.Message, "evals/") || strings.Contains(res.Message, "testing/")) {
+			t.Errorf("unexpected warning with --allow-dirs: %s", res.Message)
+		}
+	}
+
+	// No deep nesting warning for evals/files/
+	for _, res := range r.Results {
+		if res.Level == types.Warning && strings.Contains(res.Message, "deep nesting") {
+			t.Errorf("unexpected deep nesting warning for allowed dir: %s", res.Message)
+		}
+	}
+
+	// Should have info notes for orphan detection skipping
+	hasEvalsInfo := false
+	hasTestingInfo := false
+	for _, res := range r.Results {
+		if res.Level == types.Info && strings.Contains(res.Message, "evals/ skipped for orphan detection") {
+			hasEvalsInfo = true
+		}
+		if res.Level == types.Info && strings.Contains(res.Message, "testing/ skipped for orphan detection") {
+			hasTestingInfo = true
+		}
+	}
+	if !hasEvalsInfo {
+		t.Error("expected info note about evals/ being skipped for orphan detection")
+	}
+	if !hasTestingInfo {
+		t.Error("expected info note about testing/ being skipped for orphan detection")
+	}
+
+	// Recognized dirs should still have orphan pass results
+	hasReferencesPass := false
+	hasScriptsPass := false
+	for _, res := range r.Results {
+		if res.Level == types.Pass && strings.Contains(res.Message, "all files in references/ are referenced") {
+			hasReferencesPass = true
+		}
+		if res.Level == types.Pass && strings.Contains(res.Message, "all files in scripts/ are referenced") {
+			hasScriptsPass = true
+		}
+	}
+	if !hasReferencesPass {
+		t.Error("expected pass for references/ orphan check")
+	}
+	if !hasScriptsPass {
+		t.Error("expected pass for scripts/ orphan check")
+	}
+}
+
+func TestValidateCommand_AllowedDirsSkill_PartialAllow(t *testing.T) {
+	dir := fixtureDir(t, "allowed-dirs-skill")
+
+	// Only allow evals, not testing
+	r := structure.Validate(dir, structure.Options{AllowDirs: []string{"evals"}})
+
+	// evals/ should not warn
+	for _, res := range r.Results {
+		if res.Level == types.Warning && strings.Contains(res.Message, "unknown directory: evals/") {
+			t.Error("unexpected warning for evals/ with --allow-dirs=evals")
+		}
+	}
+
+	// testing/ should still warn
+	hasTestingWarning := false
+	for _, res := range r.Results {
+		if res.Level == types.Warning && strings.Contains(res.Message, "unknown directory: testing/") {
+			hasTestingWarning = true
+		}
+	}
+	if !hasTestingWarning {
+		t.Error("expected warning for testing/ when not in --allow-dirs")
+	}
+}
+
 func TestDetectAndResolve_NoSkill(t *testing.T) {
 	dir := t.TempDir()
 	_, _, _, err := detectAndResolve([]string{dir})

cmd/validate_structure.go

@@ -12,6 +12,7 @@ var (
 	strictStructure             bool
 	structAllowExtraFrontmatter bool
 	structAllowFlatLayouts      bool
+	structAllowDirs             []string
 )
 
 var validateStructureCmd = &cobra.Command{
@@ -30,6 +31,8 @@ func init() {
 		"suppress warnings for non-spec frontmatter fields")
 	validateStructureCmd.Flags().BoolVar(&structAllowFlatLayouts, "allow-flat-layouts", false,
 		"allow files at the skill root without warnings and treat them as standard content for token counting")
+	validateStructureCmd.Flags().StringSliceVar(&structAllowDirs, "allow-dirs", nil,
+		"comma-separated list of directory names to accept without warnings (e.g. --allow-dirs=evals,testing)")
 	validateCmd.AddCommand(validateStructureCmd)
 }
 
@@ -43,6 +46,7 @@ func runValidateStructure(cmd *cobra.Command, args []string) error {
 		SkipOrphans:           skipOrphans,
 		AllowExtraFrontmatter: structAllowExtraFrontmatter,
 		AllowFlatLayouts:      structAllowFlatLayouts,
+		AllowDirs:             structAllowDirs,
 	}
 	eopts := exitOpts{strict: strictStructure}
 

structure/checks.go

@@ -42,10 +42,18 @@ var knownExtraneousFiles = map[string]string{
 // CheckStructure validates the directory layout of a skill package. It checks
 // for the required SKILL.md file, flags unrecognized directories and extraneous
 // root files, and warns about deep nesting in recognized directories.
+// Directories listed in opts.AllowDirs are accepted without warning and are
+// exempt from deep-nesting checks.
 func CheckStructure(dir string, opts Options) []types.Result {
 	ctx := types.ResultContext{Category: "Structure"}
 	var results []types.Result
 
+	// Build a set of user-allowed directories.
+	allowedDirs := make(map[string]bool, len(opts.AllowDirs))
+	for _, d := range opts.AllowDirs {
+		allowedDirs[d] = true
+	}
+
 	// Check SKILL.md exists
 	skillPath := filepath.Join(dir, "SKILL.md")
 	if _, err := os.Stat(skillPath); os.IsNotExist(err) {
@@ -72,7 +80,7 @@ func CheckStructure(dir string, opts Options) []types.Result {
 			}
 			continue
 		}
-		if !recognizedDirs[name] {
+		if !recognizedDirs[name] && !allowedDirs[name] {
 			msg := fmt.Sprintf("unknown directory: %s/", name)
 			if subEntries, err := os.ReadDir(filepath.Join(dir, name)); err == nil {
 				fileCount := 0
@@ -93,7 +101,9 @@ func CheckStructure(dir string, opts Options) []types.Result {
 		}
 	}
 
-	// Check for deep nesting in recognized directories
+	// Check for deep nesting in recognized directories only.
+	// Allowed directories are exempt because the validator cannot know
+	// their expected internal structure.
 	for dirName := range recognizedDirs {
 		subdir := filepath.Join(dir, dirName)
 		if _, err := os.Stat(subdir); os.IsNotExist(err) {

structure/checks_test.go

@@ -205,4 +205,96 @@ func TestCheckStructure(t *testing.T) {
 		results := CheckStructure(dir, Options{AllowFlatLayouts: true})
 		requireResultContaining(t, results, types.Warning, "unknown directory: extras/")
 	})
+
+	t.Run("allow-dirs suppresses warning for allowed directory", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "evals/evals.json", `{"tests": []}`)
+		results := CheckStructure(dir, Options{AllowDirs: []string{"evals"}})
+		requireResult(t, results, types.Pass, "SKILL.md found")
+		requireNoLevel(t, results, types.Warning)
+	})
+
+	t.Run("allow-dirs with multiple directories", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "evals/evals.json", `{"tests": []}`)
+		writeFile(t, dir, "testing/test1.md", "test content")
+		results := CheckStructure(dir, Options{AllowDirs: []string{"evals", "testing"}})
+		requireResult(t, results, types.Pass, "SKILL.md found")
+		requireNoLevel(t, results, types.Warning)
+	})
+
+	t.Run("allow-dirs partial allows still warn for non-allowed dirs", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "evals/evals.json", `{"tests": []}`)
+		writeFile(t, dir, "extras/file.md", "content")
+		results := CheckStructure(dir, Options{AllowDirs: []string{"evals"}})
+		requireNoResultContaining(t, results, types.Warning, "evals/")
+		requireResultContaining(t, results, types.Warning, "unknown directory: extras/")
+	})
+
+	t.Run("allow-dirs silently accepts already-recognized directory", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "scripts/setup.sh", "#!/bin/bash")
+		results := CheckStructure(dir, Options{AllowDirs: []string{"scripts"}})
+		requireResult(t, results, types.Pass, "SKILL.md found")
+		requireNoLevel(t, results, types.Warning)
+	})
+
+	t.Run("allow-dirs exempt from deep nesting checks", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "evals/files/test1.txt", "test input")
+		results := CheckStructure(dir, Options{AllowDirs: []string{"evals"}})
+		requireResult(t, results, types.Pass, "SKILL.md found")
+		requireNoResultContaining(t, results, types.Warning, "deep nesting")
+	})
+
+	t.Run("allow-dirs does not exempt recognized dirs from deep nesting", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "evals/files/test1.txt", "test input")
+		if err := os.MkdirAll(filepath.Join(dir, "references", "subdir"), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		results := CheckStructure(dir, Options{AllowDirs: []string{"evals"}})
+		requireNoResultContaining(t, results, types.Warning, "evals/")
+		requireResult(t, results, types.Warning, "deep nesting detected: references/subdir/")
+	})
+
+	t.Run("allow-dirs with allow-flat-layouts", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "README.md", "readme")
+		writeFile(t, dir, "notes.txt", "notes")
+		writeFile(t, dir, "evals/evals.json", `{"tests": []}`)
+		results := CheckStructure(dir, Options{AllowFlatLayouts: true, AllowDirs: []string{"evals"}})
+		requireResult(t, results, types.Pass, "SKILL.md found")
+		requireNoLevel(t, results, types.Warning)
+	})
+
+	t.Run("allow-dirs with allow-flat-layouts still warns on non-allowed dirs", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "README.md", "readme")
+		writeFile(t, dir, "extras/file.md", "content")
+		writeFile(t, dir, "evals/evals.json", `{"tests": []}`)
+		results := CheckStructure(dir, Options{AllowFlatLayouts: true, AllowDirs: []string{"evals"}})
+		requireNoLevel(t, results, types.Error)
+		requireNoResultContaining(t, results, types.Warning, "README.md")
+		requireNoResultContaining(t, results, types.Warning, "evals/")
+		requireResultContaining(t, results, types.Warning, "unknown directory: extras/")
+	})
+
+	t.Run("allow-dirs hint still shown for non-allowed unknown dirs", func(t *testing.T) {
+		dir := t.TempDir()
+		writeFile(t, dir, "SKILL.md", "content")
+		writeFile(t, dir, "evals/evals.json", `{"tests": []}`)
+		writeFile(t, dir, "extras/file.md", "content")
+		results := CheckStructure(dir, Options{AllowDirs: []string{"evals"}})
+		requireResultContaining(t, results, types.Warning, "should this be references/ or assets/?")
+	})
 }