Skill auto-triggering: recall=0% in headless mode (claude -p) regardless of description content

[daviddelven]

Summary

Custom skills never auto-trigger when invoked via claude -p (non-interactive/headless mode), regardless of how explicit or directive the skill description is. Precision is 100% (no false positives), but recall is persistently 0% across multiple description iterations.

Environment

• OS: Windows 11 Home 10.0.26200
• Claude plan: Claude Pro (OAuth — no ANTHROPIC_API_KEY)
• Claude Code version: latest (VS Code extension + CLI)
• skill-creator: marketplace version (Skills 2.0)
• Skill location: C:\Users\<user>\.claude\skills\<skill-name>\SKILL.md

Steps to Reproduce

1. Create a custom skill in ~/.claude/skills/<skill>/SKILL.md with a description following Skills 2.0 format
2. Create a trigger eval set (trigger-eval.json) with 20 queries (10 should-trigger, 10 should-not)
3. Run run_eval.py from skill-creator:

   python -m scripts.run_eval \
     --eval-set trigger-eval.json \
     --skill-path ~/.claude/skills/<skill> \
     --runs-per-query 3 --verbose

4. Observe results

Observed Behavior

All should-trigger queries score trigger_rate: 0.0 — the skill never auto-triggers in any of the claude -p subprocess calls spawned by run_eval.py.

Example output:

[FAIL] rate=0/3 expected=True: can you search youtube for claude code tutorials?
[FAIL] rate=0/3 expected=True: find me 5 youtube videos about python async programming
[FAIL] rate=0/3 expected=True: /yt-search IfcOpenShell tutorial
[PASS] rate=0/3 expected=False: how do i download a youtube video to watch offline?
[PASS] rate=0/3 expected=False: search google for recent news about llm agents

Results: 10/20 passed — precision=100%, recall=0%, accuracy=50%

Description Tested (v2 — very explicit)

description: >
  Fetches live YouTube search results via yt-dlp — real titles, channels,
  view counts, upload dates, and URLs that Claude cannot know from training data.
  ALWAYS use this skill whenever the user wants to search YouTube or find YouTube
  videos on any topic. Never answer YouTube search requests from memory; only this
  skill can provide current, accurate video results.
  Trigger on any request containing: "search youtube", "find youtube videos",
  "look up youtube tutorials", "youtube results", "yt-search", "/yt-search",
  "search yt", "pull up youtube videos", "what youtube videos", "find me videos about".
  Do NOT trigger for: summarizing a specific video URL, downloading videos, or
  general web searches with no YouTube intent.

Even with ALWAYS, explicit keyword lists, and negative examples, recall remains 0%.

Expected Behavior

Skills with highly directive descriptions matching the query content should auto-trigger with meaningful recall (>50%) in headless mode.

Additional Notes

• The skill works perfectly when invoked explicitly via /yt-search <query> slash command
• run_loop.py (the automated optimization loop) is not usable for Claude Pro users — it calls anthropic.Anthropic() SDK directly and requires ANTHROPIC_API_KEY, which is not available with OAuth authentication
• The run_eval.py script required a Windows-specific fix: select.select() does not work with subprocess pipes on Windows (WinError 10038), replaced with a queue.Queue + background thread pattern
• Tested across 2 full description iterations with no improvement in recall

Workaround

None for auto-triggering. Users must invoke skills explicitly via slash commands.

Suggested Investigation

• Does claude -p pass available skills to the model differently than interactive mode?
• Is there a flag or env variable to enable skill auto-triggering in headless mode?
• Should run_eval.py work differently for Pro/OAuth users (no API key)?

[nathanschram]

Confirming: 0% recall for project-local plugin skills in claude -p

We're experiencing the identical issue with our ContextDocs plugin (3 skills: ai-context, context-guard, context-verify). Extensive testing confirms this is a platform-level limitation, not a description quality issue.

Our data

• 300 claude -p invocations across 5 iterations of automated description optimisation (using the skill-creator plugin's run_loop.py)
• 0% trigger rate on all iterations — no description change improved activation
• 20 eval queries (10 should-trigger, 10 should-not), 3 runs each, 5 iterations
• Model: claude-haiku-4-5-20251001
• Platform: Linux (Ubuntu), Claude Code 2.1.72, Anthropic API key auth

Key additional finding: global vs project-local disparity

We have a sibling plugin (PitchDocs) that is globally installed with pitchdocs: prefixed namespace. In the same eval harness, PitchDocs achieves 69% slash command activation while our project-local ContextDocs gets 6.7%.

This suggests headless mode loads global plugin manifests correctly but skips or partially loads the dynamic project-local plugin discovery that the interactive REPL performs.

Evidence from raw stream-json output

Examining the raw --output-format stream-json output from claude -p:

• The init event's skills array does include our unprefixed skills (ai-context, context-guard, context-verify)
• The slash_commands list also includes them
• But the model never invokes the Skill tool — it receives the NL query and proceeds to do the work directly using Bash, Read, Write etc., bypassing the skill entirely
• With prefixed form (/contextdocs:ai-context), the CLI parser returns "Unknown skill" — confirming the namespace isn't registered for project-local plugins

Separate activation eval results (not using run_loop)

We also ran activation evals via a separate harness (run-activation-evals.sh) with 20 test cases × 3 runs:

Category	Pass	Total	Rate
Slash commands	2	30	6.7%
NL triggers	6	15	40%
Negative tests	15	15	100%

NL triggers occasionally work (nondeterministic), confirming the skill metadata IS present in context but the model doesn't reliably consult it.

Multi-model consensus

We consulted GPT-5.4 and Gemini 3.1 Pro via external analysis. Both independently concluded (8-9/10 confidence) that this is a headless mode architectural limitation where the REPL's dynamic local namespace resolution is bypassed in single-shot -p execution.

Also related

• [BUG] Skill tool description example teaches model to pass short names, causing "Unknown skill" errors #26436 — Skill tool examples teach short names but plugins register fully-qualified names
• [BUG] /install-github-app generates claude-code-review.yml using a terminal CLI plugin in CI, causing silent failures #26227 — code-review plugin designed for interactive terminal doesn't work in headless CI

Workaround

For now, temporarily installing the plugin globally (so it gets a pluginname: namespace) restores headless mode functionality. But this shouldn't be necessary — project-local plugins should have parity with global ones in -p mode.

[GMedlin]

Additional root cause: run_eval.py detection logic incompatible with deferred tools

We've identified concrete bugs in the skill-creator plugin's run_eval.py that produce false 0% trigger rates, independent of the headless mode namespace issue described above.

Bug 1: ToolSearch not recognized as passthrough

run_eval.py line ~137-141 checks if Claude's first tool call is Skill or Read:

if tool_name in ("Skill", "Read"):
    pending_tool_name = tool_name
    accumulated_json = ""
else:
    return False  # Any other tool = not triggered

Claude Code now uses deferred tools -- the Skill tool isn't available until Claude calls ToolSearch to fetch its schema. So Claude's actual sequence is:

1. ToolSearch (to resolve the deferred Skill tool)
2. Skill (the actual skill invocation)

But step 1 hits the else: return False branch, killing detection before step 2 ever happens.

Bug 2: Multi-message flow not handled

Even with ToolSearch recognized, the message_stop handler returns False after the first message. The Skill call comes in message 2 (after ToolSearch resolves in message 1). The script needs to continue reading across message boundaries when ToolSearch was seen.

Bug 3: Installed skill name not detected

The eval creates a temp command {name}-skill-{uuid} and checks for that exact name. But when the real skill is already installed, Claude calls it by its registered name (e.g. design) rather than the temp command name. Detection always misses.

Bug 4: Race condition on process exit

The threading-based pipe reader (needed for Windows compat -- see #33467) can miss final events if the subprocess exits before the reader thread appends the last lines. Without reader.join() + a final drain pass, the script returns False on events it never processed.

Patched version

We maintain a standalone patched eval runner at run_eval_win.py with 5 fixes:

1. select.select replaced with threading-based pipe reader (Windows compat)
2. ToolSearch recognized as passthrough tool
3. Multi-message flow: continues reading across message_stop boundaries when ToolSearch was seen
4. Dual skill name detection: checks for both temp command name AND installed skill name
5. reader.join() + final drain loop to prevent race condition false negatives

Plus minor improvements: time.monotonic() for timeout, dead code removal.

Eval results with patched runner

25-query eval set (12 should-trigger, 13 should-not-trigger), 3 runs per query, Opus 4.6:

• Overall: 15/25 (60%)
• Should-trigger: 3/12 (25%) -- core design queries (pricing page build, Stitch design.md import, component build) trigger reliably at 67-100%
• Should-NOT-trigger: 12/13 (92%) -- excellent rejection rate; only false positive is an image generation query with design-adjacent language

The remaining should-trigger gaps are an eval methodology limitation: claude -p with 60+ available skills rationally handles many frontend queries directly rather than routing through a skill. In real sessions, CLAUDE.md routing tables provide the additional context needed.

Related issues

• skill-creator: run_loop.py fails on Windows with socket error in claude -p subprocess #33467 -- select.select Windows socket bug (same root cause as Fix 1)
• The ToolSearch/deferred tools interaction (Fixes 2-3) affects any eval tooling that inspects Claude's first tool call

[dzivkovi]

Confirming on identical setup (Windows 11 build 10.0.26200, Claude Code latest, marketplace skill-creator) and adding two pieces of root-cause evidence:

1. The skill triggers in interactive sessions but not via claude -p.

Same query, same description, same temporary command file (mimicking what run_eval.py creates):

• cwd=<project-with-CLAUDE.md> → skill triggers, full pipeline runs end-to-end
• cwd=~ → no trigger

This points at find_project_root() in scripts/run_eval.py walking up looking for any .claude/ directory. From the skill-creator dir on Windows it lands at ~ (because ~/.claude/ exists), so the harness invokes claude -p with cwd=~. No project anchor, no CLAUDE.md, no project-level skills — and the model behaves more cautiously in that empty context. May be the underlying cause of the recall asymmetry.

2. There's also a separate detection bug in _process_event.

The function returns False after the first non-Skill tool_use it sees in the stream. Modern Sonnet/Opus often invoke Bash, Read, or Agent before committing to the Skill tool. Empirically (manual claude -p run with stream-json captured): a successful skill invocation involved 8 total tool_use events with the actual Skill call as #6. The harness exited at #1 and missed it.

Patched locally with a "keep scanning until result event" rewrite + early-True only on confirmed match. Happy to draft a PR if there's interest in addressing the project-root resolution and _process_event early-return together — both surfaced through the same investigation.