feat: drive interactive skills via an LLM responder (#303)

[adamdougal]

Summary

Adds a responder — an LLM-backed surrogate user that drives interactive (multi-turn) skills during evals. When an agent asks a follow-up question, the responder classifies it and decides whether to reply, stop the conversation, or abstain (the question can't be answered from its brief), letting us evaluate back-and-forth skills without scripting every turn. It reuses the same Copilot engine as the agent under test (no extra LLM deployment) but runs in its own isolated, persistent session, configured per task under inputs.responder.

Related issue

Closes #303

Agent handoff

• Scope: New responder feature end-to-end — config + validation, classifier, orchestration loop, outcome surfacing through the web API and dashboard, JSON schema, and docs.
• Key files changed: internal/models/testcase.go & outcome.go (config + ResponderInfo), internal/responder/responder.go (classifier with persistent session + teardown), internal/orchestration/runner.go (executeResponderLoop, injectable classifier factory), internal/execution/copilot.go (DeleteSession), internal/webapi/{types,store}.go, web/src/components/RunDetail.tsx & api/client.ts (responder badge), schemas/task.schema.json, README + site/ docs.
• Important decisions: inputs.responder is a sibling of follow_up_prompts and mutually exclusive with it; responder runs in a separate, non-ephemeral Copilot session with explicit Close() teardown to avoid polluting the agent transcript; abstain marks the task errored, stop ends normally, cap exhaustion stops the loop and grades what exists; model is optional and defaults to the eval's config.model; each task builds its own classifier (concurrency-safe).
• Follow-ups or known gaps: The completed outcome value is effectively unreachable in practice (a self-initiated stop returns stopped); left as-is and considered acceptable.

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor or maintenance
• CI/CD or release change

Validation

• go test ./...
• make lint or golangci-lint run
• Docs site checked, if docs changed
• Web/dashboard checks run, if web/ changed
• Manual validation completed: run-detail Playwright e2e (chromium) 5/5 passing
• Not applicable; reason:

Documentation

• README updated, if user-facing behavior changed
• site/ docs updated, if CLI, YAML, dashboard, or validator behavior changed
• Examples updated, if relevant
• Not applicable

Risk and rollback

• Risk level: Low
• Rollback plan: Feature is fully additive and gated on the new optional inputs.responder field — tasks without it are unaffected. Revert the branch's commits (or the squash-merge commit) to fully remove it; no data migrations or schema-compat concerns.

Notes for reviewers

The responder's session lifecycle is the area most worth a close look: Classify lazily creates a persistent session on the first call and resumes it thereafter, and executeResponderLoop defers Close() with a detached 30s context so teardown still runs on cancellation. CopilotEngine.DeleteSession removes the session from both e.sessions and e.usageCollectors (the latter fixed a collector leak). Also worth confirming: the load-time mutual-exclusivity validation between responder and follow_up_prompts, and that the orchestration branch gives Responder precedence over FollowUps

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds an LLM-backed "responder" that role-plays the user for interactive, multi-turn skills, enabling evaluation of skills whose follow-up questions cannot be pre-scripted.

Changes:

• New internal/responder package implementing a Classifier that drives a persistent surrogate-user session and emits reply/stop/abstain decisions via structured tool calls.
• Runner integration (executeResponderLoop) that drives the agent loop, merges responses, and records a ResponderInfo summary with outcomes completed/stopped/abstained/cap_exhausted/error.
• Config/schema/validation, API/dashboard surfacing, docs, and tests for the new inputs.responder field.

Show a summary per file

File	Description
internal/responder/responder.go	New responder Classifier with persistent session + 3 decision tools.
internal/responder/responder_test.go	Unit tests for tools, session reuse, cleanup, and model defaulting.
internal/orchestration/runner.go	Adds executeResponderLoop/sendResponderReply and newClassifier hook.
internal/orchestration/responder_loop_test.go	Tests reply→stop, abstain→error, cap-exhausted scenarios.
internal/models/testcase.go	Adds ResponderConfig on TaskStimulus + validation.
internal/models/testcase_test.go	Validation tests for responder config.
internal/models/outcome.go	Adds ResponderInfo and outcome constants.
internal/models/outcome_test.go	JSON serialization test for Responder.
internal/execution/copilot.go	New DeleteSession for explicit teardown.
internal/webapi/types.go	Adds ResponderInfoResponse.
internal/webapi/store.go	Maps run.Responder to API response.
internal/webapi/additional_test.go	Test for responder mapping.
internal/validation/schema_test.go	Schema acceptance test for responder.
schemas/task.schema.json	Schema for inputs.responder.
web/src/api/client.ts	TypeScript ResponderInfo type.
web/src/components/RunDetail.tsx	ResponderBadge for task rows.
web/dist/index.html	Rebuilt asset reference.
site/src/content/docs/, README.md, docs/plans/	Documentation and design notes.

Copilot's findings

• Files reviewed: 20/21 changed files
• Comments generated: 3

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.06977% with 45 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@7692027). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
internal/orchestration/runner.go	55.95%	27 Missing and 10 partials ⚠️
internal/responder/responder.go	94.05%	4 Missing and 2 partials ⚠️
internal/models/testcase.go	84.61%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #304   +/-   ##
=======================================
  Coverage        ?   75.55%           
=======================================
  Files           ?      162           
  Lines           ?    19368           
  Branches        ?        0           
=======================================
  Hits            ?    14633           
  Misses          ?     3686           
  Partials        ?     1049           

Flag	Coverage Δ
go-implementation	75.55% <79.06%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[spboyer]

Second pass on the responder loop after the earlier round of fixes landed. The session lifecycle, mutual-exclusivity validation, and dashboard mapping all look clean. A few nits worth tightening:

Issues to address:

• internal/responder/responder.go:79 - decisionRecorder mutated from SDK tool-handler goroutines without synchronization (real race if the model ever emits parallel tool calls)
• internal/models/outcome.go:77 - doc comment still lists the removed completed outcome value
• schemas/task.schema.json:159 - schema doesn't enforce responder ↔ follow_up_prompts mutual exclusivity that Validate() rejects at runtime

[Copilot]

Copilot's findings

• Files reviewed: 20/21 changed files
• Comments generated: 5

[Copilot]

Copilot's findings

• Files reviewed: 21/22 changed files
• Comments generated: 4