feat: complete issue #322 with triage automation and regression loop#326
Merged
github-actions[bot] merged 12 commits intoJun 15, 2026
Conversation
Replace TODO placeholders in squad CI/preview/release workflows with: - Go mod verify, go vet, go test with coverage, race detector - Node.js web UI build (Node 22 with npm cache) - Binary build and integration test for end-to-end validation - Release creation with gh CLI (main) and pre-release (insider) All workflows now mirror go-ci.yml proven build/test pattern: - squad-ci.yml: Runs on dev/preview/main/insider branches and PRs - squad-preview.yml: Validates on preview branch pushes - squad-release.yml: Runs tests and creates release on main - squad-insider-release.yml: Runs tests and creates pre-release on insider Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Phase 2: Implement genuine failure-handling for evaluation runs - Add failures/handler.go with CaptureFailure method - Capture stderr/stdout (10KB truncation), exit code, failed validators - Extract error patterns via regex (timeout, OOM, permission, connection, panic, exception) - Generate human-readable triage summary in Markdown with actionable recommendations - Extend RunResult.FailureArtifacts for diagnostic data in results JSON - Add comprehensive test coverage (handler_test.go) - All tests passing, 79.1% coverage maintained Next: Wire handler into EvalRunner, add --auto-file-issue flag, display triage in CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#322 - Revert go.mod back to 1.26 (dependency azure-dev/cli/azd requires >= 1.26) - Restore go-version: 1.26 in go-ci.yml and squad-ci.yml - Add linters.exclusions.paths to .golangci.yml to skip web/node_modules and web/dist (uses v2 schema: linters.exclusions.paths) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…322 v4 with 'latest' resolves to v1.64.8 (built with Go 1.24) which cannot lint Go 1.26 projects. Upgrade to action@v7 with pinned v2.10.1 (built with Go 1.26) to match go-ci.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add exitCode parameter to CaptureFailure and populate ExitCode field - Sort FailedGraders for deterministic output - Sort extractErrorPatterns result for deterministic output - Fix contains() helper to use strings.Contains - Fix TestTruncate: correct wantLen and assert actual length - Fix TestExtractErrorPatterns: assert specific expected patterns - Update TestCaptureFailure: pass exit code, assert ExitCode field - Drop unused contains() helper (now using strings.Contains directly) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use 'gh release view' to check if a release exists before creating, so genuine failures (permission errors, network issues) surface as workflow failures rather than being swallowed by '|| echo'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- handler.go: use time.Now().UTC() for cross-machine consistency - handler.go: filter empty strings from extractErrorPatterns matches - handler.go: fix truncate to enforce hard maxLen (prefix len = maxLen - suffix len) - handler_test.go: update TestTruncate wantLen to maxLen (30) after hard-limit fix - squad-preview/release/insider-release.yml: add favicon.svg to web/dist stub - squad-release/insider-release.yml: change 'passed' to 'completed' in integration test message since exit code 1 (eval failures) is treated as non-fatal - go-ci.yml lint job: fix go-version indentation to 10 spaces under with: - squad-ci.yml lint job: fix go-version indentation to 10 spaces under with: Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- handler_test.go: remove unused truncSuffix const (fixes lint) - handler.go: use regexp.MustCompile for hard-coded patterns to surface typos at runtime instead of silently ignoring compile errors - squad-ci.yml: add lfs: true to lint job checkout to match build job - .golangci.yml exclusions.paths is valid v2 config (config verify passes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In golangci-lint v2, path exclusions belong under the top-level exclusions key, not under linters.exclusions. Moving them ensures they are applied correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rove-waza-with-real-agentic-product-c-577bc9 # Conflicts: # .github/workflows/squad-insider-release.yml # .github/workflows/squad-release.yml # internal/failures/handler.go # internal/failures/handler_test.go
- add CLI auto issue filing and triage highlights for failed runs - wire failure artifact capture into runner execution paths - add weekly regression loop workflow with baseline artifact comparison and issue upsert - harden auto-merge workflow with trusted-author + label gates - document new run flag and CI workflows, and fix lint package resolution Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds automated triage + regression loop capabilities to Waza, including capturing failure artifacts, surfacing triage summaries in CLI output, and integrating recurring regression detection + issue upserts via GitHub Actions.
Changes:
- Add failure artifact capture for failed/error runs and surface triage highlights in CLI summaries.
- Add
waza run --auto-file-issueto upsert a GitHub issue with aggregated triage details for failing runs. - Add a scheduled weekly regression workflow and tighten auto-merge gating; update docs accordingly.
Show a summary per file
| File | Description |
|---|---|
site/src/content/docs/reference/cli.mdx |
Documents new --auto-file-issue flag and usage example. |
site/src/content/docs/guides/ci-cd.mdx |
Adds documentation for the weekly regression loop workflow and auto-merge safety boundaries. |
README.md |
Updates CLI flag table and links to new workflows. |
Makefile |
Modifies make lint invocation logic. |
internal/orchestration/runner.go |
Captures failure artifacts for failed/error runs via a new failure handler. |
internal/orchestration/runner_test.go |
Adds/updates tests to assert failure artifacts are captured and status mapping is correct. |
internal/failures/handler.go |
Minor cleanup and truncate behavior adjustment in failure handler. |
internal/failures/handler_test.go |
Minor cleanup in tests. |
cmd/waza/cmd_run.go |
Implements --auto-file-issue, triage highlights in summary output, and GitHub issue upsert helpers via gh. |
cmd/waza/cmd_run_test.go |
Adds tests for triage highlights/headline parsing and auto-issue creation/update behavior. |
.golangci.yml |
Adds path exclusion configuration for linting. |
.github/workflows/weekly-regression-loop.yml |
Introduces a scheduled + manual regression loop that compares against latest successful baseline artifact and files issues on regressions. |
.github/workflows/auto-merge.yml |
Tightens auto-merge conditions and adds merge-state precheck. |
Copilot's findings
- Files reviewed: 13/13 changed files
- Comments generated: 5
golangci-lint config verify was failing because the top-level 'exclusions' property is not valid in the current schema. Removed this section as the web directory is not a Go package and does not need explicit exclusion. Fixes: Go Build and Test CI failure due to golangci-lint config verification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
spboyer
pushed a commit
that referenced
this pull request
Jun 15, 2026
- Simplify Makefile lint target to use filesystem paths (./...) instead of complex go list transformation with path conversion - Fix findOpenAutoIssue to properly quote GitHub search marker and separate stdout/stderr buffers to avoid JSON parse errors - Add GH_TOKEN environment variable to auto-merge.yml 'Check merge state' step - Add main branch gating to weekly-regression-loop.yml workflow via github.ref condition on evaluate job - Disable govet inline check in golangci.yml to prevent spurious failures from node_modules Go code All review comments from Copilot reviewer addressed. Tests pass, linting passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #322
What changed
--auto-file-issuetowaza runto upsert a GitHub issue with aggregated triage details for failed/error runs.auto-merge.ymlto require trusted author association,maintarget, and theagent-mergelabel.make lintpackage resolution to use repository-relative package patterns.Verification
go test ./cmd/waza ./internal/orchestrationmake lintmake buildmake testcd site && npm ci --silent && npm run build