Harden VS Code extension E2E tests by adamint · Pull Request #18217 · microsoft/aspire

adamint · 2026-06-15T17:25:52Z

Description

This hardens the VS Code extension E2E shards that have been flaky in recent CI runs.

The recurring actionable failures were mostly two patterns:

zero-to-running timed out waiting for an integrated-browser editor title containing the dashboard host even though the dashboard URL was known and the HTTP probe succeeded.
Several endpoint/browser checks relied on VS Code editor titles as the signal that navigation happened, which is fragile under runner load.

This PR removes brittle integrated-browser title waits where the test already has stronger HTTP/state coverage, makes the tree action browser command await simpleBrowser.show and return the resolved endpoint URL through the E2E bridge, changes the debug-dashboard build-failure test to assert the stable debug-console failure summary/log path and then verify the injected compiler symbol in that surfaced CLI log, forces Aspire CLI E2E output to English, writes E2E control payloads atomically, and retries mutable fixture file writes on Windows file-lock errors.

I also checked the other recent failing shards. The tree-actions and apphost-tree failures were clustered in a cancelled PR-specific run that failed with ASPIRE009 from a CLI bundle experiment. The one standalone discovery-configuration failure was a Windows EBUSY while writing .aspire/settings.json, which is covered by the retry helper here.

Validation:

cd extension
corepack yarn run compile-e2e
corepack yarn run lint
corepack yarn run test

corepack yarn run test finished with 877 passing and 1 pending.

Fixes # (issue)

Checklist

Is this feature complete?
- Yes. Ready to ship.
- No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
- Yes
- No
Did you add public API?
- Yes
  - If yes, did you have an API Review for it?
    - Yes
    - No
  - Did you add <remarks /> and <code /> elements on your triple slash comments?
    - Yes
    - No
- No
Does the change make any security assumptions or guarantees?
- Yes
  - If yes, have you done a threat model and had a security review?
    - Yes
    - No
- No

Reduce E2E flakes by removing brittle integrated-browser title waits where HTTP/state checks already prove readiness, checking build failure diagnostics via the CLI log path, forcing the CLI locale to English, and adding Windows-safe retries for mutable fixture files and atomic E2E control writes.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-15T17:26:16Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18217

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18217"

Copilot

Pull request overview

This PR hardens VS Code extension E2E tests that were flaky in CI by addressing two main failure patterns: integrated-browser editor title waits timing out (replaced with HTTP/state coverage that already existed), and debug console text waits timing out for build failure symbols (replaced with log file validation). Additionally, it forces English CLI output, makes E2E control file writes atomic, and adds retry logic for Windows file-lock errors.

Changes:

Removes brittle waitForEditorTitle assertions in zeroToRunning and treeActions tests where HTTP probes already provide equivalent coverage, and validates build-failure symbols via the CLI log file rather than debug console text.
Adds atomic write (writeJsonFileAtomic) for E2E control payloads and retry-on-EBUSY/EPERM (writeFileWithRetry) for mutable fixture files to prevent Windows file-lock flakiness.
Sets DOTNET_CLI_UI_LANGUAGE: 'en' in the E2E CLI environment to prevent localized output from breaking text-matching assertions.

Show a summary per file

File	Description
`extension/scripts/run-e2e.js`	Adds `DOTNET_CLI_UI_LANGUAGE: 'en'` to the CLI environment
`extension/src/test-e2e/helpers/assertions.ts`	Introduces `writeJsonFileAtomic`, `renameFileWithRetry`, `isRetryableRenameError`, and `sleepSynchronously` for atomic control file writes
`extension/src/test-e2e/helpers/fixtures.ts`	Introduces `writeFileWithRetry`, `isRetryableFileSystemError`, and `sleepSynchronously` for retrying mutable fixture file writes on Windows
`extension/src/test-e2e/zeroToRunning.e2e.test.ts`	Removes the flaky `waitForEditorTitle(dashboardHost)` assertion
`extension/src/test-e2e/treeActions.e2e.test.ts`	Removes `waitForEditorTitle` import and usage for endpoint URL host check
`extension/src/test-e2e/debugDashboard.e2e.test.ts`	Replaces debug console symbol wait with log-file-based validation via `getLogPathFromDebugConsoleOutput` and `waitForLogFileText`
`extension/src/test/e2eLaunchProfile.test.ts`	Updates meta-test assertions to verify the new patterns (atomic writes, retry helpers, no editor title waits)

Copilot's findings

Files reviewed: 7/7 changed files
Comments generated: 0

Keep the debug-dashboard sentinel assertion on the debug console flush path, and make the tree action browser-open command await simpleBrowser.show while returning the resolved endpoint URL through the E2E bridge.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Avoid waiting for compiler diagnostics to stream inline to the debug console. Assert the stable debug-console failure summary and log path, then verify the injected compiler symbol in the surfaced CLI log.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 9/9 changed files
Comments generated: 0 new

Suppress intentional experimental interaction APIs in the generated E2E AppHost fixture so local fallback SDKs do not fail startup before tree-action coverage can run. Also retry EBUSY during atomic E2E control-file renames and keep a lightweight debug-console compiler diagnostic assertion while validating the full injected symbol in the CLI log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-15T18:11:41Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Assert terminal hyperlink output before parsing the debug-dashboard log path, so regressions fail with the intended message. Scope the EBUSY meta-test to the atomic rename helper so it actually guards the Windows control-file rename retry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 9/9 changed files
Comments generated: 1

Export the existing synchronous sleep helper from assertions and reuse it in fixture writes instead of maintaining a second Atomics.wait implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

github-actions · 2026-06-15T19:32:13Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Use lightweight file-based secondary AppHost candidates for discovery-only E2E tests to avoid CI MSBuild contention, and wait for stopped AppHosts to disappear before deleting fixture directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Assert restored discovery clears stale secondary AppHost candidates and retry EACCES for Windows fixture writes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 0 new

Retry EBUSY for extension-side atomic state-file renames and give command-palette E2E enough timeout headroom for slow discovery diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-15T20:32:30Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Avoid failing teardown when extension state is stale after the Aspire CLI has already stopped the AppHost. Wait on the captured AppHost PID instead, and force cleanup only when that process is still alive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 0 new

Keep successful Aspire CLI stop teardown wait-only so stale extension state cannot terminate a recycled PID. If stop fails or times out, require the captured AppHost process to exit before reporting teardown success. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When Aspire CLI stop times out or reports a stop failure, use aspire ps --format json to confirm whether the AppHost is still visible before failing teardown. This keeps stale extension state from failing cleanup while preserving failure when the CLI still reports the AppHost running. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 0 new

Use aspire ps as the stop-failure authority, wait on the PID reported by ps, and re-check ps before treating teardown as complete. Also route AppHost source/config cleanup through retrying helpers so Windows file locks do not short-circuit cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 1

+}
+async function waitForRunningAppHostProcessExitFromState(appHostPath: string, timeoutMs: number): Promise<void> {


github-actions · 2026-06-15T22:23:11Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

github-actions · 2026-06-15T23:14:29Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

ellahathaway · 2026-06-15T23:54:51Z

+        const openedEndpoint = await executeE2eControlCommand({ name: 'openInIntegratedBrowser', appHostPath, resourceName: 'e2e-worker' });
        await waitForCommandOutcome('aspire-vscode.openInIntegratedBrowser', 'success', 60000, before);
-        assert.ok((await waitForEditorTitle(new URL(endpointUrl).host, 120000, { matchCase: false })).toLowerCase().includes(new URL(endpointUrl).host.toLowerCase()));
+        assert.strictEqual((openedEndpoint.result as { url?: string }).url, endpointUrl);


This assertion verifies the endpoint is healthy, but it doesn’t prove the integrated browser actually opened the expected URL in VS Code. The test can still pass if openInIntegratedBrowser is a no-op or opens the wrong target, as long as the endpoint responds.

Copilot AI review requested due to automatic review settings June 15, 2026 17:25

Copilot started reviewing on behalf of adamint June 15, 2026 17:26 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

adamint added 2 commits June 15, 2026 13:35

Copilot AI review requested due to automatic review settings June 15, 2026 17:40

Copilot started reviewing on behalf of adamint June 15, 2026 17:42 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jun 15, 2026

[repo-pulse] 📊 Repo Pulse — Daily Activity Dashboard #16404

Open

Copilot AI review requested due to automatic review settings June 15, 2026 18:24

Copilot started reviewing on behalf of adamint June 15, 2026 18:25 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

Comment thread extension/src/test-e2e/helpers/fixtures.ts Outdated

adamint marked this pull request as ready for review June 15, 2026 18:36

Address extension E2E helper duplication

401cb3e

Export the existing synchronous sleep helper from assertions and reuse it in fixture writes instead of maintaining a second Atomics.wait implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

adamint marked this pull request as draft June 15, 2026 18:38

adamint marked this pull request as ready for review June 15, 2026 18:41

Copilot AI review requested due to automatic review settings June 15, 2026 18:41

Copilot started reviewing on behalf of adamint June 15, 2026 18:42 View session

Copilot AI reviewed Jun 15, 2026

adamint marked this pull request as draft June 15, 2026 19:41

adamint and others added 2 commits June 15, 2026 15:46

Address VS Code E2E review hardening

7b9d558

Assert restored discovery clears stale secondary AppHost candidates and retry EACCES for Windows fixture writes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 15, 2026 19:58

Copilot started reviewing on behalf of adamint June 15, 2026 19:58 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

Harden extension E2E state writes

c93415c

Retry EBUSY for extension-side atomic state-file renames and give command-palette E2E enough timeout headroom for slow discovery diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 15, 2026 20:38

Copilot started reviewing on behalf of adamint June 15, 2026 20:39 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

adamint and others added 2 commits June 15, 2026 16:54

Copilot AI review requested due to automatic review settings June 15, 2026 21:04

Copilot started reviewing on behalf of adamint June 15, 2026 21:05 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

adamint and others added 2 commits June 15, 2026 17:19

Fix VS Code E2E debug and cleanup hardening

442779a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 15, 2026 21:51

Copilot started reviewing on behalf of adamint June 15, 2026 21:51 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

Comment thread extension/src/test-e2e/helpers/fixtures.ts

Comment on lines +340 to +341

}

async function waitForRunningAppHostProcessExitFromState(appHostPath: string, timeoutMs: number): Promise<void> {

ellahathaway reviewed Jun 15, 2026

View reviewed changes

adamint marked this pull request as ready for review June 16, 2026 00:04

		}
		async function waitForRunningAppHostProcessExitFromState(appHostPath: string, timeoutMs: number): Promise<void> {

Conversation

adamint commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

github-actions Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

ellahathaway Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adamint commented Jun 15, 2026 •

edited

Loading

github-actions Bot commented Jun 15, 2026 •

edited

Loading