Add localhost embeddings support for scenario automation by gryan11 · Pull Request #3947 · microsoft/vscode-copilot-chat

gryan11 · 2026-02-23T22:21:44Z

Updates to support semantic search with blackbird in MSBench. MSBench test run here: https://msbenchapp.azurewebsites.net/run-analysis/22452275228

ScenarioAutomationCAPIClientImpl: intercepts EmbeddingsCodeSearch requests, strips CAPI auth headers, sends clean JSON to localhost:4443
ScenarioAutomationGithubCodeSearchService: overrides getRemoteIndexState to always return Ready, bypassing production index checks
Register MockGithubAvailableEmbeddingTypesService in scenario automation to use metis_1024_I16_Binary without hitting production endpoint
Move all scenario-automation service overrides into the isScenarioAutomation if/else branch in services.ts

rwoll · 2026-02-24T00:18:59Z

+import { IFetcherService } from '../../networking/common/fetcherService';
+import { CAPIClientImpl } from './capiClientImpl';
+
+const SCENARIO_AUTOMATION_CODE_SEARCH_URL = 'https://localhost:4443/embeddings/code/search';


@gryan11 - Avoid hard coding this. I suggest adding a formal vscode setting, marked as experimental, that allows you to set the URL (and potentially specify an environment variable for auth).

By doing so, you'll be able to open vscode (outside test scenarios) and then configure it to use your local or (remote) staging endpoint to dogfood and debug.

There's similar settings: see github.copilot.advanced.debug.overrideCapiUrl and github.copilot.advanced.debug.overrideProxyUrl for an example. Essentially, you want a similar feature.

- ScenarioAutomationCAPIClientImpl: intercepts EmbeddingsCodeSearch requests, strips CAPI auth headers, sends clean JSON to localhost:4443 - ScenarioAutomationGithubCodeSearchService: overrides getRemoteIndexState to always return Ready, bypassing production index checks - Register MockGithubAvailableEmbeddingTypesService in scenario automation to use metis_1024_I16_Binary without hitting production endpoint - Move all scenario-automation service overrides into the isScenarioAutomation if/else branch in services.ts

Add github.copilot.chat.advanced.debug.overrideEmbeddingsUrl setting (experimental) so the embeddings code search endpoint can be configured in VS Code settings instead of being hardcoded in ScenarioAutomationCAPIClientImpl.

When set, adds Authorization: Bearer header to embeddings requests sent to the override URL in ScenarioAutomationCAPIClientImpl.

rwoll · 2026-02-26T00:17:20Z

@gryan11 - ping me on teams when you need another review.

Add ScenarioAutomation overrides so the semantic_search tool works in msbench evaluation runs with Blackbird local server: - ScenarioAutomationEndpointProviderImpl: fall back to copilot-base when a model family (e.g. copilot-fast/gpt-4o-mini) is unavailable via the capi proxy - StaticGitHubAuthenticationService: override copilotToken getter to return a placeholder when the real token is missing or is noAuth, so WorkspaceChunkSearchService can initialize - ScenarioAutomationWorkspaceChunkSearchService: new service that prevents both index states from reporting disabled simultaneously and gracefully handles searchFileChunks errors - services.ts: register the new service in isScenarioAutomation block - searchSubagentPrompt.tsx: encourage semantic_search usage in search subagent prompt - package.json/configurationService.ts: sync overrideEmbeddingsUrl default between package.json and code to fix extension activation - Remove stale copilot-chat-0.37.0.vsix that caused DEPENDENCY errors Tested: msbench run 22411601902 — 0 infra errors, semantic_search called in 3/11 tasks returning valid Blackbird code snippets. https://msbenchapp.azurewebsites.net/run-analysis/22411601902 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Override copilotToken getter in StaticGitHubAuthenticationService to return a placeholder token when overrideEmbeddingsUrl is configured and the real token is missing or noAuth. This allows WorkspaceChunkSearchService.tryInit() to pass the copilotToken check and initialize, enabling semantic_search to route requests through the Blackbird local server. The override is conditioned on DebugOverrideEmbeddingsUrl being set, so normal extension test behaviour is preserved. Also remove the silent try/catch from ScenarioAutomationWorkspaceChunkSearchService.searchFileChunks so errors propagate to logs instead of being swallowed. Test run: https://msbenchapp.azurewebsites.net/run-analysis/22452275228 (90.91% resolved, 0 errors, semantic_search returning valid Blackbird code snippets in 3/11 tasks) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The default value for DebugOverrideEmbeddingsUrl was set to 'http://localhost:4443/api/' which caused the copilotToken placeholder to always activate, breaking the ConversationFeature extension test. Reset the default to undefined so the placeholder only activates when the setting is explicitly configured (e.g. via vscodeSettings in the msbench config YAML). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds scenario-automation-specific overrides to enable semantic search/embeddings traffic to be redirected to a local endpoint (e.g. Blackbird) during MSBench runs, avoiding production-only dependencies (auth/index checks/model availability).

Changes:

Introduces scenario automation implementations for CAPI client, GitHub code search, and workspace chunk search to keep the search pipeline running and route embeddings code search to an override URL.
Adds a new advanced/debug setting (github.copilot.chat.debug.overrideEmbeddingsUrl) to configure the embeddings override endpoint.
Updates scenario automation service registration to use mock embedding model types and isolate overrides inside the isScenarioAutomation branch.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/platform/workspaceChunkSearch/node/scenarioAutomationWorkspaceChunkSearchService.ts	Forces remote index state to “initializing” when both local+remote would be disabled, so fallback search strategies still run.
src/platform/remoteCodeSearch/common/scenarioAutomationGithubCodeSearchService.ts	Overrides remote index state checks to always report “Ready” in scenario automation.
src/platform/endpoint/node/scenarioAutomationCAPIClientImpl.ts	Intercepts embeddings code search requests and forwards them (as clean JSON) to the configured override URL.
src/platform/configuration/common/configurationService.ts	Adds `ConfigKey.Advanced.DebugOverrideEmbeddingsUrl` setting definition/migration.
src/platform/authentication/common/staticGitHubAuthenticationService.ts	Returns a placeholder Copilot token when embeddings override URL is configured, to allow service initialization.
src/extension/prompt/vscode-node/scenarioAutomationEndpointProviderImpl.ts	Adds fallback to `copilot-base` when scenario automation cannot resolve a requested model family.
src/extension/extension/vscode-node/services.ts	Moves scenario automation overrides into the `isScenarioAutomation` branch; registers new scenario automation services/mocks.
package.nls.json	Adds localized description string for the new embeddings override setting.
package.json	Contributes the new `github.copilot.chat.debug.overrideEmbeddingsUrl` setting.

Copilot · 2026-02-27T21:45:20Z

+	override async getRemoteIndexState(_auth: { readonly silent: boolean }, githubRepoId: GithubRepoId, _telemetryInfo: TelemetryCorrelationId, _token: CancellationToken): Promise<Result<RemoteCodeSearchIndexState, RemoteCodeSearchError>> {
+		this._log.trace(`ScenarioAutomationGithubCodeSearchService::getRemoteIndexState(${toGithubNwo(githubRepoId)}). Returning Ready for local endpoint.`);
+		return Result.ok({ status: RemoteCodeSearchIndexStatus.Ready, indexedCommit: undefined });
+	}


Scenario automation can run with no GitHub PAT/OAuth token (createStaticGitHubTokenProvider() returns undefined when IS_SCENARIO_AUTOMATION and no env tokens are set). In that case, the inherited GithubCodeSearchService.searchRepo() will still call getGithubAccessToken() and throw "No valid auth token", which will break semantic search even though getRemoteIndexState() is forced to Ready. Consider overriding searchRepo() in this scenario service to not require a GitHub session and to issue the EmbeddingsCodeSearch request with a placeholder secretKey (relying on ScenarioAutomationCAPIClientImpl to strip auth headers) or otherwise handle unauthenticated runs gracefully.

Suggested change

}

}

/**

* Scenario automation runs may not have a real GitHub PAT/OAuth token configured.

* Override the base token acquisition to return a placeholder token so that

* GithubCodeSearchService.searchRepo() can proceed without throwing

* "No valid auth token". The ScenarioAutomationCAPIClientImpl is responsible

* for stripping or ignoring auth headers derived from this value.

*/

protected override async getGithubAccessToken(_auth: { readonly silent: boolean }): Promise<string> {

this._log.trace('ScenarioAutomationGithubCodeSearchService::getGithubAccessToken() using placeholder token for scenario automation.');

return 'scenario-automation-placeholder-token';

}

mjbvz · 2026-03-02T23:00:37Z

+ * status is reported as `initializing` so that the search pipeline proceeds
+ * to the TF-IDF / Blackbird fallback strategies.
+ */
+export class ScenarioAutomationWorkspaceChunkSearchService extends WorkspaceChunkSearchService {


Instead of extending extends WorkspaceChunkSearch (which then requires also overriding GithubCodeSearchService, can you implement the IWorkspaceChunkSearchService entirely here. Most of the operations should be noops

Extending WorkspaceChunkService is a design choice to ensure the only change in MSBench runs is using the local blackbird endpoint and otherwise all production behavior is being evaluated correctly. Otherwise, we can't guarantee MSBench evaluations are using the full production behavior for vscode agent.

Even if the current logic in WorkspaceChunkService will be functionally equivalent in MSBench evaluations now if it is removed, we can't guarantee that will always be the case in the future.

I'd like to keep them separate so that I don't have to worry about refactoring WorkspaceChunkSearchService

Look at the existing SimulationCodeSearchChunkSearchService. It's doing the same basic thing but also just implements the IWorkspaceChunkSearchService interface

Updated to address this.

- ScenarioAutomationCAPIClientImpl: wrap fetch in try/catch with contextual error message including the override URL - ScenarioAutomationGithubCodeSearchService: override searchRepo() to provide a placeholder auth token when no GitHub session is available and DebugOverrideEmbeddingsUrl is configured, preventing 'No valid auth token' errors in NoAuth/BYOK scenarios - ScenarioAutomationWorkspaceChunkSearchService: add trace log when overriding disabled → initializing state Test run: https://msbenchapp.azurewebsites.net/run-analysis/22628670595 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resolve merge conflicts: - package.json: preserve debug.overrideChatEngine and debug.overrideEmbeddingsUrl settings - services.ts: keep IGithubApiFetcherService unconditional registration, preserve conditional IGithubCodeSearchService registration for scenario automation - configurationService.ts: preserve DebugOverrideChatEngine and DebugOverrideEmbeddingsUrl config keys - Fix scenarioAutomationGithubCodeSearchService.ts import of getGithubMetadataHeaders to point directly at githubApiFetcherService (no longer re-exported from chunkingEndpointClientImpl) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…WorkspaceChunkSearchService Address mjbvz's review comment on PR microsoft#3947: instead of extending the production WorkspaceChunkSearchService class, implement the IWorkspaceChunkSearchService interface directly with mostly no-ops. - ScenarioAutomationWorkspaceChunkSearchService now implements the interface directly, calling the Blackbird endpoint via IFetcherService in searchFileChunks() without depending on any production strategies - Remove ScenarioAutomationGithubCodeSearchService entirely since the new implementation bypasses the production code search pipeline - Register production GithubCodeSearchService in scenario automation block (no longer needs a custom override) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The merge with origin/main added @opentelemetry/* packages to package.json but the lock file was not regenerated, causing esbuild failures. Run npm install to sync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

In scenario automation (msbench), the testbed git repo has no remotes, so getGitHubRepoInfoFromContext returns undefined. Fall back to the SWEBENCH_REPO env var set by the Blackbird local setup. Also improve error logging to include the response body on failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resolve merge conflicts: - services.ts: keep unconditional ICAPIClientService registration from main, keep scenario automation overrides for search services - staticGitHubAuthenticationService.ts: merge new StrictAuthenticationPresentationOptions with our ConfigKey/createTestExtendedTokenInfo imports - configurationService.ts: keep both DebugOverrideEmbeddingsUrl and new DebugPromptOverrideFile settings - package.json: keep all three debug settings - scenarioAutomationWorkspaceChunkSearchService.ts: update to match refactored IWorkspaceChunkSearchService interface (removed localIndexState, isFullWorkspace, hasFastSearch; added isAvailable, callSite) - scenarioAutomationCAPIClientImpl.ts: fix FetchOptions type mismatch by importing platform FetchOptions separately from CAPI FetchOptions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tions Now that ScenarioAutomationWorkspaceChunkSearchService directly calls the Blackbird endpoint via IFetcherService, the CAPI client override (which intercepted EmbeddingsCodeSearch requests) is no longer needed. Also remove MockGithubAvailableEmbeddingTypesService and the redundant IGithubCodeSearchService/ICAPIClientService registrations from the scenario automation block — the production versions registered unconditionally are sufficient since our service bypasses the production search pipeline entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mjbvz · 2026-04-01T22:07:51Z

 		 * Note: this should not be used while self-hosting because it might lead to
 		 * a fundamental different experience compared to our end-users.
 		*/
+		export const DebugOverrideChatEngine = defineAndMigrateSetting<string | undefined>('chat.advanced.debug.overrideChatEngine', 'chat.debug.overrideChatEngine', undefined);


These are only used for automation, right? Do we need these settings then or can the ScenarioAutomation chunk search just hardcode values?

Updated to remove this.

mjbvz · 2026-04-01T22:08:38Z

+	 */
+	override get copilotToken(): CopilotToken | undefined {
+		const token = this._tokenStore.copilotToken;
+		if (this._configurationService.getConfig(ConfigKey.Advanced.DebugOverrideEmbeddingsUrl)) {


I feel like this affects much more than embeddings so it's confusing to have DebugOverrideEmbeddingsUrl. Maybe give that setting a better name or see if we can avoid needing it altogether?

Updated to hardcode in chunk search.

- Remove DebugOverrideChatEngine and DebugOverrideEmbeddingsUrl settings from package.json, package.nls.json, and configurationService.ts - Hardcode the Blackbird local endpoint URL in ScenarioAutomationWorkspaceChunkSearchService (scenario-automation-only) - Simplify StaticGitHubAuthenticationService copilotToken override to unconditionally return a placeholder when token is missing/noAuth, removing the confusing DebugOverrideEmbeddingsUrl dependency Tested: msbench run 23901603934 — 100% resolved (3/3), 0 errors, semantic_search returning valid Blackbird code snippets in all tasks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mjbvz · 2026-04-02T18:20:55Z

+	override get copilotToken(): CopilotToken | undefined {
+		const token = this._tokenStore.copilotToken;
+		if (token?.isNoAuthUser) {
+			return new CopilotToken(createTestExtendedTokenInfo());


This will break real no-auth users in production code, right? I'd like to understand why this is needed. I don't think we should do this if it's only for ScenarioAutomationEndpointProviderImpl Is there a better approach that keeps this isolated to just the test code?

This isn't needed since msbench env does auth, I've removed it.

Create a scenario-automation-specific auth service instead of modifying StaticGitHubAuthenticationService, which is shared with tests and could affect real no-auth users. ScenarioAutomationAuthenticationService extends StaticGitHubAuthenticationService and only overrides copilotToken to replace noAuth tokens with a placeholder. This keeps the change isolated to scenario automation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…search_evaluation_override

Merge latest main and update ScenarioAutomationWorkspaceChunkSearchService to implement the renamed triggerIndexing method (was triggerRemoteIndexing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The copilotToken override is not needed — the CAPI proxy provides a real token (SKU free_limited_copilot), so isNoAuthUser is always false and the override never fires. Revert to using StaticGitHubAuthenticationService directly, eliminating all changes to the authentication layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

out of date

Updates to support semantic search with blackbird in MSBench. - ScenarioAutomationWorkspaceChunkSearchService: directly calls Blackbird localhost:4443 embeddings endpoint bypassing production strategies - ScenarioAutomationEndpointProviderImpl: add fallback to copilot-base when model family resolution fails in scenario automation - Move IWorkspaceChunkSearchService registration into isScenarioAutomation if/else branch in services.ts - Move IGithubCodeSearchService and IGithubAvailableEmbeddingTypesService registrations to shared (non-branched) section Moved from microsoft/vscode-copilot-chat#3947 Co-authored-by: gryan11 <415784+gryan11@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rwoll · 2026-04-07T23:20:09Z

Moved to microsoft/vscode#308394 (the copilot chat extension now lives in the vscode repo).

* Add localhost embeddings support for scenario automation Updates to support semantic search with blackbird in MSBench. - ScenarioAutomationWorkspaceChunkSearchService: directly calls Blackbird localhost:4443 embeddings endpoint bypassing production strategies - ScenarioAutomationEndpointProviderImpl: add fallback to copilot-base when model family resolution fails in scenario automation - Move IWorkspaceChunkSearchService registration into isScenarioAutomation if/else branch in services.ts - Move IGithubCodeSearchService and IGithubAvailableEmbeddingTypesService registrations to shared (non-branched) section Moved from microsoft/vscode-copilot-chat#3947 Co-authored-by: gryan11 <415784+gryan11@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix scenario automation workspace chunk search review feedback Agent-Logs-Url: https://github.com/microsoft/vscode/sessions/be3a88b1-d355-40f5-94df-a71c1d2429b6 Co-authored-by: rwoll <11915034+rwoll@users.noreply.github.com> --------- Co-authored-by: gryan11 <415784+gryan11@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rwoll <11915034+rwoll@users.noreply.github.com>

rwoll reviewed Feb 24, 2026

View reviewed changes

gryan11 force-pushed the ryangabriel/semantic_search_evaluation_override branch 4 times, most recently from c7d7d24 to 4a4937e Compare February 25, 2026 22:38

gryan11 added 3 commits February 25, 2026 22:48

Add optional COPILOT_EMBEDDINGS_AUTH_TOKEN env var for embeddings auth

e0dfd4f

When set, adds Authorization: Bearer header to embeddings requests sent to the override URL in ScenarioAutomationCAPIClientImpl.

gryan11 force-pushed the ryangabriel/semantic_search_evaluation_override branch 2 times, most recently from f610ea6 to 4123940 Compare February 25, 2026 23:13

gryan11 force-pushed the ryangabriel/semantic_search_evaluation_override branch from 4123940 to bc6b15b Compare February 26, 2026 14:30

gryan11 and others added 2 commits February 26, 2026 22:12

gryan11 marked this pull request as ready for review February 27, 2026 21:34

Copilot AI review requested due to automatic review settings February 27, 2026 21:34

Copilot started reviewing on behalf of gryan11 February 27, 2026 21:35 View session

vs-code-engineering Bot assigned dbaeumer Feb 27, 2026

vs-code-engineering Bot added the triage-needed label Feb 27, 2026

Copilot AI reviewed Feb 27, 2026

View reviewed changes

mjbvz reviewed Mar 2, 2026

View reviewed changes

dbaeumer assigned rwoll and unassigned dbaeumer Mar 3, 2026

dbaeumer removed the triage-needed label Mar 3, 2026

gryan11 and others added 5 commits March 3, 2026 15:59

gryan11 force-pushed the ryangabriel/semantic_search_evaluation_override branch from 64679f7 to fe4add1 Compare March 31, 2026 17:58

gryan11 and others added 2 commits March 31, 2026 20:02

gryan11 force-pushed the ryangabriel/semantic_search_evaluation_override branch from 2154b4a to 30048b0 Compare April 1, 2026 13:24

mjbvz previously requested changes Apr 1, 2026

View reviewed changes

gryan11 force-pushed the ryangabriel/semantic_search_evaluation_override branch from 4edb85e to b26b3b0 Compare April 2, 2026 13:47

mjbvz reviewed Apr 2, 2026

View reviewed changes

gryan11 and others added 4 commits April 7, 2026 13:10

Merge remote-tracking branch 'origin/main' into ryangabriel/semantic_…

03c57c3

…search_evaluation_override

Merge origin/main, fix triggerIndexing interface change

035f246

Merge latest main and update ScenarioAutomationWorkspaceChunkSearchService to implement the renamed triggerIndexing method (was triggerRemoteIndexing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

gryan11 mentioned this pull request Apr 7, 2026

Add configurable semantic search endpoint for local blackbird testing #3493

Closed

rwoll reviewed Apr 7, 2026

View reviewed changes

Comment thread package-lock.json

Revert package-lock.json to match main

214a5de

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rwoll approved these changes Apr 7, 2026

View reviewed changes

rwoll requested a review from mjbvz April 7, 2026 17:29

rwoll enabled auto-merge April 7, 2026 18:03

rwoll mentioned this pull request Apr 7, 2026

feat(testing): support alternative semantic index endpoints microsoft/vscode#308394

Merged

rwoll closed this Apr 7, 2026

auto-merge was automatically disabled April 7, 2026 23:20
Pull request was closed

-	}
+	}
+	/**
+	 * Scenario automation runs may not have a real GitHub PAT/OAuth token configured.
+	 * Override the base token acquisition to return a placeholder token so that
+	 * GithubCodeSearchService.searchRepo() can proceed without throwing
+	 * "No valid auth token". The ScenarioAutomationCAPIClientImpl is responsible
+	 * for stripping or ignoring auth headers derived from this value.
+	 */
+	protected override async getGithubAccessToken(_auth: { readonly silent: boolean }): Promise<string> {
+		this._log.trace('ScenarioAutomationGithubCodeSearchService::getGithubAccessToken() using placeholder token for scenario automation.');
+		return 'scenario-automation-placeholder-token';
+	}

Conversation

gryan11 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rwoll commented Feb 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gryan11 Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rwoll commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gryan11 commented Feb 23, 2026 •

edited

Loading

gryan11 Apr 2, 2026 •

edited

Loading