[UITests] New framework around WinApp CLI, no WinAppDriver or Selenium.#48467
Open
khmyznikov wants to merge 11 commits into
Open
[UITests] New framework around WinApp CLI, no WinAppDriver or Selenium.#48467khmyznikov wants to merge 11 commits into
khmyznikov wants to merge 11 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| /// Full end-to-end Color Picker scenario, driven entirely through the Settings UI: | ||
| /// 1. From the Settings app, navigate to the Color Picker page via the utilities stack. | ||
| /// 2. On the page, toggle the module OFF and verify <c>PowerToys.ColorPickerUI</c> exits. | ||
| /// 3. Toggle it back ON and verify <c>PowerToys.ColorPickerUI</c> respawns. |
| "PowerToys.ColorPickerUI did not exit within 10s after toggling module OFF."); | ||
| TestContext.WriteLine("Toggled OFF; ColorPickerUI process exited."); | ||
|
|
||
| // -- 4. Toggle the module ON and verify the runner respawns ColorPickerUI ------- |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new, opt-in UI test harness (Microsoft.PowerToys.UITest.Next) that drives PowerToys via winappcli (shelling out to winapp.exe) rather than WinAppDriver/Selenium, and adds two initial consumer test projects (Color Picker E2E + Settings navigation smoke). It also adds a small product-side UIA hook in Color Picker to make the live HEX value readable via UI Automation.
Changes:
- Add
src/common/UITestAutomation.Next/winappcli-based harness (Session/Element wrappers, window discovery, Win32 input + clipboard helpers). - Add new UI test projects for Color Picker and Settings using the new harness; remove the legacy empty ColorPicker UI test stub.
- Add a hidden
TextBlockin ColorPicker overlay to expose HEX via a stable AutomationId; update solution + spell-check allowlist.
Reviewed changes
Copilot reviewed 28 out of 30 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| src/settings-ui/UITest-Settings.Next/UITest-Settings.Next.csproj | New Settings UI smoke-test project (MSTest + Testing Platform app). |
| src/settings-ui/UITest-Settings.Next/SettingsNavigationSmokeTests.cs | Parameterized smoke test that clicks all Settings nav items and asserts process stays alive. |
| src/modules/colorPicker/UITest-ColorPicker/UITest-ColorPicker.csproj | Removes legacy WinAppDriver/Appium ColorPicker UI test stub project. |
| src/modules/colorPicker/UITest-ColorPicker/ColorPickerUITest.cs | Removes legacy stub test class. |
| src/modules/colorPicker/ColorPickerUI/Views/MainView.xaml | Adds hidden UIA hook (ColorHexAutomationPeer) bound to ColorText. |
| src/modules/colorPicker/ColorPicker.UITests/ColorPickerUITest.md | Adds Color Picker UI test checklist documentation. |
| src/modules/colorPicker/ColorPicker.UITests/ColorPickerEndToEndTests.cs | Adds full Color Picker E2E scenario driven via Settings + hotkey + overlay/editor assertions. |
| src/modules/colorPicker/ColorPicker.UITests/ColorPicker.UITests.csproj | New ColorPicker UI test project using the Next harness. |
| src/modules/colorPicker/ColorPicker.UITests/AssemblyInfo.cs | Disables MSTest parallelization for ColorPicker UI tests. |
| src/common/UITestAutomation.Next/Windows.cs | Window discovery/wait helpers wrapping winapp ui list-windows. |
| src/common/UITestAutomation.Next/WindowControl.cs | Best-effort Win32 close/focus/kill helpers for cleanup. |
| src/common/UITestAutomation.Next/WinappCli.cs | Process wrapper around winapp.exe with stdout/stderr capture + JSON helpers. |
| src/common/UITestAutomation.Next/UITestBase.cs | MSTest base class that preflights winappcli and initializes a session per test. |
| src/common/UITestAutomation.Next/UITestAutomation.Next.csproj | New shared harness project definition. |
| src/common/UITestAutomation.Next/SessionHelper.cs | Launch + readiness flow for module processes/windows (shell execute, UIA window wait). |
| src/common/UITestAutomation.Next/Session.cs | Session abstraction supporting HWND (-w) and process (-a) scoping + search/inspect/screenshot. |
| src/common/UITestAutomation.Next/MouseHelper.cs | Win32 mouse input helper (SetCursorPos/mouse_event). |
| src/common/UITestAutomation.Next/ModuleConfigData.cs | PowerToysModule enum and installed-path process/title metadata mapping. |
| src/common/UITestAutomation.Next/KeyboardHelper.cs | Hybrid Win32 + WinForms SendKeys chord sender for global hotkeys. |
| src/common/UITestAutomation.Next/Element/Window.cs | Window element wrapper. |
| src/common/UITestAutomation.Next/Element/ToggleSwitch.cs | ToggleSwitch wrapper (ClassName filter + ToggleState helpers). |
| src/common/UITestAutomation.Next/Element/TextBox.cs | TextBox wrapper (set-value/get-value). |
| src/common/UITestAutomation.Next/Element/NavigationViewItem.cs | NavigationViewItem wrapper. |
| src/common/UITestAutomation.Next/Element/Element.cs | Core element wrapper (invoke/click/focus/property/value/wait helpers). |
| src/common/UITestAutomation.Next/Element/Button.cs | Button wrapper. |
| src/common/UITestAutomation.Next/ClipboardHelper.cs | STA-thread clipboard helper + polling for expected clipboard updates. |
| src/common/UITestAutomation.Next/By.cs | Selector abstraction (By.Name, By.AccessibilityId, By.Slug). |
| PowerToys.slnx | Registers new harness + new UI test projects in solution. |
| .github/actions/spell-check/expect.txt | Adds winapp / winappcli to spell-check allowlist. |
| injection in KeyboardHelper. (Same approach as the legacy harness.) | ||
| --> | ||
| <UseWindowsForms>true</UseWindowsForms> | ||
| <TreatWarningsAsErrors>false</TreatWarningsAsErrors> |
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsPackable>false</IsPackable> | ||
| <TreatWarningsAsErrors>false</TreatWarningsAsErrors> |
Comment on lines
+18
to
+23
| public TextBox SetText(string value) | ||
| { | ||
| Assert.IsNotNull(Owner, "TextBox is not bound to a Session."); | ||
| WinappCli.InvokeAssertSuccess("ui", "set-value", Selector, value, "-w", Owner!.WindowHandleArg); | ||
| return this; | ||
| } |
Comment on lines
+30
to
+36
| Assert.IsNotNull(Owner, "TextBox is not bound to a Session."); | ||
| var r = WinappCli.Invoke("ui", "get-value", Selector, "-w", Owner!.WindowHandleArg, "--json"); | ||
| if (!r.Success) | ||
| { | ||
| return string.Empty; | ||
| } | ||
|
|
Comment on lines
+191
to
+201
| /// <summary>Find a descendant matching <paramref name="by"/>, scoped under this element via its slug.</summary> | ||
| public T Find<T>(By by, int timeoutMS = 5000) | ||
| where T : Element, new() | ||
| { | ||
| EnsureBound(); | ||
|
|
||
| // winappcli scopes a search beneath an element by passing the parent's selector to inspect. | ||
| // For most cases (within the same window) the global search is fine and faster; if you need | ||
| // strict scoping under a subtree, use a slug By that prefixes with the parent's slug. | ||
| return Owner!.FindUnder<T>(by, timeoutMS); | ||
| } |
|
|
||
| while (DateTime.UtcNow < deadline) | ||
| { | ||
| var r = WinappCli.Invoke("ui", "list-windows", "--json"); |
Comment on lines
+124
to
+136
| using var p = Process.Start(psi) ?? throw new InvalidOperationException( | ||
| $"Failed to start winapp.exe ({ExecutablePath.Value}). {InstallHint}"); | ||
|
|
||
| var stdoutTask = p.StandardOutput.ReadToEndAsync(); | ||
| var stderrTask = p.StandardError.ReadToEndAsync(); | ||
| p.WaitForExit(); | ||
|
|
||
| return new Result( | ||
| p.ExitCode, | ||
| stdoutTask.GetAwaiter().GetResult(), | ||
| stderrTask.GetAwaiter().GetResult(), | ||
| args); | ||
| } |
Comment on lines
+36
to
+41
| <TextBlock | ||
| x:Name="ColorHexAutomationPeer" | ||
| AutomationProperties.AutomationId="ColorHexAutomationPeer" | ||
| IsHitTestVisible="False" | ||
| Opacity="0" | ||
| Text="{Binding ColorText}" /> |
Comment on lines
+20
to
+30
| private static readonly string Root = Path.Combine( | ||
| Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles), | ||
| "PowerToys"); | ||
|
|
||
| public static string ExePathFor(PowerToysModule module) => module switch | ||
| { | ||
| PowerToysModule.PowerToysSettings => Path.Combine(Root, "WinUI3Apps", "PowerToys.Settings.exe"), | ||
| PowerToysModule.Runner => Path.Combine(Root, "PowerToys.exe"), | ||
| PowerToysModule.ColorPicker => Path.Combine(Root, "PowerToys.ColorPickerUI.exe"), | ||
| _ => throw new ArgumentOutOfRangeException(nameof(module), module, null), | ||
| }; |
Comment on lines
+94
to
+100
| case Key.Backspace: chord.Append("{BACKSPACE}"); break; | ||
| case Key.Delete: chord.Append("{DELETE}"); break; | ||
| default: | ||
| // Letter / digit keys map to their lowercase character for SendKeys. | ||
| chord.Append(((char)k).ToString().ToLowerInvariant()); | ||
| break; | ||
| } |
This comment has been minimized.
This comment has been minimized.
…ensions Add ComboBox, CheckBox, RadioButton, Slider, TextBlock wrappers (all driven via winapp ui invoke/get-property/get-value/set-value). Element: add DoubleClick (click --double), ScrollIntoView (scroll-into-view), live properties IsEnabled/IsOffscreen/Displayed/Selected/AutomationId + GetAttribute (get-property), WaitForValue with --contains. Make GetProperty tolerant of non-string/error output and expose EnsureBound to subclasses. Session: add WaitForElement (wait-for appear). Fix: TextBox set-value/get-value hardcoded -w <hwnd>, which targeted window 0 under process-scoped (-a) sessions; now uses the session's TargetFlag/TargetValue.
…stics ModulePaths: expand PowerToysModule enum to 10 modules; resolve exe via POWERTOYS_INSTALL_DIR override -> installed build -> repo dev-build output (x64/ARM64, Debug/Release). useInstallerForTest forces installed layout. Lets tests run against either an installed PowerToys or a local dev build. EnvironmentConfig: IsInPipeline / UseInstallerForTest / Platform (ported from legacy harness). SettingsConfigHelper: dependency-free (System.Text.Json.Nodes) ConfigureGlobalModuleSettings + UpdateModuleSettings, writing the per-user settings JSON directly (no Settings.UI.Library coupling). Session diagnostics (CLI-first): Screenshot element-crop + --capture-screen + non-asserting TryScreenshot; Inspect --interactive/--hide-disabled/--hide-offscreen; InspectAncestors; GetFocused/GetFocusedName. UITestBase: capture a --capture-screen PNG and attach it on test failure. SessionHelper: RestartScope (kill -> wait exit -> relaunch + wait window).
…dow helpers) MouseHelper: add GetMousePosition, LeftDown/Up, RightDown/Up, MiddleDown/Up/Click, DoubleClick, ScrollWheel/Up/Down, and a stepped Drag(from,to). winappcli has no drag/wheel/raw-cursor verbs, so these stay Win32. KeyboardHelper: extend Key enum with digits, F1-F12, arrows, Home/End/PageUp/PageDown/Insert; add PressKey/ReleaseKey/SendKey/SendKeySequence with extended-key handling for nav keys. Element: add CLI-first Scroll(direction)/ScrollToEdge (winapp ui scroll) plus Win32 Drag/DragTo/KeyDownAndDrag using the element's search-reported center. Elements: add Pane/Thumb/Custom/Tab wrappers (drag inherited from Element). WindowHelper (new): WindowSize enum + SetWindowSize/SetMainWindowSize, GetWindowBounds/Center, GetDisplaySize/GetScreenCenter, GetPixelColor/GetPixelColorHex (GDI) — lets ColorPicker-style tests read on-screen pixels without a hidden XAML peer. IsWindowOpen stays CLI-based via WindowsFinder. Session: add Attach(module, size) — window-scoped session with optional preset resize.
ElevationHelper (new): IsCurrentProcessElevated / IsProcessElevated via OpenProcessToken + TokenElevation. Session.IsElevated surfaces the target process's elevation (null when no PID). MonitorInfo (new): GetAll / GetPrimary / Count via EnumDisplayMonitors + GetMonitorInfo, returning per-display bounds, work area, and primary flag for multi-monitor utility tests. Session.Status(): winapp ui status --json for connection diagnostics. Intentionally deferred (heavy / external-dep, and existing primitives already cover the common cases): perceptual-hash VisualAssert (GetPixelColor + Screenshot cover basic visual checks) and FFmpeg ScreenRecording (failure --capture-screen screenshots cover diagnostics).
| /// <summary>Center point of the window in screen pixels.</summary> | ||
| public static (int CenterX, int CenterY) GetWindowCenter(IntPtr hWnd) | ||
| { | ||
| var (l, t, rgt, b) = GetWindowBounds(hWnd); |
| public static (int CenterX, int CenterY) GetWindowCenter(IntPtr hWnd) | ||
| { | ||
| var (l, t, rgt, b) = GetWindowBounds(hWnd); | ||
| return (l + ((rgt - l) / 2), t + ((b - t) / 2)); |
|
|
||
| while (DateTime.UtcNow < deadline) | ||
| { | ||
| var r = WinappCli.Invoke("ui", "list-windows", "--json"); |
Comment on lines
+22
to
+23
| /// Settings is launched directly (not via <c>PowerToys.exe</c>) so this test exercises just | ||
| /// the shell navigation path and doesn't depend on the runner's tray/elevation/module startup. |
Comment on lines
+5
to
+9
| using System.Reflection; | ||
| using Microsoft.PowerToys.UITest.Next; | ||
| using Microsoft.VisualStudio.TestTools.UnitTesting; | ||
|
|
||
| namespace Microsoft.Settings.UITests; |
Comment on lines
+5
to
+10
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsPackable>false</IsPackable> | ||
| <TreatWarningsAsErrors>false</TreatWarningsAsErrors> | ||
| <RootNamespace>Microsoft.Settings.UITests</RootNamespace> | ||
| <AssemblyName>Settings.UITests</AssemblyName> |
Comment on lines
+5
to
+10
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsPackable>false</IsPackable> | ||
| <TreatWarningsAsErrors>false</TreatWarningsAsErrors> | ||
| <RootNamespace>Microsoft.ColorPicker.UITests</RootNamespace> | ||
| <AssemblyName>ColorPicker.UITests</AssemblyName> |
Comment on lines
+11
to
+14
| <UseWindowsForms>true</UseWindowsForms> | ||
| <TreatWarningsAsErrors>false</TreatWarningsAsErrors> | ||
| <RootNamespace>Microsoft.PowerToys.UITest.Next</RootNamespace> | ||
| <AssemblyName>Microsoft.PowerToys.UITest.Next</AssemblyName> |
@check-spelling-bot Report🔴 Please reviewSee the 📂 files view, the 📜action log, 👼 SARIF report, or 📝 job summary for details.Unrecognized words (2)respawns These words are not needed and should be removedDWRITE LWIN nonstd VCENTER VREDRAWTo accept these unrecognized words as correct and remove the previously acknowledged and now absent words, you could run the following commands... in a clone of the git@github.com:microsoft/PowerToys.git repository curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/cfb6f7e75bbfc89c71eaa30366d0c166f1bd9c8c/apply.pl' |
perl - 'https://github.com/microsoft/PowerToys/actions/runs/27376175101/attempts/1' &&
git commit -m 'Update check-spelling metadata'OR To have the bot accept them for you, comment in the PR quoting the following line: Warnings
|
| Count | |
|---|---|
| 2 |
See
If the flagged items are 🤯 false positives
If items relate to a ...
-
binary file (or some other file you wouldn't want to check at all).
Please add a file path to the
excludes.txtfile matching the containing file.File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.
^refers to the file's path from the root of the repository, so^README\.md$would exclude README.md (on whichever branch you're using). -
well-formed pattern.
If you can write a pattern that would match it,
try adding it to thepatterns.txtfile.Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.
Note that patterns can't match multiline strings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add winappcli-based UI test harness (no WinAppDriver / Selenium)
Summary
Introduces a new UI test harness —
Microsoft.PowerToys.UITest.Next— that drives PowerToysmodules through Microsoft's winappcli (UI Automation
CLI) instead of WinAppDriver + Selenium. Engine is a single executable shelled out from
C#; no third-party NuGet packages, no driver process, no Appium server. Adds two real consumers:
a full ColorPicker end-to-end scenario and a Settings shell navigation smoke test.
This is opt-in and additive — the existing
UITestAutomationlibrary and theWinAppDriver-based test projects are untouched. Both can coexist while we evaluate the new
harness.
Inspired in part by #48414, which lands
the same architectural bet (winappcli, AutomationId selectors, no WinAppDriver) at a smaller
scope. This PR generalizes it into a reusable library.
Why
WinAppDriver + Selenium is a legacy pre-agentic solution that is no longer actively maintained. It's unreliable, heavyweight, and slow. To achieve 100% UI test coverage, we should leverage modern, reliable solutions, and WinApp CLI is a strong candidate.
What's in this PR
Harness library —
src/common/UITestAutomation.Next/WinappCli.cswinapp.exe.Invoke/InvokeAssertSuccess/InvokeJson/IsAvailable/TryResolveExecutable.Resultcarries the args and emitsDescribeFailure()likewinapp ui invoke X -w 12345 -> exit 1; stderr: ...Session.cs-w) or process (-a) viaTargetScope.Find<T>/FindAll<T>/Inspect/Screenshot/SendKeys.Session.FromProcess(...)factory for the single-window-per-process caseSessionHelper.csEnsureRunning(scope, timeout)returns whether the call had to launch (so cleanup only kills what we started). UsesUseShellExecute=trueso child handles don't keep MSTest hangingUITestBase.csWinappCli.IsAvailable()once per process and fails fast with the install hint ifwinapp.exeisn't on PATHElement/*.csElement,Button,ToggleSwitch,TextBox,NavigationViewItem,Window.Click/MouseClick/Focus/GetProperty/GetValue/HelpText/WaitForProperty/WaitForGoneplus coords (X/Y/Width/Height)By.csBy.Name/By.AccessibilityId/By.Id/By.SlugWindows.csWindowsFinder.ListAll/ListByApp/WaitForWindowByApp/WaitForWindowByProcess. Notes the winappcli bug where unfilteredlist-windowsdrops untitled windowsWindowControl.csTryCloseByApp/TryFocusByApp/SafeCloseAndFocus/TryKillProcess— forfinallyblocksKeyboardHelper.cskeybd_event+SendKeys.SendWaitchord sender — required for global PowerToys hotkeysMouseHelper.csMoveTo/LeftClick/RightClick/LeftClickAtWin32 wrappersClipboardHelper.csClipboardaccess withWaitForTextModuleConfigData.csPowerToysModuleenum + path/process-name resolutionTests
src/modules/colorPicker/ColorPicker.UITests/— replaces the previous emptyUITest-ColorPickerstub. One test,ColorPickerEndToEndTests.NavigateReadShortcutActivateAndCapture, drives the full E2E:PowerToys.ColorPickerUIexits; toggle ON, verify it respawnsShortcutControl(EditButton.HelpText)src/settings-ui/Settings.UITests/—SettingsNavigationSmokeTests.NavigationItem_NavigatesWithoutCrashingis one[TestMethod]parameterized with[DynamicData], producing 31 discrete results — one perNavigationViewIteminShellPage.xaml. For each item: navigate, settle 250ms, assertPowerToys.Settingsis still alive. Catches FailFast regressions inShellViewModel.Frame_NavigationFailedthat pure-logic unit tests can't reach (the failure path needs aNavigationFailedEventArgswhich is a sealed WinRT projection).Product change
src/modules/colorPicker/ColorPickerUI/Views/MainView.xaml— adds a hiddenTextBlockautomation peer:The visible
ColorTextBlockhasAutomationProperties.Name="{Binding ColorName}", which masks the HEX value in the UIA tree (you see "White" instead of#FFFFFF). This zero-impact peer mirrorsColorTextso tests can read the actually-displayed HEX.Opacity=0+IsHitTestVisible=Falsekeep it out of the visual layout and out of accessibility focus.Project wiring
PowerToys.slnx— registersUITestAutomation.Nextunder/common/,ColorPicker.UITestsunder/modules/colorpicker/Tests/, andSettings.UITestsunder/settings-ui/Tests/. OriginalUITest-ColorPickerstub csproj removed..github/actions/spell-check/expect.txt— addswinapp/winappcli.Not in this PR
winapp.exeis expected to be pre-staged on the test agent image. If it's missing,UITestBasefails the first test with the install hint (winget install Microsoft.winappcli) rather than producing 30 opaque per-test errors.UITestAutomationlibrary or any of the existing*.UITestsprojects.Validation
x64|Debug(emptybuild.<config>.<plat>.errors.log):src/common/UITestAutomation.Next/src/modules/colorPicker/ColorPicker.UITests/src/settings-ui/Settings.UITests/dotnet testvia Microsoft.Testing.Platform (already enabled repo-wide inDirectory.Build.props).winapp 0.3.2fromwinget install Microsoft.winappcli.Notes for reviewers
UseShellExecute = trueinSessionHelper.EnsureRunningis intentional —falsemakes child processes inherit the test host's stdin/stdout/stderr handles, which keeps MTP/Test Explorer marking the run as "in progress" until the spawned PowerToys exits.-a) targeting in the Settings smoke test handles single-instance handoff: the EXE you launch may exit with code 0 immediately after signalling an existing owner, so the alive check usesProcess.GetProcessesByNamerather than the launcher PID.SelectsOnInvoked="False"and only expand on click —Element.ClicktriesInvokePattern → TogglePattern → SelectionItemPattern → ExpandCollapsePatternso the same call works for both leaves and groups.winapp ui list-windows -a <name>returns windows that the unfiltered call drops (e.g. ColorPicker editor).WindowsFinder.ListByAppuses the filtered form. Reported upstream.Before Merge
winappcliinstall step to the UI-test pipeline.