Scenario: Improved keyboard handling

[zadjii-msft]

The Terminal's keyboard input is fundamentally backed by VT sequences, which limits the range of keys that we can actually send relative to what the console was capable of. This megathread is tracking all the issues that might be related to this:

• Some keybindings used by PSReadLine aren't getting through Some keybindings used by PSReadLine aren't getting through #879
• Bug Report: Control+Space not sent to terminal emulator. Bug Report: Control+Space not sent to terminal emulator. #2865
• Shift+Enter always submits, breaking PSReadline features [Windows.Terminal] Shift+Enter always submits, breaking PSReadline features [Windows.Terminal] #530
• Powershell: Ctrl-Alt-? does not work in Windows Terminal Powershell: Ctrl-Alt-? does not work in Windows Terminal #3079
• Bug: ctrl+break is not ctrl+c Bug: ctrl+break is not ctrl+c #1119
• Something wrong with keyboard modifiers processing? Something wrong with keyboard modifiers processing? #1694
• Numeric input not accepted by choice.exe Numeric input not accepted by choice.exe #3608
• Ctrl+Keys that can't be encoded as VT should still fall through as the unmodified character Ctrl+Keys that can't be encoded as VT should still fall through as the unmodified character #3483
• Modifier keys are not properly propagated to application hosted in Windows Terminal Modifier keys are not properly propagated to application hosted in Windows Terminal #4334 / Holding a command key (Ctrl,Shift,Alt) does not pass through to FAR Manager #4446
• We may be able to fix the Ctrl+Backspace/Ctrl+H mixup that resulted in Incorrect keycode for "CTRL+H" in Windows Terminal #5957.

Before 2.0, we're hoping on adding support for parsing a much broader variety of keys via conpty.

Keys that we definitely need to support, that don't have unique VT sequences:

• Ctrl+Space (Some keybindings used by PSReadLine aren't getting through #879, Bug Report: Control+Space not sent to terminal emulator. #2865)
• Shift+Enter (Shift+Enter always submits, breaking PSReadline features [Windows.Terminal] #530)
• Ctrl+Break (Bug: ctrl+break is not ctrl+c #1119)
• Ctrl+Alt+? (Powershell: Ctrl-Alt-? does not work in Windows Terminal #3079)
• Ctrl, Alt, Shift, (without another keydown/up) (Numeric input not accepted by choice.exe #3608, Modifier keys are not properly propagated to application hosted in Windows Terminal #4334, Holding a command key (Ctrl,Shift,Alt) does not pass through to FAR Manager #4446)

[zadjii-msft]

This is a work-in-progress

Potential Solutions:

Create our own format for INPUT_RECORDs

• If we wanted to do this, then we'd probably want to have the Terminal only send input as this format, and not use the existing translator to synthesize VT sequences
  • Consider sending a ctrl down, '^A', ctrl up. We wouldn't want to send this as three sequences, because conpty will take the '^A' and synthesize another ctrl down, ctrl up pair.
• With conpty passthrough mode, we'd still need the InpustStateMachineEngine to convert these sequences into INPUT_RECORDs to translate back to VT
• Wouldn't really expect client apps to ever need this format, but it could always be possible for them to need it in the future.

Pros:

• Definitely gets us all the information that we need.
• Can handle solo modifiers
• Can handle keydown and keyup separately
• We can make the sequence however we want to parse it.

Cons:

• No reference implementation, so we'd be flying blind
• We'd be defining our own VT sequences for these, which we've never done before. This was inevitable, however, this is still the first time we'd be doing this.
• Something about Microsoft extending well-defined protocols being bad 😆
• By having the Terminal send all input as this protocol, VT Input passthrough to apps that want VT input won't work anymore for the Terminal. That's okay

Scenarios

User is typing into WSL from WT

WT -> conpty[1] -> wsl

• conpty 1 will ask for win32-input-mode from WT
• When the user types keys in WT, WT will translate them into win32 sequences and send them to conpty[1]
• conpty[1] will translate those win32 sequences into INPUT_RECORDs
• when WSL reads the input, conpty 1 will translate the INPUT_RECORDs into VT sequences corresponding to whatever input mode the linux app is in

User is typing into cmd.exe running in WSL interop

WT -> conpty[1] -> wsl -> conpty[2] -> cmd.exe

(presuming you start from the previous scenario, and launch cmd.exe inside wsl)

• conpty 2 will ask for win32-input-mode from conpty 1
• When the user types keys in WT, WT will translate them into win32 sequences and send them to conpty[1]
• conpty[1] will translate those win32 sequences into INPUT_RECORDs
• when WSL reads the input, conpty 1 will translate the INPUT_RECORDs into VT sequences for the win32 input that conpty 2 requested
• conpty 2 will get those sequences, and will translate those win32 sequences into INPUT_RECORDs
• when cmd.exe reads the input, they'll receive the full INPUT_RECORDs they're expecting

kitty extension

Reference

Pros:

• Not terribly difficult to decode
• Unique from anything else we'd be processing, as it's an APC sequence (\x1b_)
• From their docs:

  All printable key presses without modifier keys are sent just as in the normal mode. ... For non printable keys and key combinations including one or more modifiers, an escape sequence encoding the key event is sent

  • I think like this. ASCII and other keyboard layout chars (things that would hit SendChar) would still just come through as the normal char.

Cons:

• Their encoding table is mad. Look at this. What order is that in? Obviously the first column is sorted alphabetically, but the mapping of key->char is in a seemingly bonkers order.
• I can't get it working locally, so hard to test 😐
• They do declare the fullkbd terminfo capability to identify that they support this mode, but I'm not sure anyone else uses it.
  • I'm also not sure that any client apps are reading this currently.
• This isn't designed to be full KEY_EVENTs - where would we put the scancode (for apps that think that's important)?
  • We'd have to extend this protocol anyways

xterm "Set key modifier options"

Notably looking at modifyOtherKeys.

Pros:

• xterm implements this so there's a reference implementation
• relatively easy to parse these sequences. CSI 27 ; <modifiers> ; <key> ~

Cons:

• Only sends the sequence on key-up
• Doesn't send modifiers all on their own

DECPCTERM

VT100.net doc

Pros:

• Enables us to send key-down and key-up keys independently
• Enables us to send modifiers on their own
• Part of the VT 510 standard

Cons:

• neither xterm nor gnome-terminal (VTE) seem to implement this. I'm not sure if anyone's got a reference implementation for us to work with.
• Unsure how this would work with other keyboard layouts
  • this doc seems to list the key-down/up codes for all the en-us keyboard keys, but the scancodes for these are different for up and down. That would seem to imply we couldn't just shove the Win32 scancode in those bits

DECPKM, DECSKMR

DECPKM
DECSKMR
DECEKBD

Pros:

• Enables us to send key-down and key-up keys independently
• Enables us to send modifiers on their own
• Part of the VT 510 standard

Cons:

• neither xterm nor gnome-terminal (VTE) seem to implement this. I'm not sure if anyone's got a reference implementation for us to work with.
• not sure that "a three-character ISO key position name, for example C01" is super compatible with our Win32 VKEYs.

Things to keep in mind

• Would this encoding be mixable with other normal VT processing easily?
  • How would the Terminal know when it should send a <chosen_encoding> key vs a normally encoded one?
    • For ex, Ctrl+space - should we send NUL or <chosen_encoding's version of ctrl+space>
• If there's a scenario where Windows Terminal might not be connected to a conpty, then how does conpty enable <chosen_encoding>?
• Is the goal "Full INPUT_RECORD fidelity" or "Make the above scenarios work"?
  • One could imagine having the Terminal special-case the above keys, and send the xterm modifyOtherKeys sequences just for those scenarios.
    • This would not work for shift all by itself.
  • In my opinion, "just making the above work" is a subset of "full INPUT_RECORD", and inevitably we're going to want "full INPUT_RECORD"

Current Conclusions

This is not a formal conclusion, just where I'm currently leaning. Expect this to change.

April 15, 2020: I'm leaning towards synthesizing our own format for INPUT_RECORDs currently.

• The kitty format isn't bad, but it's not good, and it's not widespread
• The DEC* formats don't have any reference implementations, so it's not like any other terminal emulators are using them.
• xterm's modifyOtherKeys doesn't actually get us solo modifier up/downs

The goal we're trying to achieve is communicating INPUT_RECORDs from the terminal to the client app via conpty. This isn't supposed to be a *nix terminal compatible communication, it's supposed to be fundamentally Win32-like.

References

• Why Is It so Hard to Detect Keyup Event on Linux?
  • and the HackerNews discussion

[j4james]

Another option to consider is Key Position Mode (DECKPM) which reports the state of the modifier keys along with each key press (see DECEKBD for the actual report format). I don't think it has separate keyup and keydown events, but it does have an option to report modifier keypresses separately (see DECSMKR).

How would the Terminal know when it should send a <chosen_encoding> key vs a normally encoded one?

I assumed this is something the conpty host would initiate on startup, requesting whichever mode (or modes) we decided to support. If the terminal at the other end supported the requested mode, then it would start reporting keypresses in that format.

In some cases, if the key reporting sequences can mean different things in different modes, the host might need to know whether the client had actually accepted the request to use a particular mode. In the case of DECKPM you could determine that by querying the mode state with DECRQM. In the case of DECPCTERM, you could determine whether the client supported the option from the DA1 report.