Add support for VT52 emulation (#4789)

## Summary of the Pull Request

This PR adds support for the core VT52 commands, and implements the `DECANM` private mode sequence, which switches the terminal between ANSI mode and VT52-compatible mode.

## References

PR #2017 defined the initial specification for VT52 support.
PR #4044 removed the original VT52 cursor ops that conflicted with VT100 sequences.

## PR Checklist
* [x] Closes #976
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #2017

## Detailed Description of the Pull Request / Additional comments

Most of the work involves updates to the parsing state machine, which behaves differently in VT52 mode. `CSI`, `OSC`, and `SS3` sequences are not applicable, and there is one special-case escape sequence (_Direct Cursor Address_), which requires an additional state to handle parameters that come _after_ the final character.

Once the parsing is handled though, it's mostly just a matter of dispatching the commands to existing methods in the `ITermDispatch` interface. Only one new method was required in the interface to handle the _Identify_ command.

The only real new functionality is in the `TerminalInput` class, which needs to generate different escape sequences for certain keys in VT52 mode. This does not yet support _all_ of the VT52 key sequences, because the VT100 support is itself not yet complete. But the basics are in place, and I think the rest is best left for a follow-up issue, and potentially a refactor of the `TerminalInput` class.

I should point out that the original spec called for a new _Graphic Mode_ character set, but I've since discovered that the VT terminals that _emulate_ VT52 just use the existing VT100 _Special Graphics_ set, so that is really what we should be doing too. We can always consider adding the VT52 graphic set as a option later, if there is demand for strict VT52 compatibility. 

## Validation Steps Performed

I've added state machine and adapter tests to confirm that the `DECANM` mode changing sequences are correctly dispatched and forwarded to the `ConGetSet` handler. I've also added state machine tests that confirm the VT52 escape sequences are dispatched correctly when the ANSI mode is reset.

For fuzzing support, I've extended the VT command fuzzer to generate the different kinds of VT52 sequences, as well as mode change sequences to switch between the ANSI and VT52 modes.

In terms of manual testing, I've confirmed that the _Test of VT52 mode_ in Vttest now works as expected.

Author: j4james

src/host/outputStream.cpp

@@ -239,6 +239,23 @@ bool ConhostInternalGetSet::PrivateSetKeypadMode(const bool fApplicationMode)
     return NT_SUCCESS(DoSrvPrivateSetKeypadMode(fApplicationMode));
 }
 
+// Routine Description:
+// - Sets the terminal emulation mode to either ANSI-compatible or VT52.
+//   PrivateSetAnsiMode is an internal-only "API" call that the vt commands can execute,
+//     but it is not represented as a function call on out public API surface.
+// Arguments:
+// - ansiMode - set to true to enable the ANSI mode, false for VT52 mode.
+// Return Value:
+// - true if successful. false otherwise.
+bool ConhostInternalGetSet::PrivateSetAnsiMode(const bool ansiMode)
+{
+    auto& stateMachine = _io.GetActiveOutputBuffer().GetStateMachine();
+    stateMachine.SetAnsiMode(ansiMode);
+    auto& terminalInput = _io.GetActiveInputBuffer()->GetTerminalInput();
+    terminalInput.ChangeAnsiMode(ansiMode);
+    return true;
+}
+
 // Routine Description:
 // - Connects the PrivateSetScreenMode call directly into our Driver Message servicing call inside Conhost.exe
 //   PrivateSetScreenMode is an internal-only "API" call that the vt commands can execute,

src/host/outputStream.hpp

@@ -74,6 +74,7 @@ class ConhostInternalGetSet final : public Microsoft::Console::VirtualTerminal::
     bool PrivateSetCursorKeysMode(const bool applicationMode) override;
     bool PrivateSetKeypadMode(const bool applicationMode) override;
 
+    bool PrivateSetAnsiMode(const bool ansiMode) override;
     bool PrivateSetScreenMode(const bool reverseMode) override;
     bool PrivateSetAutoWrapMode(const bool wrapAtEOL) override;
 

src/terminal/adapter/DispatchTypes.hpp

@@ -82,6 +82,7 @@ namespace Microsoft::Console::VirtualTerminal::DispatchTypes
     enum PrivateModeParams : unsigned short
     {
         DECCKM_CursorKeysMode = 1,
+        DECANM_AnsiMode = 2,
         DECCOLM_SetNumberOfColumns = 3,
         DECSCNM_ScreenMode = 5,
         DECOM_OriginMode = 6,

src/terminal/adapter/ITermDispatch.hpp

@@ -54,6 +54,7 @@ class Microsoft::Console::VirtualTerminal::ITermDispatch
     virtual bool SetCursorKeysMode(const bool applicationMode) = 0; // DECCKM
     virtual bool SetKeypadMode(const bool applicationMode) = 0; // DECKPAM, DECKPNM
     virtual bool EnableCursorBlinking(const bool enable) = 0; // ATT610
+    virtual bool SetAnsiMode(const bool ansiMode) = 0; // DECANM
     virtual bool SetScreenMode(const bool reverseMode) = 0; // DECSCNM
     virtual bool SetOriginMode(const bool relativeMode) = 0; // DECOM
     virtual bool SetAutoWrapMode(const bool wrapAtEOL) = 0; // DECAWM
@@ -92,6 +93,7 @@ class Microsoft::Console::VirtualTerminal::ITermDispatch
 
     virtual bool DeviceStatusReport(const DispatchTypes::AnsiStatusType statusType) = 0; // DSR, DSR-OS, DSR-CPR
     virtual bool DeviceAttributes() = 0; // DA1
+    virtual bool Vt52DeviceAttributes() = 0; // VT52 Identify
 
     virtual bool DesignateCharset(const wchar_t wchCharset) = 0; // SCS
 

src/terminal/adapter/adaptDispatch.cpp

@@ -714,6 +714,19 @@ bool AdaptDispatch::DeviceAttributes()
     return _WriteResponse(L"\x1b[?1;0c");
 }
 
+// Routine Description:
+// - VT52 Identify - Reports the identity of the terminal in VT52 emulation mode.
+//   An actual VT52 terminal would typically identify itself with ESC / K.
+//   But for a terminal that is emulating a VT52, the sequence should be ESC / Z.
+// Arguments:
+// - <none>
+// Return Value:
+// - True if handled successfully. False otherwise.
+bool AdaptDispatch::Vt52DeviceAttributes()
+{
+    return _WriteResponse(L"\x1b/Z");
+}
+
 // Routine Description:
 // - DSR-OS - Reports the operating status back to the input channel
 // Arguments:
@@ -956,6 +969,9 @@ bool AdaptDispatch::_PrivateModeParamsHelper(const DispatchTypes::PrivateModePar
         // set - Enable Application Mode, reset - Normal mode
         success = SetCursorKeysMode(enable);
         break;
+    case DispatchTypes::PrivateModeParams::DECANM_AnsiMode:
+        success = SetAnsiMode(enable);
+        break;
     case DispatchTypes::PrivateModeParams::DECCOLM_SetNumberOfColumns:
         success = _DoDECCOLMHelper(enable ? DispatchTypes::s_sDECCOLMSetColumns : DispatchTypes::s_sDECCOLMResetColumns);
         break;
@@ -1129,6 +1145,20 @@ bool AdaptDispatch::DeleteLine(const size_t distance)
     return _pConApi->DeleteLines(distance);
 }
 
+// - DECANM - Sets the terminal emulation mode to either ANSI-compatible or VT52.
+// Arguments:
+// - ansiMode - set to true to enable the ANSI mode, false for VT52 mode.
+// Return Value:
+// - True if handled successfully. False otherwise.
+bool AdaptDispatch::SetAnsiMode(const bool ansiMode)
+{
+    // When an attempt is made to update the mode, the designated character sets
+    // need to be reset to defaults, even if the mode doesn't actually change.
+    _termOutput = {};
+
+    return _pConApi->PrivateSetAnsiMode(ansiMode);
+}
+
 // Routine Description:
 // - DECSCNM - Sets the screen mode to either normal or reverse.
 //    When in reverse screen mode, the background and foreground colors are switched.

src/terminal/adapter/adaptDispatch.hpp

@@ -58,6 +58,7 @@ namespace Microsoft::Console::VirtualTerminal
         bool SetGraphicsRendition(const std::basic_string_view<DispatchTypes::GraphicsOptions> options) override; // SGR
         bool DeviceStatusReport(const DispatchTypes::AnsiStatusType statusType) override; // DSR, DSR-OS, DSR-CPR
         bool DeviceAttributes() override; // DA1
+        bool Vt52DeviceAttributes() override; // VT52 Identify
         bool ScrollUp(const size_t distance) override; // SU
         bool ScrollDown(const size_t distance) override; // SD
         bool InsertLine(const size_t distance) override; // IL
@@ -68,6 +69,7 @@ namespace Microsoft::Console::VirtualTerminal
         bool SetCursorKeysMode(const bool applicationMode) override; // DECCKM
         bool SetKeypadMode(const bool applicationMode) override; // DECKPAM, DECKPNM
         bool EnableCursorBlinking(const bool enable) override; // ATT610
+        bool SetAnsiMode(const bool ansiMode) override; // DECANM
         bool SetScreenMode(const bool reverseMode) override; // DECSCNM
         bool SetOriginMode(const bool relativeMode) noexcept override; // DECOM
         bool SetAutoWrapMode(const bool wrapAtEOL) override; // DECAWM

src/terminal/adapter/conGetSet.hpp

@@ -46,6 +46,7 @@ namespace Microsoft::Console::VirtualTerminal
         virtual bool PrivateSetCursorKeysMode(const bool applicationMode) = 0;
         virtual bool PrivateSetKeypadMode(const bool applicationMode) = 0;
 
+        virtual bool PrivateSetAnsiMode(const bool ansiMode) = 0;
         virtual bool PrivateSetScreenMode(const bool reverseMode) = 0;
         virtual bool PrivateSetAutoWrapMode(const bool wrapAtEOL) = 0;
 

src/terminal/adapter/termDispatch.hpp

@@ -48,6 +48,7 @@ class Microsoft::Console::VirtualTerminal::TermDispatch : public Microsoft::Cons
     bool SetCursorKeysMode(const bool /*applicationMode*/) noexcept override { return false; } // DECCKM
     bool SetKeypadMode(const bool /*applicationMode*/) noexcept override { return false; } // DECKPAM, DECKPNM
     bool EnableCursorBlinking(const bool /*enable*/) noexcept override { return false; } // ATT610
+    bool SetAnsiMode(const bool /*ansiMode*/) noexcept override { return false; } // DECANM
     bool SetScreenMode(const bool /*reverseMode*/) noexcept override { return false; } // DECSCNM
     bool SetOriginMode(const bool /*relativeMode*/) noexcept override { return false; }; // DECOM
     bool SetAutoWrapMode(const bool /*wrapAtEOL*/) noexcept override { return false; }; // DECAWM
@@ -86,6 +87,7 @@ class Microsoft::Console::VirtualTerminal::TermDispatch : public Microsoft::Cons
 
     bool DeviceStatusReport(const DispatchTypes::AnsiStatusType /*statusType*/) noexcept override { return false; } // DSR, DSR-OS, DSR-CPR
     bool DeviceAttributes() noexcept override { return false; } // DA1
+    bool Vt52DeviceAttributes() noexcept override { return false; } // VT52 Identify
 
     bool DesignateCharset(const wchar_t /*wchCharset*/) noexcept override { return false; } // SCS
 

src/terminal/adapter/ut_adapter/adapterTest.cpp

@@ -162,6 +162,18 @@ class TestGetSet final : public ConGetSet
         return _privateSetKeypadModeResult;
     }
 
+    bool PrivateSetAnsiMode(const bool ansiMode) override
+    {
+        Log::Comment(L"PrivateSetAnsiMode MOCK called...");
+
+        if (_privateSetAnsiModeResult)
+        {
+            VERIFY_ARE_EQUAL(_expectedAnsiMode, ansiMode);
+        }
+
+        return _privateSetAnsiModeResult;
+    }
+
     bool PrivateSetScreenMode(const bool /*reverseMode*/) override
     {
         Log::Comment(L"PrivateSetScreenMode MOCK called...");
@@ -744,6 +756,8 @@ class TestGetSet final : public ConGetSet
     bool _privateSetKeypadModeResult = false;
     bool _cursorKeysApplicationMode = false;
     bool _keypadApplicationMode = false;
+    bool _privateSetAnsiModeResult = false;
+    bool _expectedAnsiMode = false;
     bool _privateAllowCursorBlinkingResult = false;
     bool _enable = false; // for cursor blinking
     bool _privateSetScrollingRegionResult = false;
@@ -1691,6 +1705,26 @@ class AdapterTest
         VERIFY_IS_TRUE(_pDispatch.get()->SetKeypadMode(true));
     }
 
+    TEST_METHOD(AnsiModeTest)
+    {
+        Log::Comment(L"Starting test...");
+
+        // success cases
+        // set ansi mode = true
+        Log::Comment(L"Test 1: ansi mode = true");
+        _testGetSet->_privateSetAnsiModeResult = true;
+        _testGetSet->_expectedAnsiMode = true;
+
+        VERIFY_IS_TRUE(_pDispatch.get()->SetAnsiMode(true));
+
+        // set ansi mode = false
+        Log::Comment(L"Test 2: ansi mode = false.");
+        _testGetSet->_privateSetAnsiModeResult = true;
+        _testGetSet->_expectedAnsiMode = false;
+
+        VERIFY_IS_TRUE(_pDispatch.get()->SetAnsiMode(false));
+    }
+
     TEST_METHOD(AllowBlinkingTest)
     {
         Log::Comment(L"Starting test...");

src/terminal/input/terminalInput.cpp

@@ -68,6 +68,15 @@ static constexpr std::array<TermKeyMap, 6> s_cursorKeysApplicationMapping{
     TermKeyMap{ VK_END, L"\x1bOF" },
 };
 
+static constexpr std::array<TermKeyMap, 6> s_cursorKeysVt52Mapping{
+    TermKeyMap{ VK_UP, L"\033A" },
+    TermKeyMap{ VK_DOWN, L"\033B" },
+    TermKeyMap{ VK_RIGHT, L"\033C" },
+    TermKeyMap{ VK_LEFT, L"\033D" },
+    TermKeyMap{ VK_HOME, L"\033H" },
+    TermKeyMap{ VK_END, L"\033F" },
+};
+
 static constexpr std::array<TermKeyMap, 20> s_keypadNumericMapping{
     TermKeyMap{ VK_TAB, L"\x09" },
     TermKeyMap{ VK_BACK, L"\x7f" },
@@ -150,6 +159,29 @@ static constexpr std::array<TermKeyMap, 20> s_keypadApplicationMapping{
     // TermKeyMap{ VK_TAB, L"\x1bOI" },   // So I left them here as a reference just in case.
 };
 
+static constexpr std::array<TermKeyMap, 20> s_keypadVt52Mapping{
+    TermKeyMap{ VK_TAB, L"\x09" },
+    TermKeyMap{ VK_BACK, L"\x7f" },
+    TermKeyMap{ VK_PAUSE, L"\x1a" },
+    TermKeyMap{ VK_ESCAPE, L"\x1b" },
+    TermKeyMap{ VK_INSERT, L"\x1b[2~" },
+    TermKeyMap{ VK_DELETE, L"\x1b[3~" },
+    TermKeyMap{ VK_PRIOR, L"\x1b[5~" },
+    TermKeyMap{ VK_NEXT, L"\x1b[6~" },
+    TermKeyMap{ VK_F1, L"\x1bP" },
+    TermKeyMap{ VK_F2, L"\x1bQ" },
+    TermKeyMap{ VK_F3, L"\x1bR" },
+    TermKeyMap{ VK_F4, L"\x1bS" },
+    TermKeyMap{ VK_F5, L"\x1b[15~" },
+    TermKeyMap{ VK_F6, L"\x1b[17~" },
+    TermKeyMap{ VK_F7, L"\x1b[18~" },
+    TermKeyMap{ VK_F8, L"\x1b[19~" },
+    TermKeyMap{ VK_F9, L"\x1b[20~" },
+    TermKeyMap{ VK_F10, L"\x1b[21~" },
+    TermKeyMap{ VK_F11, L"\x1b[23~" },
+    TermKeyMap{ VK_F12, L"\x1b[24~" },
+};
+
 // Sequences to send when a modifier is pressed with any of these keys
 // Basically, the 'm' will be replaced with a character indicating which
 //      modifier keys are pressed.
@@ -220,6 +252,11 @@ const wchar_t* const CTRL_QUESTIONMARK_SEQUENCE = L"\x7F";
 const wchar_t* const CTRL_ALT_SLASH_SEQUENCE = L"\x1b\x1f";
 const wchar_t* const CTRL_ALT_QUESTIONMARK_SEQUENCE = L"\x1b\x7F";
 
+void TerminalInput::ChangeAnsiMode(const bool ansiMode) noexcept
+{
+    _ansiMode = ansiMode;
+}
+
 void TerminalInput::ChangeKeypadMode(const bool applicationMode) noexcept
 {
     _keypadApplicationMode = applicationMode;
@@ -231,29 +268,44 @@ void TerminalInput::ChangeCursorKeysMode(const bool applicationMode) noexcept
 }
 
 static const std::basic_string_view<TermKeyMap> _getKeyMapping(const KeyEvent& keyEvent,
+                                                               const bool ansiMode,
                                                                const bool cursorApplicationMode,
                                                                const bool keypadApplicationMode) noexcept
 {
-    if (keyEvent.IsCursorKey())
+    if (ansiMode)
     {
-        if (cursorApplicationMode)
+        if (keyEvent.IsCursorKey())
         {
-            return { s_cursorKeysApplicationMapping.data(), s_cursorKeysApplicationMapping.size() };
+            if (cursorApplicationMode)
+            {
+                return { s_cursorKeysApplicationMapping.data(), s_cursorKeysApplicationMapping.size() };
+            }
+            else
+            {
+                return { s_cursorKeysNormalMapping.data(), s_cursorKeysNormalMapping.size() };
+            }
         }
         else
         {
-            return { s_cursorKeysNormalMapping.data(), s_cursorKeysNormalMapping.size() };
+            if (keypadApplicationMode)
+            {
+                return { s_keypadApplicationMapping.data(), s_keypadApplicationMapping.size() };
+            }
+            else
+            {
+                return { s_keypadNumericMapping.data(), s_keypadNumericMapping.size() };
+            }
         }
     }
     else
     {
-        if (keypadApplicationMode)
+        if (keyEvent.IsCursorKey())
         {
-            return { s_keypadApplicationMapping.data(), s_keypadApplicationMapping.size() };
+            return { s_cursorKeysVt52Mapping.data(), s_cursorKeysVt52Mapping.size() };
         }
         else
         {
-            return { s_keypadNumericMapping.data(), s_keypadNumericMapping.size() };
+            return { s_keypadVt52Mapping.data(), s_keypadVt52Mapping.size() };
         }
     }
 }
@@ -560,7 +612,7 @@ bool TerminalInput::HandleKey(const IInputEvent* const pInEvent)
     }
 
     // Check any other key mappings (like those for the F1-F12 keys).
-    const auto mapping = _getKeyMapping(keyEvent, _cursorApplicationMode, _keypadApplicationMode);
+    const auto mapping = _getKeyMapping(keyEvent, _ansiMode, _cursorApplicationMode, _keypadApplicationMode);
     if (_translateDefaultMapping(keyEvent, mapping, senderFunc))
     {
         return true;