ANSI escape sequences in logs interfere with regex filtering

[sarub0b0]

Description

When container logs contain ANSI escape sequences (control characters for colors, cursor movements, etc.), regex filtering doesn't work as expected because the invisible control characters interfere with pattern matching.

This issue was discovered while investigating log filtering with netshoot containers, which often include shell output with ANSI escape sequences.

Current Behavior

For example, if a log line contains:

\x1b[31mERROR\x1b[0m: Connection failed

• Visually displayed as: ERROR: Connection failed (with red color)
• Actual string contains: \x1b[31mERROR\x1b[0m: Connection failed
• Regex ^ERROR match: ❌ Fails (line actually starts with \x1b)
• User expectation: ✅ Should match (user sees "ERROR" at the start)

Expected Behavior

Users should be able to filter logs based on what they visually see, not based on invisible control characters.

Examples of Affected Patterns

1. Line-start matching: ^ERROR doesn't match lines that appear to start with "ERROR"
2. Line-end matching: failed$ doesn't match lines that appear to end with "failed"
3. Empty line filtering: ^$ doesn't match visually empty lines that contain only control characters

Impact

• Affected containers: Debug containers (netshoot), interactive shells, applications with colored output
• User experience: Users need to know about invisible control characters to write effective filters
• Workaround difficulty: Currently no workaround available

Technical Background

This project already has an ANSI escape sequence parser (src/ansi/parser.rs) that handles:

• CSI sequences (colors, cursor movements, etc.)
• Graphics rendering (SGR)
• Cursor control

However, the parser doesn't cover all possible control sequences (OSC, DCS, terminal-specific sequences, etc.).

Proposed Solution

Maintain two versions of log content:

1. Raw content: For display (preserves colors and formatting)
2. Plain content: For filtering (all ANSI escape sequences removed)

Example implementation approach:

pub struct FilterableLogContent {
    pub raw: String,    // For display
    pub plain: String,  // For filtering
}

Implementation Considerations

1. Scope of stripping:

   • CSI sequences: \x1b[... (already parsed)
   • OSC sequences: \x1b]... (not yet covered)
   • Other control characters: C0/C1 control codes

2. Stripping method:

   • Option A: Regex-based (simple, covers 95-99% of cases)
   • Option B: Parser-based (accurate but requires extending existing parser)
   • Option C: Hybrid (recommended)

3. Performance impact: Need to strip ANSI codes for every log line

4. Memory impact: Storing both raw and plain versions doubles string storage

5. Compatibility: Ensure existing log display functionality isn't affected

Reference

• ANSI escape sequences: ECMA-48 (ISO/IEC 6429)
• Existing parser: src/ansi/parser.rs
• Related PR: fix: remove leading space from log content after timestamp parsing #872 (fixed leading space issue)

Additional Notes

This is a separate issue from #871 (leading space after timestamp). While both affect regex filtering, they have different root causes and solutions.

This issue is marked for future consideration and is not blocking current functionality.

[sarub0b0]

Verification: Rust regex crate behavior with ANSI escape sequences

I created a test to verify how Rust's regex crate handles ANSI escape sequences in strings.

Test Code

use regex::Regex;

fn main() {
    println!("=== Testing Rust regex crate behavior with ANSI escape sequences ===\n");

    // Test 1: Line-start matching
    let text1 = "\x1b[31mERROR\x1b[0m: Connection failed";
    let re1 = Regex::new(r"^ERROR").unwrap();
    println!("Test 1: Line-start matching");
    println!("  Text (visual): ERROR: Connection failed");
    println!("  Text (actual): {:?}", text1);
    println!("  Regex: ^ERROR");
    println!("  Match: {}", re1.is_match(text1));
    println!("  Expected: false (because line starts with \\x1b)\n");

    // Test 2: Pattern including ANSI codes
    let text2 = "\x1b[31mERROR\x1b[0m: Connection failed";
    let re2 = Regex::new(r"^\x1b\[31mERROR").unwrap();
    println!("Test 2: Pattern including ANSI codes");
    println!("  Text: {:?}", text2);
    println!("  Regex: ^\\x1b\\[31mERROR");
    println!("  Match: {}", re2.is_match(text2));
    println!("  Expected: true\n");

    // Test 3: Partial matching
    let text3 = "\x1b[31mERROR\x1b[0m: Connection failed";
    let re3 = Regex::new(r"ERROR").unwrap();
    println!("Test 3: Partial matching (no anchor)");
    println!("  Text: {:?}", text3);
    println!("  Regex: ERROR");
    println!("  Match: {}", re3.is_match(text3));
    println!("  Expected: true (partial match works)\n");

    // Test 4: Line-end matching
    let text4 = "Connection failed\x1b[0m";
    let re4 = Regex::new(r"failed$").unwrap();
    println!("Test 4: Line-end matching");
    println!("  Text (visual): Connection failed");
    println!("  Text (actual): {:?}", text4);
    println!("  Regex: failed$");
    println!("  Match: {}", re4.is_match(text4));
    println!("  Expected: false (because line ends with \\x1b[0m)\n");

    // Test 5: Empty line (control characters only)
    let text5 = "\x1b[0m";
    let re5 = Regex::new(r"^$").unwrap();
    println!("Test 5: Empty line with control characters");
    println!("  Text (visual): (empty)");
    println!("  Text (actual): {:?}", text5);
    println!("  Regex: ^$");
    println!("  Match: {}", re5.is_match(text5));
    println!("  Expected: false (not empty due to control chars)\n");

    // Test 6: Matching after stripping ANSI codes
    let text6 = "\x1b[31mERROR\x1b[0m: Connection failed";
    let stripped = strip_ansi_simple(text6);
    let re6 = Regex::new(r"^ERROR").unwrap();
    println!("Test 6: Matching after stripping ANSI codes");
    println!("  Original: {:?}", text6);
    println!("  Stripped: {:?}", stripped);
    println!("  Regex: ^ERROR");
    println!("  Match: {}", re6.is_match(&stripped));
    println!("  Expected: true\n");

    // Test 7: Complex ANSI sequence
    let text7 = "\x1b[1;38;2;255;0;0mERROR\x1b[0m";
    let re7 = Regex::new(r"^ERROR").unwrap();
    println!("Test 7: Complex ANSI sequence (24-bit color)");
    println!("  Text: {:?}", text7);
    println!("  Regex: ^ERROR");
    println!("  Match: {}", re7.is_match(text7));
    println!("  Expected: false\n");

    println!("=== Conclusion ===");
    println!("Rust regex crate treats ANSI escape sequences as regular characters.");
    println!("To match visual text, ANSI codes must be stripped first.");
}

fn strip_ansi_simple(s: &str) -> String {
    let re = Regex::new(r"\x1b\[[0-9;]*[a-zA-Z]").unwrap();
    re.replace_all(s, "").to_string()
}

Test Results

=== Testing Rust regex crate behavior with ANSI escape sequences ===

Test 1: Line-start matching
  Text (visual): ERROR: Connection failed
  Text (actual): "\u{1b}[31mERROR\u{1b}[0m: Connection failed"
  Regex: ^ERROR
  Match: false
  Expected: false (because line starts with \x1b)

Test 2: Pattern including ANSI codes
  Text: "\u{1b}[31mERROR\u{1b}[0m: Connection failed"
  Regex: ^\x1b\[31mERROR
  Match: true
  Expected: true

Test 3: Partial matching (no anchor)
  Text: "\u{1b}[31mERROR\u{1b}[0m: Connection failed"
  Regex: ERROR
  Match: true
  Expected: true (partial match works)

Test 4: Line-end matching
  Text (visual): Connection failed
  Text (actual): "Connection failed\u{1b}[0m"
  Regex: failed$
  Match: false
  Expected: false (because line ends with \x1b[0m)

Test 5: Empty line with control characters
  Text (visual): (empty)
  Text (actual): "\u{1b}[0m"
  Regex: ^$
  Match: false
  Expected: false (not empty due to control chars)

Test 6: Matching after stripping ANSI codes
  Original: "\u{1b}[31mERROR\u{1b}[0m: Connection failed"
  Stripped: "ERROR: Connection failed"
  Regex: ^ERROR
  Match: true
  Expected: true

Test 7: Complex ANSI sequence (24-bit color)
  Text: "\u{1b}[1;38;2;255;0;0mERROR\u{1b}[0m"
  Regex: ^ERROR
  Match: false
  Expected: false

=== Conclusion ===
Rust regex crate treats ANSI escape sequences as regular characters.
To match visual text, ANSI codes must be stripped first.

Key Findings

1. Line anchors (^, $) don't work with visual text - They match the actual string including control characters
2. Partial matching works - Patterns without anchors can match the visible text
3. Stripping ANSI codes enables proper filtering - After removing escape sequences, all regex patterns work as expected

This confirms that the proposed solution (maintaining both raw and plain text) is necessary for intuitive regex filtering.

[sarub0b0]

Investigation: regex crate documentation on control characters

I reviewed the regex crate documentation to understand how it handles control characters and ANSI escape sequences.

Key Findings from Documentation

1. Control Character Handling

The regex engine treats control characters as regular characters when matching. There is no special handling for ANSI escape sequences.

From the documentation:

• Control characters can be matched using hex escape sequences like \x1B
• Standard escape sequences are supported: \a (bell, \x07), \t (tab), \n (newline), \r (carriage return), etc.

2. The . (dot) Metacharacter

• By default: matches any character except newline (includes control characters)
• With s flag: matches newline as well
• Control characters like \x1b are treated as normal characters and matched by .

3. Anchors (^, $)

• ^: matches the start of line/string
• $: matches the end of line/string
• They match the actual string boundaries, not visual text boundaries

4. Byte-level Matching

• bytes::Regex type available for searching &[u8] haystacks
• Unicode mode can be disabled to match bytes directly

What's NOT in the Documentation

• ❌ No mention of ANSI escape sequences
• ❌ No automatic stripping or special treatment of control characters
• ❌ No distinction between "visual text" and "actual string"

POSIX Character Classes

The documentation includes POSIX character classes:

• [[:print:]]: printable characters ([ -~], ASCII \x20-\x7E)
• [[:graph:]]: graphical characters ([!-~], ASCII \x21-\x7E)
• [[:cntrl:]]: control characters ([\x00-\x1F\x7F])

Note: [[:cntrl:]] includes the escape character \x1b (byte 27) used in ANSI sequences.

Conclusion

The documentation confirms our test results:

• regex treats control characters as regular characters
• To filter logs based on visual text, ANSI escape sequences must be stripped beforehand
• This validates the proposed solution of maintaining both raw (for display) and plain (for filtering) versions of log content

[sarub0b0]

Investigation: Can POSIX character classes be used for ANSI stripping?

I tested whether POSIX character classes like [[:print:]] and [[:cntrl:]] could be used to strip ANSI escape sequences.

Test Code

use regex::Regex;

fn main() {
    println!("=== Testing POSIX character classes with ANSI escape sequences ===\n");

    let text = "\x1b[31mERROR\x1b[0m: Connection failed";
    println!("Original text:");
    println!("  Visual: ERROR: Connection failed");
    println!("  Actual: {:?}", text);
    println!("  Bytes: {:?}\n", text.as_bytes());

    // Test 1: Extract printable characters only
    println!("Test 1: Extract [[:print:]] only");
    let re_print = Regex::new(r"[[:print:]]+").unwrap();
    let printable: String = re_print.find_iter(text).map(|m| m.as_str()).collect();
    println!("  Pattern: [[:print:]]+");
    println!("  Result: {:?}", printable);
    println!("  Expected: Parts without \\x1b but includes [31m and [0m\n");

    // Test 2: Remove control characters
    println!("Test 2: Remove [[:cntrl:]]");
    let re_cntrl = Regex::new(r"[[:cntrl:]]").unwrap();
    let no_cntrl = re_cntrl.replace_all(text, "");
    println!("  Pattern: [[:cntrl:]]");
    println!("  Result: {:?}", no_cntrl);
    println!("  Expected: \\x1b removed but [31m and [0m remain\n");

    // Test 3: Extract graphical characters only
    println!("Test 3: Extract [[:graph:]] only");
    let re_graph = Regex::new(r"[[:graph:]]+").unwrap();
    let graphical: String = re_graph.find_iter(text).map(|m| m.as_str()).collect();
    println!("  Pattern: [[:graph:]]+");
    println!("  Result: {:?}", graphical);
    println!("  Expected: Similar to [[:print:]] but no spaces\n");

    // Test 4: Check what [[:cntrl:]] matches
    println!("Test 4: What [[:cntrl:]] matches");
    let re_cntrl_find = Regex::new(r"[[:cntrl:]]").unwrap();
    let cntrl_matches: Vec<_> = re_cntrl_find.find_iter(text).collect();
    println!("  Found {} control characters:", cntrl_matches.len());
    for m in cntrl_matches {
        println!("    {:?} at position {}", m.as_str().as_bytes(), m.start());
    }
    println!();

    // Test 5: Proper ANSI stripping (for comparison)
    println!("Test 5: Proper ANSI stripping with specific pattern");
    let re_ansi = Regex::new(r"\x1b\[[0-9;]*[a-zA-Z]").unwrap();
    let stripped = re_ansi.replace_all(text, "");
    println!("  Pattern: \\x1b\\[[0-9;]*[a-zA-Z]");
    println!("  Result: {:?}", stripped);
    println!("  Expected: Clean text without ANSI codes\n");

    // Test 6: Complex example with multiple control chars
    println!("Test 6: Complex text with various control characters");
    let complex = "\x1b[1;38;2;255;0;0mERROR\x1b[0m\nLine2\tTabbed";
    println!("  Original: {:?}", complex);

    let no_cntrl_complex = re_cntrl.replace_all(complex, "");
    println!("  Remove [[:cntrl:]]: {:?}", no_cntrl_complex);

    let stripped_complex = re_ansi.replace_all(complex, "");
    println!("  ANSI stripping: {:?}", stripped_complex);
    println!("  Note: [[:cntrl:]] removes \\n and \\t too!\n");

    println!("=== Conclusion ===");
    println!("[[:print:]] and [[:cntrl:]] are NOT sufficient for ANSI stripping because:");
    println!("1. [[:cntrl:]] only matches \\x1b, not the full sequence [31m");
    println!("2. [[:print:]] extracts printable parts but includes [31m, [0m");
    println!("3. [[:cntrl:]] also removes useful control chars like \\n and \\t");
    println!("4. Proper ANSI stripping requires pattern: \\x1b\\[[0-9;]*[a-zA-Z]");
}

Test Results

=== Testing POSIX character classes with ANSI escape sequences ===

Original text:
  Visual: ERROR: Connection failed
  Actual: "\u{1b}[31mERROR\u{1b}[0m: Connection failed"
  Bytes: [27, 91, 51, 49, 109, 69, 82, 82, 79, 82, 27, 91, 48, 109, 58, 32, 67, 111, 110, 110, 101, 99, 116, 105, 111, 110, 32, 102, 97, 105, 108, 101, 100]

Test 1: Extract [[:print:]] only
  Pattern: [[:print:]]+
  Result: "[31mERROR[0m: Connection failed"
  Expected: Parts without \x1b but includes [31m and [0m

Test 2: Remove [[:cntrl:]]
  Pattern: [[:cntrl:]]
  Result: "[31mERROR[0m: Connection failed"
  Expected: \x1b removed but [31m and [0m remain

Test 3: Extract [[:graph:]] only
  Pattern: [[:graph:]]+
  Result: "[31mERROR[0m:Connectionfailed"
  Expected: Similar to [[:print:]] but no spaces

Test 4: What [[:cntrl:]] matches
  Found 2 control characters:
    [27] at position 0
    [27] at position 10

Test 5: Proper ANSI stripping with specific pattern
  Pattern: \x1b\[[0-9;]*[a-zA-Z]
  Result: "ERROR: Connection failed"
  Expected: Clean text without ANSI codes

Test 6: Complex text with various control characters
  Original: "\u{1b}[1;38;2;255;0;0mERROR\u{1b}[0m\nLine2\tTabbed"
  Remove [[:cntrl:]]: "[1;38;2;255;0;0mERROR[0mLine2Tabbed"
  ANSI stripping: "ERROR\nLine2\tTabbed"
  Note: [[:cntrl:]] removes \n and \t too!

=== Conclusion ===
[[:print:]] and [[:cntrl:]] are NOT sufficient for ANSI stripping because:
1. [[:cntrl:]] only matches \x1b, not the full sequence [31m
2. [[:print:]] extracts printable parts but includes [31m, [0m
3. [[:cntrl:]] also removes useful control chars like \n and \t
4. Proper ANSI stripping requires pattern: \x1b\[[0-9;]*[a-zA-Z]

Key Findings

POSIX character classes are NOT suitable for ANSI stripping:

1. [[:cntrl:]] only matches the escape character (\x1b), not the full sequence

   • Result: [31mERROR[0m - the printable parts [31m and [0m remain

2. [[:print:]] includes the printable parts of ANSI sequences

   • Characters like [, 3, 1, m are printable, so they're extracted

3. [[:cntrl:]] removes ALL control characters including \n and \t

   • This breaks log structure by removing newlines and tabs
   • Not desirable for log filtering

4. Proper ANSI stripping requires specific pattern matching

   • CSI sequences: \x1b\[[0-9;]*[a-zA-Z]
   • This preserves useful control chars (newlines, tabs) while removing ANSI codes

Conclusion

The proposed solution in this issue (stripping ANSI codes with dedicated patterns) is the correct approach. POSIX character classes cannot be used as a shortcut.