Add regex index extractors

These three files / scripts are 3 different implementations (python,
regex, regex-filtered) of the same thing: taking a regex set and a
bunch of needles, for each needle find the first matching regex, and
output its index (0-indexed).

This is the core loop of ua-parser, and allows validating that
regex-filtered matches a more naive version of the same process.
Happily I couldn't find any divergence although that means I did a
fair amount of useless work. Also the python version is really slow
compared to even the regex one, so probably don't use that...

`paste` allows using it to combine index extraction of multiple
domains as well as the original needle as TSV documents if that's of
use. This could also be expanded to multi-index extraction if that's a
need for anyone and should be checked more extensively.

Note that only the python version supports stdin input at this point,
I couldn't be arsed to do that with the Rust ones, but process
substitution ought work fine anyway? The needles are read on the go so
they should not need to be an actual file.

This may not be in a state fit for performance checking as the output
loop of the rust version is the worst (no buffering, no
stdout-locking).

Author: masklinn

regex-filtered/examples/matchindex.rs

@@ -0,0 +1,38 @@
+use clap::Parser;
+use std::io::BufRead;
+
+#[derive(Parser)]
+struct Args {
+    regexes: String,
+    useragents: String,
+}
+
+fn main() {
+    let Args {
+        regexes,
+        useragents,
+    } = Args::parse();
+    let regexes: Vec<_> = std::io::BufReader::new(std::fs::File::open(regexes).unwrap())
+        .lines()
+        .map(|l| regex::Regex::new(&l.unwrap()).unwrap())
+        .collect();
+
+    let mut uas = std::io::BufReader::new(std::fs::File::open(useragents).unwrap());
+    let mut line = String::with_capacity(150);
+    while let Ok(n) = uas.read_line(&mut line) {
+        if n == 0 {
+            break;
+        }
+        let line_ = line.strip_suffix("\n").unwrap_or(&line);
+        let m = regexes
+            .iter()
+            .enumerate()
+            .find(|(_, regex)| regex.is_match(line_));
+        if let Some((i, _)) = m {
+            println!("{i}");
+        } else {
+            println!("-");
+        }
+        line.clear();
+    }
+}

regex-filtered/examples/matchindex2.rs

@@ -0,0 +1,40 @@
+use clap::Parser;
+use std::io::BufRead;
+
+#[derive(Parser)]
+struct Args {
+    regexes: String,
+    useragents: String,
+}
+
+fn main() {
+    let Args {
+        regexes,
+        useragents,
+    } = Args::parse();
+    let regexes = regex_filtered::Builder::new()
+        .push_all(
+            std::io::BufReader::new(std::fs::File::open(regexes).unwrap())
+                .lines()
+                .map(Result::unwrap),
+        )
+        .unwrap()
+        .build()
+        .unwrap();
+
+    let mut uas = std::io::BufReader::new(std::fs::File::open(useragents).unwrap());
+    let mut line = String::with_capacity(150);
+    while let Ok(n) = uas.read_line(&mut line) {
+        if n == 0 {
+            break;
+        }
+        let line_ = line.strip_suffix("\n").unwrap_or(&line);
+        let m = regexes.matching(line_).next();
+        if let Some((i, _)) = m {
+            println!("{i}");
+        } else {
+            println!("-");
+        }
+        line.clear();
+    }
+}

scripts/matchindex

@@ -0,0 +1,32 @@
+#!/usr/bin/env python
+
+import argparse
+import re
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    'regexes',
+    help="regexes to try on the user agents",
+)
+parser.add_argument(
+    'useragents',
+    type=argparse.FileType(),
+    help="user agents to parse, `-` for stdin",
+)
+args = parser.parse_args()
+
+with open(args.regexes) as r:
+    regexes = [
+        re.compile(pattern.rstrip('\n'))
+        for pattern in r
+    ]
+
+with args.useragents as r:
+    for u in r:
+        u = u.rstrip('\n')
+        for i, p in enumerate(regexes):
+            if p.search(u):
+                print(i)
+                break
+        else:
+            print('-')