Re:Regex Engine Internals as a Library
by telotortium on Wednesday July 05, 2023 @04:24PM
(#36606586)
Could you not encode the tokens into 8-bit bytes? Rust regex supports byte strings: https://docs.rs/regex/latest/regex/bytes/struct.Regex.html
I find myself desiring a DFA over LLM tokens right now, which are 16 bits each, at least for this model.
I, personally, am very pleased with the idea of attempting to build a dense DFA with that vocabulary. Maybe your computer would explode!