News for hackers. Stuff that matters.
Re:Regex Engine Internals as a Library by LoganDark on Wednesday July 05, 2023 @01:20PM (#36603627)
> They just usually don't gain much traction because I suspect you're overstating their general utility. :-)

I find myself desiring a DFA over LLM tokens right now, which are 16 bits each, at least for this model.

I, personally, am very pleased with the idea of attempting to build a dense DFA with that vocabulary. Maybe your computer would explode!

Re:Regex Engine Internals as a Library by telotortium on Wednesday July 05, 2023 @04:24PM (#36606586)
Could you not encode the tokens into 8-bit bytes? Rust regex supports byte strings: https://docs.rs/regex/latest/regex/bytes/struct.Regex.html
Re:Regex Engine Internals as a Library by LoganDark on Thursday July 06, 2023 @06:54AM (#36614002)
I'm dumb.

Thank you, I'll look into it.