Re:Tokenisation Is NP-Complete
by conartist6 on Tuesday December 24, 2024 @07:33AM
(#42501506)
CSTML and BABLR are designed to close this gap completely for people in your position.
We invented a new kind of zero just so that you could write arbitrary parsers and deal with things that are missing
That property of how good the design is for helping write and debug (useful error messages) seems like yet another metric that could be used
I also don't know if tokenizers can be divorced from parsers. Some languages have constructs that require context that comes from the parser