I'm struggling with this idea. You could end up tokenizing the inside of an unopened string literal for (contrived) ex :
String s = hello world";
[-> typename:'String' id's' op'=' id'hello' id'world' ...]
foo("blah"); (...)
[-> strlit';\nfoo(' id'blah' ...]
unknown id 'hello'
In a mature product, you would apply heuristics to help you provide the most useful feedback to the user while optimizing the time it takes to do so.
Users love consolidated, efficient feedback without having to peel back errors one at a time. It’s a distinguishing feature when you can find a way to do it that suits your input, and specifically so because it can be a hard problem to solve well!
Re:The Super Tiny Compiler
by 1letterunixname on Thursday March 02, 2023 @11:33PM
(#35005839)
Precise diagnostics usually follow one of 2 approaches:
1. Hand-written, recursive descent
2. Error matching expanded grammar beyond minimal gramar
Re:The Super Tiny Compiler
by 1letterunixname on Thursday March 02, 2023 @11:31PM
(#35005825)
Strings typically have a token state context. Antlr allows pushing and popping a context state. Not a CFG and won't work with simple tools or book approaches.
Re:The Super Tiny Compiler
by PretzelPirate on Friday March 03, 2023 @09:09AM
(#35009820)
Check out this section of Crafting Interpreters on Error Recovery: https://craftinginterpreters.com/parsing-expressions.html#pa...
In general it's useful for a compiler to not just hit the first error it can find in the source code and immediately error out. Instead, if you keep parsing after encountering an error, you can often encounter more errors, so that you can give the user a list of errors they need to fix, not just the first one. In that sense, a compiler's job isn't over the instant it encounters a syntax error, so the extra tokenization would not be useless.