Re:Lessons learned building a toy compiler
by auggierose on Friday July 07, 2017 @12:15PM
(#14719573)
Well you can do both at the same time, it is called local lexing: https://arxiv.org/abs/1702.03277
[ Parent
| on HN ]
Re:Lessons learned building a toy compiler
by throwaway2016a on Friday July 07, 2017 @12:31PM
(#14719716)
I did mention in my post that the separate lexing step is not strictly necessary. I was just pointing out that a lexer follows the definition of "quintessential" more than parser does.
That paper is very recent. It sounds like it is still doing both steps just concurrently and with feedback. It isn't skipping the lexing step altogether. The benefit of separation is decreased code complexity and easier debugging.
Re:Lessons learned building a toy compiler
by auggierose on Friday July 07, 2017 @01:12PM
(#14720074)
Yes, both stages are still present in that approach, and that is a good thing, because clearly it is useful to separate lexing and parsing. It's just that the two stages are intertwined and thus lexing can be dependent on the parsing context. So it is not really going from "raw text to parsing".