It works because a comment by definition goes until a newline, so you can’t have consecutive comments without a newline token in between.
I’ll look into if there’s a better way to do it than this based on your suggestion, thanks!
I’ll look into if there’s a better way to do it than this based on your suggestion, thanks!
(returning ‘comment’ and ‘whitespace’ tokens would fix this, and would make it possible to reuse the lexer for pretty-printing/syntax coloring)