News for hackers. Stuff that matters.
Re:On the Complexity and Performance of Parsing with Derivat... by jblow on Saturday June 25, 2016 @01:07PM (#11977023)
Except no, lexing is trivial; it is by far the easiest part of writing a compiler. You don't need anything fancy, you just type in the code.
Re:On the Complexity and Performance of Parsing with Derivat... by programLyrique on Saturday June 25, 2016 @01:45PM (#11977197)
When you add unicode, it can become more complicated (or rather, cumbersome)...
Re:On the Complexity and Performance of Parsing with Derivat... by jblow on Saturday June 25, 2016 @05:56PM (#11978184)
Well, you just need to call unicode_next_character all the time instead of saying s++, similarly for whitespace, similarly for asking whether a character can initiate or continue an identifier, etc. It does not change the basic nature of the task at all.
Re:On the Complexity and Performance of Parsing with Derivat... by programLyrique on Sunday June 26, 2016 @04:38AM (#11979908)
Sure, but if you are using a language without support for unicode, and you don't use a dedicated library (which would be already using a kind of lexer, wouldn't it?), you have also to parse these unicode characters yourself.
Re:On the Complexity and Performance of Parsing with Derivat... by yoklov on Sunday June 26, 2016 @06:09PM (#11983017)
A unicode_next_character function is very simple to write regardless of unicode support in your language.

I usually write it as a small (256 byte) lookup table where each entry tells you how many characters to skip next. If you don't use a lookup table, its 4 single-line `if` statements (and if you do it's a one liner, plus however many lines the table takes).

Re:On the Complexity and Performance of Parsing with Derivat... by jblow on Monday June 27, 2016 @02:41AM (#11984514)
There's free source code for basic Unicode operations all over the internet.
Re:On the Complexity and Performance of Parsing with Derivat... by nulltype on Sunday June 26, 2016 @12:45AM (#11979410)
Here's a video of Rob Pike, developer of Go, talking about a lexer he built for the Go text/template package:

https://www.youtube.com/watch?v=HxaD_trXwRE

It supports unicode.

Re:On the Complexity and Performance of Parsing with Derivat... by wolfgke on Saturday June 25, 2016 @01:13PM (#11977043)
If you want to write a lexer from scratch, this is IMHO not trivial. If you use a generator it is much easier, but so is also using a parser generator for the CFG parsing stage.
Re:On the Complexity and Performance of Parsing with Derivat... by jblow on Saturday June 25, 2016 @05:54PM (#11978173)
If you can't write a lexer by hand, just forget trying to write a compiler that does anything interesting, because the lexer is MUCH easier than any other part of the compiler.

There are a lot of reasons for this, but one of the basic ones is that the lexer does not need to interact in a complex way with the compiler's state. It is a relatively simple pipeline where characters go in one end and tokens come out the other.

Re:On the Complexity and Performance of Parsing with Derivat... by SamReidHughes on Saturday June 25, 2016 @01:38PM (#11977161)
Hand-rolling a lexer seems pretty trivial to me. The code writes itself. A generator is a tool whose limitations you have to work around, be it for a lexer or a parser.