News for hackers. Stuff that matters.
Re:Re2c: A free and open-source lexer generator for C and C++ by pubby on Friday March 01, 2019 @01:37PM (#19283892)
Lexer generators seem like a useful algorithm wrapped up in heaps of framework bullshit. Wouldn't it be better to have a simple program that inputs regex and outputs a DFA than one that handles the entire lexing process?
Re:Re2c: A free and open-source lexer generator for C and C++ by jules on Friday March 01, 2019 @03:00PM (#19284697)
Lexers run multiple regexes in tandem to determine the next token type. To do that with your method you'd have to run multiple DFAs on the same string. Lexers can do it with 1 DFA by having multiple different final states (non-final + 1 per token type vs non-final + final).
Re:Re2c: A free and open-source lexer generator for C and C++ by ori_b on Friday March 01, 2019 @05:06PM (#19285794)
You can do that from a regex api fairly easily. Something like:

    re = regex.compilemulti([
         (".*@(.*)\\.(com|org|net)", TagEmail),
         ("http://.*, TagURL),
         ("[a-z]*", TagWord)
    ])

    match regex.tagmatch(re, pat)
    | (TagEmail, submatches):    domain=submatches[1]
    | (TagURL, submatches):      url=submatches[0]
    | (TagWord, _):              /* nothing */
    ;;
I haven't had time to do it in Myrddin's libregex (https://myrlang.org/doc/libregex/) yet, but I've got plans to do something like this.
Re:Re2c: A free and open-source lexer generator for C and C++ by jules on Saturday March 02, 2019 @04:06AM (#19288375)
Sure, and that's exactly what re2c does too. It lets you do a match on multiple regexes:

  regex1 { code that executes if regex1 matches }
  regex2 { code that executes if regex2 matches }
  regex3 { code that executes if regex3 matches }
Re:Re2c: A free and open-source lexer generator for C and C++ by ori_b on Saturday March 02, 2019 @04:45PM (#19291352)
Yes, compiled into source code, which doesn't let you change the set of regexes at runtime.
Re:Re2c: A free and open-source lexer generator for C and C++ by ksherlock on Saturday March 02, 2019 @12:28AM (#19287742)
> Wouldn't it be better to have a simple program that inputs regex and outputs a DFA than one that handles the entire lexing process?

Like... re2c?

Re:Re2c: A free and open-source lexer generator for C and C++ by pubby on Saturday March 02, 2019 @02:47PM (#19290797)
It outputs code, no? I want tools that output graphs.