Re:Re2c: A free and open-source lexer generator for C and C++
by jules on Friday March 01, 2019 @03:00PM
(#19284697)
Lexers run multiple regexes in tandem to determine the next token type. To do that with your method you'd have to run multiple DFAs on the same string. Lexers can do it with 1 DFA by having multiple different final states (non-final + 1 per token type vs non-final + final).
[ Parent
| on HN ]
Re:Re2c: A free and open-source lexer generator for C and C++
by ori_b on Friday March 01, 2019 @05:06PM
(#19285794)
You can do that from a regex api fairly easily. Something like:
re = regex.compilemulti([
(".*@(.*)\\.(com|org|net)", TagEmail),
("http://.*, TagURL),
("[a-z]*", TagWord)
])
match regex.tagmatch(re, pat)
| (TagEmail, submatches): domain=submatches[1]
| (TagURL, submatches): url=submatches[0]
| (TagWord, _): /* nothing */
;;
I haven't had time to do it in Myrddin's libregex (https://myrlang.org/doc/libregex/) yet, but I've got plans to do something like this.
Re:Re2c: A free and open-source lexer generator for C and C++
by jules on Saturday March 02, 2019 @04:06AM
(#19288375)
Sure, and that's exactly what re2c does too. It lets you do a match on multiple regexes:
regex1 { code that executes if regex1 matches }
regex2 { code that executes if regex2 matches }
regex3 { code that executes if regex3 matches }