Re:Use perfect hashing, instead of binary search, for keywor...
by rurban on Friday January 11, 2019 @03:52AM
(#18881888)
Devel::Tokenizer::C generates horrible code, even worse than gperf. Much better is my Perfect::Hash module, this postgresql module or the new perl5 mph script.
Haven't checked re2c but it looks like a naive lexer implementation generating a static DFA bytewise, not optimized for speed nor memory.
Better generators should really operate on the wordlevel or wider (SSE), not bytes. These are ~8x faster, in some case up to 200x. http://blogs.perl.org/users/rurban/2014/08/perfect-hashes-an...
In C, re2c [2] provides something similar, and there are similar tools available in other languages.
[1] https://metacpan.org/pod/Devel::Tokenizer::C
[2] http://re2c.org/