If you want speed, just build a lexer with ragel [1]. It's hard to go faster than a DFA.

[1] http://www.colm.net/open-source/ragel/