News for hackers. Stuff that matters.
Re:Doug McIlroy's C++ regular expression matching library by HelloNurse on Tuesday July 31, 2018 @11:16AM (#17653834)
Given a byte based encoding like UTF-8 or extended 8-bit Ascii variants or Shift-JIS, a regular expression over Unicode characters can be translated to a more complicated, but not necessarily slow to match, regular expression over bytes.
Re:Doug McIlroy's C++ regular expression matching library by burntsushi on Tuesday July 31, 2018 @12:07PM (#17654395)
Indeed. That's what I meant with: "For most DFA engines I've seen, they take advantage of "strings" themselves and define the alphabet to be the set of bytes. If you're searching UTF-8, then this also implies that your DFA must include UTF-8 decoding (which is generally the right choice for performance reasons anyway)." :-)
Re:Doug McIlroy's C++ regular expression matching library by HelloNurse on Tuesday July 31, 2018 @05:17PM (#17657238)
UTF-8 decoding usually means translating UTF-8 text to a more convenient character representation like UTF-32 or if you are naughty UTF-16 before processing, not adapting the application to work with UTF-8 data.
Re:Doug McIlroy's C++ regular expression matching library by burntsushi on Tuesday July 31, 2018 @07:02PM (#17658083)
I think the principle of charity applies here. What other possible interpretation of UTF-8 decoding is there for what I wrote? I mean, the automaton literally contains a UTF-8 automaton right in it, and semantically, it is totally matching in terms of codepoints.