News for hackers. Stuff that matters.
Re:2000x performance win by barrkel on Saturday December 10, 2011 @10:24AM (#3337693)
Unfortunately, it's not as simple as that. There is more than one way of representing many text sequences using Unicode; for example, combining diacritics vs accented characters. To do the Right Thing (tm), you need to normalize, and that adds at least another pass over the whole string.

If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.

Re:2000x performance win by _delirium on Saturday December 10, 2011 @12:49PM (#3338005)
In addition, conforming Unicode implementations have to ensure that their input is well-formed Unicode, and reject non-conforming characters (for some security-related reasons). So the original goal of UTF8, that it could be usable for straight byte-level search (the UTF8 RFC mentions a goal of allowing Boyer-Moore to be applied directly) is no longer really possible.

edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932

Re:2000x performance win by Confusion on Sunday December 11, 2011 @05:45AM (#3339817)
It's not necessarily grep's responsibility to normalize strings. They could very well stipulate that it is the responsibility of the user to make sure the both the input and the search expression are already normalized. That way, the normalization could be done by a separate program, resulting in the reusability and separation of concerns that is The Unix Way.
Re:2000x performance win by barrkel on Sunday December 11, 2011 @10:20AM (#3340198)
I fully agree, if you redefine grep as a byte search program, rather than for text search. Rule it out for most languages apart from English.

To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched:

    #!/usr/bin/bash
    regex="$1"; shift
    unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")"