News for hackers. Stuff that matters.
Re:Fast UTF-8 validation by jart on Tuesday October 20, 2020 @09:55PM (#24844149)
If the natural logic is broken then you've misunderstood the thing's nature. UTF-8 behaves more like a communications stream that was shoehorned into the purpose of character arrays. If you think about it in that way, then decoding can be done simply and elegantly:

    #define bsr(u)          ((sizeof(int) * 8 - 1) ^ __builtin_clz(u))
    #define ThomPikeByte(x) ((x) & (((1 << ThomPikeMsb(x)) - 1) | 3))
    #define ThomPikeMsb(x)  (((x)&0xff) < 252 ? bsr(~(x)&0xff) : 1)

    void ThompsonPikeDecoder(const char *s) {
      unsigned c, w = 0, t = 0;
      do {
        c = *s++ & 0xff;
        if (0200 <= c && c < 0300) {
          w = w << 6 | c & 077;
        } else {
          if (t) {
            printf("%04x\n", w);
            t = 0;
          }
          if (c < 0200) {
            printf("%04x\n", c);
          } else {
            w = ThomPikeByte(c);
            t = 1;
          }
        }
      } while (c);
    }
That code generalizes to any 32-bit number. It's the full superset of intended behaviors including things like stream synchronization. It'll even decode numbers that were arbitrarily banned by the IETF e.g. \300\200. Most importantly, since it doesn't require a validation pass beforehand, does that mean it goes faster than the OP's code in praxis? D:
Re:Fast UTF-8 validation by Dylan16807 on Wednesday October 21, 2020 @03:57AM (#24845687)
I strongly question "doesn't require a validation pass".

1. Your code drops naked continuation bytes entirely.

2. If a valid multibyte character is followed by extraneous continuation bytes it will not decode correctly.

3. Using the wrong number of continuation bytes is a giant mess and can give you arbitrary numbers in multiple ways.

The second one especially is a behavior that's hard to defend.