So, you want to chunk really fast?

Re:So, you want to chunk really fast? by stabbles on Monday January 05, 2026 @02:35PM (#46503612)

For the particular case of the 5 delimiters '\n', '.', '?', '!', and ';', it just happens to be so that you can do this as a single shuffle instruction, replacing the explicit lookup table.

You can do this whenever `c & 0x0F` is unique for the set of characters you're looking for.

See https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottlenec... for details.

[ Parent | on HN ]

Re:So, you want to chunk really fast? by bhavnicksm on Monday January 05, 2026 @04:05PM (#46504962)

Hey! Author of the blog here.

This is pretty cool~ Thanks for suggesting this, I will read this in detail and add it to the next (0.5.0) release of memchunk.

[ Parent | on HN ]

Re:So, you want to chunk really fast? by CyberDildonics on Monday January 05, 2026 @06:04PM (#46506402)

Why does your title not have any context?

[ Parent | on HN ]

Re:So, you want to chunk really fast? by dataflow on Monday January 05, 2026 @04:58PM (#46505584)

Note your compiler might turn that _mm256_set_epi64x into a load from memory, so there might still be memory accesses you don't expect.

[ Parent | on HN ]

Re:So, you want to chunk really fast? by akoboldfrying on Monday January 05, 2026 @07:47PM (#46507336)

This is a really neat technique, well explained at your link.

Now that I understand it, I'd describe it as: For each byte, based on its bottom 4 bits, map it to either the unique "target" value that you're looking for that has those bottom 4 bits, or if there is no such target value, to any value that is different from what it is right now. Then simply check whether each resulting byte is equal to its corresponding original byte!

Not sure if the above will help people understand it, but after you understand it, I think you'll agree with the above description :)

[ Parent | on HN ]