Wouldn’t that be true for any 32-bit hash? The birthday paradox would suggest you only need 2^16 input values to generate a collision right?

xxhash32 does give a couple of collisions on the same file, yep.

    734c8981 abloom
    734c8981 sating
    ef336a7d contravene
    ef336a7d seducers

Strange that all three collisions include "'s"

Considering that 43% of the words (in my dictionary at least) include "s" it's not so strange:

    $ cat /usr/share/dict/words | wc -l
      235976
    $ cat /usr/share/dict/words | grep s |  wc -l
      101882

GP meant that all collisions contained 's - a possessive apostrophe (if I remember my grammar correctly?). oncogene's, toadstool's, Carib's.

The /usr/share/dict/words on my macOS machine doesn't seem to have any 's words in it.

counterclaiming Doesn’t have any s and stemming doesn’t have any possessive. I think drawing conclusions from a 32bit hash against a 123k word list is fruitless. You’d expect at least 1 collision due simply to birthday paradox.

    oncogene's <> counterclaiming

    ingrates <> toadstool's

    Carib's <> stemming

all 3 pairs have one word ending in "'s", which I think is what the GGGP (?) was pointing out

Yes. But I didn't realize that 43% of the words in that dictionary contain "'s".

In that case, the probability that, for 3 collisions, there is at least one element in every collision contains "'s" is 31%. That seems reasonable to me.

but 43% of the words contain s, not 's

> counterclaiming Doesn’t have any s and stemming doesn’t have any possessive.

But "oncogene's" (collides with counterclaiming) and "Carib's" (collides with stemming) do.

Also I did point out that the 64 and 128 bit variants have no collisions.

Yeah, but it's an uninteresting test. As others point out, 32-bit XXH3 has the same issue. It wouldn't surprise me if 32-bit AES did too. It has nothing to do with the quality of the hash algorithm & more to do with the fact that you're not building a perfect hash with such a small hash size.

> Yeah, but it's an uninteresting test.

On its own, yes, fair, but it was combined with the 64 and 128 bit variants which aren't uninteresting.

> As others point out, 32-bit XXH3 has the same issue.

Yes, me.

> It has nothing to do with the quality of the hash algorithm

The original post asked "I would like to see the collision distribution on /usr/share/dict/words." and that is what they got a demonstration of - some collisions on 32 bit, none on 64 or 128 bit. What else could be done to satisfy their original request in a way that would be interesting to you?

Point to the fact that it passes smasher in addition to that 64-bit and 128-bit variants don’t have collisions and ignore 32-bit altogether. I don’t know why libraries even bother producing 32-bit variants at this point because of how easy a collision is.

I thought interesting that all three collision pairs have an 's ending, but looking into this dictionary, no less than 25% of the words finish by it. weird dict.