No you're totally right, there's nothing that says it has to be that way. But since it seems that all of the elementary particles coalesced out of a relatively homogenous energy field after the Big Bang, it doesn't seem unlikely that all of the particles could be explained by tiny permutations in an underlying universal field.