If that's genuinely the basis for this reasoning, it only makes the argument seem even more dubious to me. If the statistical argument is really so sound, it shouldn't have to rely on our understanding of cosmology.

Also, is there no phenomenon here where a sufficiently large N becomes increasingly indistinguishable from infinity?