I think this is worth challenging. A small sample means there's a higher likelihood that the sample is not very random. It's often pulled from some easily available source, like a group of college students.

An idealized perfect random small sample might be just fine for statistics, but there's little reason to believe that's the sample we're getting. Small samples should be a cause for suspicion.