It doesn’t matter if it’s surprising. It might be a measurement mistake, a mixed population, a fat tail or a poor sample. It’s different therefore an outlier.

There are many statistical tests that will tell you that based on the observations alone (with no model). Basically this depends on how many 5s there are.

Every single one of those tests makes an implicit assumption about the distribution, explicit in its derivation. Usually, it is that the distribution is Gaussian.