That's because there are hundreds of thousands or millions of draws occurring. You need to account for what's called the multiple testing problem. [1] Some ways to do this are to control the family-wise error rate (FWER) or the false discovery rate (FDR). You can control the FWER with the Bonferroni correction or the Sidak correction and you can control the FDR with the Benjamini-Hochberg procedure. You are completely correct about things like this happening by chance, and this is the cause of publication bias in science (since there is a predisposition for publishing positive results).

1. https://en.wikipedia.org/wiki/Multiple_comparisons_problem

No, it doesn't happen all the time. It happens, on average, 1 time in 649,000. That's what those odds mean.