Which is kinda funny, because nobody (ok, almost nobody) ever asks those questions when it comes to computer science or particularly IT security. Many of the "studies" thrown around would be laughed at by the community caring about scientific reliability.

If you would have the same standards for large parts of CS that shattered psychology then you'd have to admit some pretty dire things.

> Which is kinda funny, because nobody (ok, almost nobody) ever asks those questions when it comes to computer science or particularly IT security.

Actually, I've hardly seen any vendor study (especially Microsoft, but basically all vendors) which was NOT ridiculed by anyone who cares about science.

> If you would have the same standards for large parts of CS that shattered psychology then you'd have to admit some pretty dire things.

Can you give an example? Unlike "software engineering", which is neither science nor engineering but mostly a craft, Computer Science is a field of Math, and rarely has anything to do with probability (p-Hacking), or selecting a favorable data set; There's a lot of criticism directed at the relevance of test sets (e.g. MNIST), but that's basically the opposite problem, which might impede progress, but does yield out very well defined and verifyable conclusions.