That's something that varies from field to field. In physics, a new discovery requires a five-sigma result, meaning a result that has a one in 3.5 million probability of arising by chance rather than the legitimate detection of a new particle or phenomenon.
http://www.select-statistics.co.uk/blog/webmaster/one-sigma-...
In psychology, by contrast, a p-value of 0.05 (i.e. 2 sigma) is accepted as statistically significant, which by itself may explain the sad state of the field.
http://en.wikipedia.org/wiki/P-value
> I would assume you trust the hardware and only scrutinise the software?
No, a scientist doesn't trust either hardware or software, but hardware can be checked and validated. In the case of Mathematica, the software cannot be checked and validated, it must be trusted. And for a scientist, trust cannot be distinguished from faith.
I think what is really being asked is: "What level of confidence is required?".
I would assume you trust the hardware and only scrutinise the software? In contrast: are results tested on a plurality of machines with intentionally, measurably different hardware and the absence of discrepancies implies that the hardware is not a factor with regard to the problem being solved? Presumedly you'd need some sort of standardised test suite to pass on your hardware to really confidently declare the hardware to not be a factor.
I doubt this is the case, but like the previous poster, I'm quite curious as to the specific level of scrutiny applied here.