tldw; He says he doesn't advocate it but they get away with it by having the community test it out for them in their meta site. Then the community writes up the bugs.

He actually says " I'm not advocating that you shouldn't put in tests. [ The reason we can get away with this ] is that we have a great community. "

I take this to mean that he feels that StackOverflow doesn't need tests. Not that tests are useless.

User community as testers presents some interesting pros and cons.

Pros:

* Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail.

* Tests are unusually thorough

* Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing.

Cons:

* Tests cannot be run offline. Feature must be committed and deployed before tests can be run.

* Potentially large quantity of false positives (bad bug reports)

* Potentially large quantity of false negatives (nobody notices particular bug, release considered good)

* Does not work for non-user-visible features

So basically you trade the reliability of your tests for a substantial build/release speedup. Some users experience each bug, but they are the users who are actively using the meta-community and have signed up to experience more bugs. Still, lack of pre-release unit testing must radically increase the importance of VERY careful code reviews.

Not the decision I would have made, but definitely has the sorts of advantages that a small team of engineers drool would drool over.

Remember that our community writes bug reports but also vets bug reports. We rarely have to deal with bad reports. Interestingly, large quantities of false negatives are a non-issue.

Presumably the same reason why they don't have a ton of bad questions on stack overflow: their community scoring would apply just as much to bug reports

That's an accurate read.

- Stack Exchange employee