Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.

Let's say your team makes 25 commits per day.

25 * ~300 working days = 7,500 commits per year

That would take 133+ years to reach 1 in a million.

The more interesting metric to me is how often the build gets broken.

Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-test-deploy cycle at least once a day, hence the importance placed on rock-solid testing systems that should only ever fail when the tested code itself fails.

It just seemed to me like you were bragging that tests get run over and over again. They only need to get run if any new code is committed, of course.

And what kind of commit is being checked in every 9 minutes? How big is the dev team? Seems like an awful lot of commits. Is each one a full-fledged feature / bug fix for the site, or are many 1-line changes to the code?

It's true that any particular test that spuriously fails one in a million times may never fail. But if you have tens of thousands of tests, and you do tens of test runs per day, you'll have a test spuriously fail once a day or so.

He means that test failures are not acceptable in their culture, and they have a LOT of tests.