halbersa

We have a pool of MacOS hardware to run tests, and another pool to perform signing. The latter is a security risk so very tightly controlled and locked down. They are a pita to maintain.

Being able to sign on Linux will allow us to re-use the existing signing infrastructure we use for literally every other platform other than MacOS. It'll be more secure and much less maintenance.

One of the authors here, I can't exactly deny that line was added to sound impressive, so guilty as charged. However the savings are much higher than $20/day for a few reasons:

* Many tasks run on expensive instances (hardware acceleration, Windows)

* We have OSX/Android pools that run on physical devices in a data centre (these are an order of magnitude more expensive than Linux)

* There are ancillary costs. For example each task generates artifacts which incur storage costs. These artifacts are downloaded which incur transfer costs.

* There are also overhead costs (idle time, rebooting, etc) that aren't counted in the 10 years / day stat.

All these things see a corresponding decrease in costs with fewer tasks.

One of the authors here, first off thanks!

Yes a regression slipping through would far outweigh the benefits of reduced tests. The thing the post didn't make very clear is that thanks to our integration branch, the chance of a missed regression is still nearly zero. If the scheduling algorithm misses something, the failure will show up on a "backstop" push. These are pushes where we run everything, and then a human code sheriff will inspect any failures, and if something was missed figure out what caused it and back it out.

So the costs of missed regressions are: 1) More strain on the sheriffs (too much strain means the need to hire more) 2) More backouts which is annoying to developers and can mess up annotation (though we have ideas to fix the latter).

For the record, the algorithm with the 70% reduction in tests has a regression rate almost on par with the baseline (it's ~3-4% lower). This hasn't seemed to result in much additional strain on the pipeline.

> Could you ever imagine Mozilla undermining the big players like that today?

Enhanced tracking protection[1] was turned on by default for all users recently. This is a feature that hits at Google and Facebook's bottom line.. so I guess yeah, I can imagine Mozilla undermining the big players today. Not to mention other privacy initiatives like Containers/Facebook Container.

[1]: https://support.mozilla.org/en-US/kb/enhanced-tracking-prote...