I mean, it's pretty unlikely to happen, but it's possible. Plus it doesn't matter if all you cared about was that the HEAD of master passes tests, but not the whole history.
However, the current approach, if rebasing is used to squash every merge request into a single commit on top of master, allows you to have a linear history of the master branch where every commit is known to pass the whole test suite that was in existence at that moment, which, I think, helps a lot when bisecting.
I'm not sure if Rust compresses merge requests into a single commit on top of master, though.
What we often see is m+A fails because of a flake, and m+A+B+C+D+E passes and merges as a batch. We have a lot of flaky tests. :(
bors' rollup support only lets one mark things for rollup, which show up differently in the queue. It's effectively a priority=-1. This is for things which probably won't break the build, and contributors make a batch rollup every now and then (bors has a button for this). Generally when I do rollups I go through these and ensure they're green on travis.
I've considered giving homu (the engine behind bors) better support for automatically doing this (at least automatically showing travis statuses) but I don't have much time.
So far we haven't been able to stomach doing parallel integration builds because of the doubling of expenses.
As you see, our current solution is rollups, where a human batches low-risk PRs into one big PR. It works just ok, and is super high-maintenance.
In that case, it wouldn't double expenses. Assuming batches are slightly effective, it should reduce expenses-- with >50% success rate with batches of >=3 PRs, you don't have to do individual testing for the later PRs in the batch.
I think it would double expenses because our CI runs at capacity. To do parallel builds we would have to contract for double the compute resources. (With a different purchase structure for our CPU time you may be right about that, but not sure).
Using batch testing as a default would both reduce costs and increase merge speed, assuming that your commits usually pass testing.
The downside of using batching as a default is that it wouldn't test every commit in isolation. That means it wouldn't necessarily be safe to roll back to a particular commit if that commit was tested as part of a batch. E.g. if patches A, B and C were tested together, then it's not certain that patch A by itself would pass the tests.
For example, if the pass rate is 80%, then by batching 3 PRs together there's a 51.2% chance to pass with all three, which would save 2 test runs. and a 48.8% to fail, which would cost 1 extra run. That's a pretty substantial increase in throughput.
One solution is speculative batched testing. Kubernetes uses this to keep up with its very high rate of PRs (>40 merges/day) and ~1hr maximum test times.
Given a queue of patches A, B, C, D, start the normal testing against A, but also start testing the merge of A+B+C+D against the master. If the batch passes first, merge them all. If A is merged first, it's still "compatible" with A+B+C+D, so they can go ahead. Doing a batch in parallel with a single PR like this slightly increases testing load, but ideally doesn't cause any slowdowns versus the fully serialized case.
It looks like bors already supports something like this with "rollups", where simple fixes can be marked for testing together: https://internals.rust-lang.org/t/batched-merge-rollup-featu...