I saw something like that one time - the new version had one rarely-used broken API endpoint; clients who hit that would silently retry until eventually hitting an old instance which worked and then they’d be on their way. It was rarely used enough that the total number of retries didn’t trigger any alarms, and failed fast so that failing 19 times before working on the 20th attempt didn’t cause any timeouts.

We now have monitoring of success-rate-per-endpoint, so even if 99.9% of requests are successful, if a single function crashes 100% of the time we should still notice :)

If you have a marketplace of some sort, and users of your new code (at 95% rollout) cannot see postings made from some subset of other users, but the 5% on the old version still can, you might not notice that volume of sales of things posted before the rollout has dropped by 95% where you would notice if it dropped by 100%.

Most of the time this would happen, the difference between "95% of people can't see a posting" and "100% of people can't see a posting" would be pretty small, but I guess if the marketplace is very liquid the difference might actually be substantial. But unless you're uber or someplace like that, where there's a big difference between "95% of drivers can't see a ride request (so it takes slightly longer to find a driver)" and "100% of drivers can't see a ride request (so no driver is ever found)", I can't imagine too many cases where the difference is likely to make a practical difference.

This would assume that all types of activity are evenly distributed across all users. In practice, I've seen many things where a small, small subset of users are responsible for a disproportionate amount of volume, and the odds of all of them being in the 5% allocation of "old" code is higher than you'd think.

Assuming your rollout is truly randomised, I find it highly unlikely that an absolutely random 5% of your users (randomised everytime as well) are going to cause your fall. It's still possible, but I'm not giving up on an elegant idea for that reason.

It's certainly unlikely, but Murphy's Law can get you sooner or later. Best hope that hot potato doesn't land in your lap at a bad time.

Race conditions can definitely do that, speaking from experience. I’ve fixed some that have been in the source code for a year before they were discovered.