Interesting question: does a mile in CA four years ago equal a mile today? Not in the sense that roads change (though they do!), but because I worry about regressions and new bugs.

You can mitigate it a bit by simulating many rides in virtual reality--which Waymo is absolutely doing already--but add enough cities and that would get really expensive really quickly, even by Google standards. And you probably couldn't run that world suite of tests for every code submission, or even on a nightly candidate.

This is a solved problem.

You have a neural network which inspects the code change, and predicts which tests/scenarios might be affected (a change in the vision system is unlikely to affect the handbrake tests for example). You run only the most likely to fail tests. A test which hasn't been run for a long time is more likely to fail, since more code changes were done since it was last tested, so all tests end up being run on a semi-random schedule.

When any test fails, you then run back through time, bisecting every revision till you find the code commit which caused the failure.

At that point, depending on your engineering workflow, you either create a ticket to an engineer to fix this commit to pass the test before the next release, or you auto-revert that commit.

A change in the vision system can absolutely affect handbrake tests. As far as we know, last week how Uber detects obstacles changed in a way that affected latency enough that it couldn't brake in time, which is why we now have a broken person instead of a broken test.

An autonomous car is inherently a system, and you need a deep suite of integration tests, regularly run and covering all possible cases, to believe you're even approaching vehicle safety. The wider you space out when any given integration test is run, the harder it becomes to fix. And if a given test only runs once per week, sooner or later (sooner with a sufficiently large test suite) you're going to hit a place where auto-reverting the commit breaks additional tests.

And when that happens, unless you put a global lock on new code entering the system, it might be impossible to fix the regression. And putting a global lock on new code submission will make new feature development extremely slow and expensive.

Is there any real-life system that actually does this? It’s the first time I hear about it... It sounds cool, is anyone actually using something like this in production? For normal apps, I mean.