Is there any major app (say, at least 1 million SLOC) which has such good automated tests that they're comfortable shipping master without any manual QA? I'd be really surprised if there is.
Even if a team is totally committed to automating all their testing, there are some big challenges with doing so at scale:
- You run into all kinds of rare bugs (with AOSP, the emulator, host OS, etc.). When I worked at Square our biggest app had ~5 hours of automated tests, and these rare bugs made it very hard to keep our builds stable. I imagine thoroughly testing AOSP would take an order of magnitude longer.
- If you want to test on a variety real phones (which seems critical for AOSP), then testing every master commit is cost prohibitive.
- It's hard to automatically detect graphical glitches. You can take screenshots at certain times and validate that they're pixel perfect, but you might still miss a glitch in between screenshots, e.g. during an animation.
I wouldn't expect master to be stable, for Android or any other project. If you were setting up a new Linux machine and compiling the kernel yourself, surely you wouldn't build master? AOSP has release branches and RCs like android-7.1.1_r38 which will have gone though a bunch of QA.
Most users wouldn't like stock AOSP though, since the built-in apps are very primitive.
(I'm at Google, but don't work on Android.)