Re:Android devices can be fatally hacked by malicious Wi-Fi ...
by dlubarov on Friday April 07, 2017 @03:02PM
(#14062560)
The platform team does do code review and automated tests, but occasionally something slips through, so they QA release branches as well.
Is there any major app (say, at least 1 million SLOC) which has such good automated tests that they're comfortable shipping master without any manual QA? I'd be really surprised if there is.
Even if a team is totally committed to automating all their testing, there are some big challenges with doing so at scale:
- You run into all kinds of rare bugs (with AOSP, the emulator, host OS, etc.). When I worked at Square our biggest app had ~5 hours of automated tests, and these rare bugs made it very hard to keep our builds stable. I imagine thoroughly testing AOSP would take an order of magnitude longer.
- If you want to test on a variety real phones (which seems critical for AOSP), then testing every master commit is cost prohibitive.
- It's hard to automatically detect graphical glitches. You can take screenshots at certain times and validate that they're pixel perfect, but you might still miss a glitch in between screenshots, e.g. during an animation.
Is there any major app (say, at least 1 million SLOC) which has such good automated tests that they're comfortable shipping master without any manual QA? I'd be really surprised if there is.
Even if a team is totally committed to automating all their testing, there are some big challenges with doing so at scale:
- You run into all kinds of rare bugs (with AOSP, the emulator, host OS, etc.). When I worked at Square our biggest app had ~5 hours of automated tests, and these rare bugs made it very hard to keep our builds stable. I imagine thoroughly testing AOSP would take an order of magnitude longer.
- If you want to test on a variety real phones (which seems critical for AOSP), then testing every master commit is cost prohibitive.
- It's hard to automatically detect graphical glitches. You can take screenshots at certain times and validate that they're pixel perfect, but you might still miss a glitch in between screenshots, e.g. during an animation.