Re:AlphaFold: a solution to a 50-year-old grand challenge in...
by klmadfejno on Monday November 30, 2020 @03:12PM
(#25258347)
I find this disingenuous. Yes, its important that the algos can perform well on real world data, but the framing of this post begins with an anecodote about one person who had a bad model, and implicitly extrapolates that these problems are generalized throughout all neural nets.
One could say the same thing about programmers automating a task, or a number of other trivial examples. I would lean towards assuming deep mind has competent model validation teams vs. not, even if data science is hard.
Re:AlphaFold: a solution to a 50-year-old grand challenge in...
by tuatoru on Monday November 30, 2020 @01:17PM
(#25256977)
I agree. The failures have to be explicable if we are to trust a model.
Re:AlphaFold: a solution to a 50-year-old grand challenge in...
by asah on Monday November 30, 2020 @02:33PM
(#25257890)
Doesn't it depends on the application ? i.e. some applications can tolerate false positives/negatives ?
Re:AlphaFold: a solution to a 50-year-old grand challenge in...
by baybal2 on Monday November 30, 2020 @03:03PM
(#25258235)
May well be, but if you spend more compute, and human time checking for those corner cases than if you went with another, more consistent exhaustive search algorith, then the method looses to it economically.
This is more the case the more close to bruteforce you come, like encryption cracking. Imagine, spending years of HPC cluster time, trying to break a password, while knowing you have a single digit chance to miss the right key, in a way which would be completely impossible with with a conventional solution.
They always had single digit, bizarre artifacts, where the program can't sometimes recognise the very data it was trained on with most minute differences.
Other artifact was that the most "stereotypical cases" were least reliably recognised, and they hot a lot of flak for screwed up live demos, where a radiologist put a very, very obvious tumor shot onto the scanner, and it didn't work without a half an hour of wiggling the film, and a camera.
The "bruteforce" solution may well be always, 80-85% off, but off consistently, and always. NN algo so far beat them, but fail with double digit frequencies on "artifacts" which they themselves can't do anything about.
How well it deals with the later, is what I believe will measure its real world usefullness.