Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching...
by TheOtherHobbes on Friday October 25, 2019 @03:57AM
(#21352801)
It seems to mean "How well does Mechanical Turk do the task?" which is a separate thing again. And yes - error type is at least as revealing as error frequency.
I have no idea where the real human baseline is, or how to find it.
Also, consider this discussion. GLUE winners may be able to make informed parsing guesses about single text blocks, but they're years away from being able to make a useful contribution to a discussion like this one.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching...
by IshKebab on Friday October 25, 2019 @04:51AM
(#21353071)
Regarding the type of errors, it seems like the benchmark should be able to take that into account. That is, get a load of humans to do the task on the same specific examples, then for each example you know how hard it is, and what acceptable answers are (I bet a lot of the ground truth is wrong or ambiguous).
Then you can benchmark your AI but penalise it more heavily for getting things wrong that are obvious to a human.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching...
by 6gvONxR4sf7o on Friday October 25, 2019 @11:12AM
(#21355786)
That would be ideal, if money weren't a factor. Since money is a factor, I wonder what the tradeoff is between labelling each instance N more times versus just getting N times more instances labeled.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching...
by Pahr3yah on Friday October 25, 2019 @06:50AM
(#21353676)
In the context of GPT2 someone coined the expression "Humans Who Are Not Concentrating Are Not General Intelligences"
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching...
by The_Amp_Walrus on Friday October 25, 2019 @07:59AM
(#21354088)
I think it was this blogger: https://www.google.com/amp/s/srconstantin.wordpress.com/2019...
This is really really super great, let's be clear. It's just not up to the hype "omg super human" usually gets.