News for hackers. Stuff that matters.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by 6gvONxR4sf7o on Thursday October 24, 2019 @10:34PM (#21351304)
One thing to always point out in these cases is that the human baseline isn't "how well people do at this task," like it's often hyped to be. It's "how well does a person quickly and repetitively doing this do, on average." The 'quickly and repetitively' part is important because we all make more boneheaded errors in this scenario. The 'on average' part is important because the errors the algo makes aren't just fewer than people, they're different. The algos often still get certain things wrong that humans almost never would.

This is really really super great, let's be clear. It's just not up to the hype "omg super human" usually gets.

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by TheOtherHobbes on Friday October 25, 2019 @03:57AM (#21352801)
It seems to mean "How well does Mechanical Turk do the task?" which is a separate thing again. And yes - error type is at least as revealing as error frequency.

I have no idea where the real human baseline is, or how to find it.

Also, consider this discussion. GLUE winners may be able to make informed parsing guesses about single text blocks, but they're years away from being able to make a useful contribution to a discussion like this one.

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by IshKebab on Friday October 25, 2019 @04:51AM (#21353071)
Regarding the type of errors, it seems like the benchmark should be able to take that into account. That is, get a load of humans to do the task on the same specific examples, then for each example you know how hard it is, and what acceptable answers are (I bet a lot of the ground truth is wrong or ambiguous).

Then you can benchmark your AI but penalise it more heavily for getting things wrong that are obvious to a human.

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by 6gvONxR4sf7o on Friday October 25, 2019 @11:12AM (#21355786)
That would be ideal, if money weren't a factor. Since money is a factor, I wonder what the tradeoff is between labelling each instance N more times versus just getting N times more instances labeled.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by Pahr3yah on Friday October 25, 2019 @06:50AM (#21353676)
In the context of GPT2 someone coined the expression "Humans Who Are Not Concentrating Are Not General Intelligences"
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by The_Amp_Walrus on Friday October 25, 2019 @07:59AM (#21354088)
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by jcims on Thursday October 24, 2019 @11:04PM (#21351465)
Great point! It makes sense in the context of what these algorithms would generally be tasked with.