News for hackers. Stuff that matters.

A comment on Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline. Parent comment

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by IshKebab on Friday October 25, 2019 @04:51AM (#21353071)
Regarding the type of errors, it seems like the benchmark should be able to take that into account. That is, get a load of humans to do the task on the same specific examples, then for each example you know how hard it is, and what acceptable answers are (I bet a lot of the ground truth is wrong or ambiguous).

Then you can benchmark your AI but penalise it more heavily for getting things wrong that are obvious to a human.

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by 6gvONxR4sf7o on Friday October 25, 2019 @11:12AM (#21355786)
That would be ideal, if money weren't a factor. Since money is a factor, I wonder what the tradeoff is between labelling each instance N more times versus just getting N times more instances labeled.
[ Parent | on HN ]