News for hackers. Stuff that matters.

A comment on T5: The Text-to-Text Transfer Transformer. Parent comment

Re:T5: The Text-to-Text Transfer Transformer by craffel on Monday February 24, 2020 @05:18PM (#22409033)
Yes, unfortunately we have to rely on the very brittle "exact match" method of evaluating whether an answer is correct. FWIW and perhaps surprisingly, this is the primary way question-answering systems are evaluated in common benchmarks. I totally agree that fine-tuning T5 for answer grading would be super interesting!
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by modeless on Monday February 24, 2020 @05:26PM (#22409093)
I think it makes some sense to evaluate models like this, as you want to be conservative with the answers you accept (though my second example shows that it isn't always conservative), and models don't have feelings to hurt if they are docked points for not being precise enough. Humans, of course, are more sensitive.
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by lsb on Monday February 24, 2020 @07:19PM (#22409793)
Does that mean that answer grading would become like comparing summaries of a given text?
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by dmit on Monday February 24, 2020 @06:44PM (#22409597)
I'm sorry for being blunt, but is it possible that the `very brittle "exact match" method of evaluating whether an answer is correct` means value equality? Is `==` the secret sauce?
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by craffel on Monday February 24, 2020 @06:54PM (#22409653)
It's slightly more than that -- it also involves lowercasing and removing articles before testing for string equality.
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by svnpenn on Monday February 24, 2020 @07:47PM (#22409949)
Why are you replying to every single comment?
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by schoen on Monday February 24, 2020 @07:49PM (#22409959)
I think craffel (probably "Colin Raffel, Senior Research Scientist, Google Research") was directly involved in this research!
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by craffel on Monday February 24, 2020 @07:52PM (#22409975)
Yes, that's me! Sorry if I'm being overeager, I like talking about my research!
[ Parent | on HN ]
Re:T5: The Text-to-Text Transfer Transformer by schoen on Monday February 24, 2020 @08:00PM (#22410026)
I think it's amazing how frequently people involved in various CS and IT things are directly participating in threads about their work here on HN.
[ Parent | on HN ]