News for hackers. Stuff that matters.
Re:T5: The Text-to-Text Transfer Transformer by modeless on Monday February 24, 2020 @05:12PM (#22408999)
The trivia game (https://t5-trivia.glitch.me/) needs a little work.

> Q: How did Gus Grissom, Ed White and Roger B. Chaffee die in 1967?

> You: "Apollo 1" WRONG

> T5: "They were killed when their Apollo 1 spacecraft exploded" WRONG

> Correct answer: burned to death

> Q: Which Alpine peak is known in Italy as Monte Cervino?

> You: "Monte Cervino" CORRECT

I wonder how many of the problems with this game could be fixed by applying T5 itself to the answer grading.

Re:T5: The Text-to-Text Transfer Transformer by craffel on Monday February 24, 2020 @05:18PM (#22409033)
Yes, unfortunately we have to rely on the very brittle "exact match" method of evaluating whether an answer is correct. FWIW and perhaps surprisingly, this is the primary way question-answering systems are evaluated in common benchmarks. I totally agree that fine-tuning T5 for answer grading would be super interesting!
Re:T5: The Text-to-Text Transfer Transformer by modeless on Monday February 24, 2020 @05:26PM (#22409093)
I think it makes some sense to evaluate models like this, as you want to be conservative with the answers you accept (though my second example shows that it isn't always conservative), and models don't have feelings to hurt if they are docked points for not being precise enough. Humans, of course, are more sensitive.
Re:T5: The Text-to-Text Transfer Transformer by lsb on Monday February 24, 2020 @07:19PM (#22409793)
Does that mean that answer grading would become like comparing summaries of a given text?
Re:T5: The Text-to-Text Transfer Transformer by dmit on Monday February 24, 2020 @06:44PM (#22409597)
I'm sorry for being blunt, but is it possible that the `very brittle "exact match" method of evaluating whether an answer is correct` means value equality? Is `==` the secret sauce?
Re:T5: The Text-to-Text Transfer Transformer by craffel on Monday February 24, 2020 @06:54PM (#22409653)
It's slightly more than that -- it also involves lowercasing and removing articles before testing for string equality.
Re:T5: The Text-to-Text Transfer Transformer by svnpenn on Monday February 24, 2020 @07:47PM (#22409949)
Why are you replying to every single comment?
Re:T5: The Text-to-Text Transfer Transformer by schoen on Monday February 24, 2020 @07:49PM (#22409959)
I think craffel (probably "Colin Raffel, Senior Research Scientist, Google Research") was directly involved in this research!
Re:T5: The Text-to-Text Transfer Transformer by craffel on Monday February 24, 2020 @07:52PM (#22409975)
Yes, that's me! Sorry if I'm being overeager, I like talking about my research!
Re:T5: The Text-to-Text Transfer Transformer by schoen on Monday February 24, 2020 @08:00PM (#22410026)
I think it's amazing how frequently people involved in various CS and IT things are directly participating in threads about their work here on HN.
Re:T5: The Text-to-Text Transfer Transformer by Someone on Tuesday February 25, 2020 @02:36AM (#22411499)
”Correct answer: burned to death”

More likely: suffocated (as with most fire deaths) https://history.nasa.gov/Apollo204/invest.html:

”d. MEDICAL ANALYSIS

Loss of consciousness was due to cerebral hypoxia due to cardiac arrest resulting from myocardial hypoxia. Factors of temperature, pressure and environmental concentrations of carbon monoxide, carbon dioxide, oxygen and pulmonary irritants were changing extremely rapidly. It is impossible to integrate these variables on the basis of available information with the dynamic physiological and metabolic conditions they produced, in order to arrive at a precise statement of time when consciousness was lost and when death supervened. The combined effect of these environmental factors dramatically increased the lethal effect of any factor by itself. It is estimated that consciousness was lost between 15 and 30 seconds after the first suit failed. Chances of resuscitation decreased rapidly thereafter and were irrevocably lost within 4 minutes.”

Re:T5: The Text-to-Text Transfer Transformer by markdog12 on Monday February 24, 2020 @07:05PM (#22409726)
Yeah, it asked who was the director of the CIA from 1976-1981. I answered "George Bush" (lazy, typing on phone), it answered the more exact "George H. W. Bush". It marked me correct, but it was marked wrong. It also asked me what country the island of "Honsh" was in (should be Honshu). I answered Japan and was correct, though. It guessed Iceland.
Re:T5: The Text-to-Text Transfer Transformer by kazinator on Monday February 24, 2020 @07:12PM (#22409755)
TIL: Leonardo Di Caprio, not Robert Redford, played Jay Gatsby in a 1974 adaptation of The Great Gatsby.
Re:T5: The Text-to-Text Transfer Transformer by kps on Monday February 24, 2020 @06:29PM (#22409511)
Reminds me of the early Unix ‘quiz’ game.