News for hackers. Stuff that matters.

A comment on Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline. Parent comment

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by _II__II_ on Thursday October 24, 2019 @10:34PM (#21351303)
The example directly below that: "Justify the margins" and "The end justifies the means" is the one I find dubious. Obviously the former could mean to format a document, but those exact words in that structure could be a demand for someone to justify a financial margin for example. It is both true and false depending on the context.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by andrewstuart2 on Thursday October 24, 2019 @10:46PM (#21351371)
One of my favorite examples that I heard in a David Rock talk which I can no longer find on youtube: "Time flies like an arrow":

Time moves swiftly and in one direction.

Record the speed of flies in the same way you would an arrow.

Time flies, which are a kind of fly, are fond of an arrow. (e.g. Time flies like an arrow, fruit flies like a banana).

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by QuinnWilton on Thursday October 24, 2019 @11:47PM (#21351685)
It sounds like you're talking about garden-path sentences [0], and in particular: "time flies like an arrow; fruit flies like a banana" [1]. These are sentences whose structure tricks the reader into making an incorrect parse. My favourite of these has always been: "The horse raced past the barn fell".

[0] https://en.wikipedia.org/wiki/Garden-path_sentence

[1] https://en.wikipedia.org/wiki/Time_flies_like_an_arrow;_frui...

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by lsb on Friday October 25, 2019 @01:43AM (#21352210)
I've always enjoyed the multiple valid parses of "Time flies like an arrow". I can't wait for AI to generate more Escher sentences like "More people have been to Russia than I have" ( https://en.m.wikipedia.org/wiki/Comparative_illusion )
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by Doxin on Friday October 25, 2019 @03:33AM (#21352718)
You know, I only just now got the second interpretation of that sentence. I always thought of it like "Time flies like an arrow (straight and in one direction), Fruit flies like a banana (when thrown)"

Obvious in hindsight...

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by kees99 on Friday October 25, 2019 @04:47AM (#21353048)
Same here, except it's comparing fly's flight trajectory to that of a banana is new to me.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by saalweachter on Friday October 25, 2019 @10:02AM (#21355013)
"The horse raced past the barn fell, which has been haunted since all those teenagers were murdered there."

(Noun-adjective is a rare formation, but amusingly more common in the same situations where the author uses rare and archaic definitions like the adjective "fell".)

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by changoplatanero on Thursday October 24, 2019 @11:11PM (#21351516)
"I eat my rice with butter." could mean that you use butter as a utensil to eat your rice with. There is often an unlikely way of parsing the sentence that gives an alternate meaning. The point is to test the computer to see if it can distinguish the likely parse from an unlikely one.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by nl on Thursday October 24, 2019 @11:29PM (#21351613)
These aren't really alternate _parses_ though (in the sense that they don't give different parse trees). They do highlight the different possible meanings of "with" though.

I think "I eat my rice with chicken" vs "I eat my rice with children" vs "I eat my rice with chopsticks" is the canonical example here.

There's a whole field in NLP involved in showing what changes happen to entities mentioned in a sentence as a a side effect of the sentence, and this example shows it pretty well.

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by changoplatanero on Friday October 25, 2019 @02:27AM (#21352433)
Wouldnt those be different parse trees? Like the "with X" could either be attached either to the verb or the noun
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by lultimouomo on Friday October 25, 2019 @02:59AM (#21352586)
I think it's more clear if you say "I usually eat X with Y", i.e. Y it's either the company, the tool or the condiment that you eat with (contrasted with "I'm eating my X", where X is a dish like "rice with chicken")
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by nl on Friday October 25, 2019 @02:56AM (#21352574)
Yes, possibly.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by ekianjo on Friday October 25, 2019 @02:33AM (#21352468)
A good demonstration that context (and cultural conditioning) is everything to understand what a text actually means.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by otakucode on Friday October 25, 2019 @08:38AM (#21354344)
Not to mention something that almost all NLP systems are resounding terrible at - short-term memory. If we've been talking about corporate financials for an hour and I say 'Justify the margins', it should be crystal clear what I mean. But most automated systems try to operate without a hint of memory or 'state' being tracked.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by nebulous1 on Friday October 25, 2019 @07:19AM (#21353843)
I'm guessing this is intentional. To a human, although this could be somebody being asked to justify their financial margins that's not a very likely answer. The human can easily see that, while it's possible they're the same meaning, given the lack of any other context the answer is that they're not.

The enemy could have landed several of our aircraft on one of their runways. Agassi may have beaten Becker over the head with his tennis racket. I suspect part of the test is that there can be other meanings that do technically work.

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by OrangeMango on Friday October 25, 2019 @09:36AM (#21354746)
> The enemy could have landed several of our aircraft on one of their runways.

This is something that actually does happen. Less than 10 or 20 years ago, China did it to an US Air Force reconnaissance aircraft.

[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by hn_throwaway_99 on Friday October 25, 2019 @09:42AM (#21354815)
This is a good point I hadn't thought of. Honestly, I'm really not surprised anymore that the humans only scored 89%.
[ Parent | on HN ]
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by sjg007 on Friday October 25, 2019 @04:19AM (#21352906)
The ends justify the means.
[ Parent | on HN ]