> Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.
1. It's a paper from 2017. Unless you follow academic ML research, you will not have heard of it.
2. That paper's title is also inscrutable unless you've gone and read at least the abstract.
Edit: the linked pytorch implementation looks interesting, these layer types promise pretty incredible things https://github.com/ml-jku/hopfield-layers
In any event, the authors definitely chose that title as a callback to the well known paper "Attention is all you need", which introduced Transformers. So that probably influenced their decision to use "is" instead of "are".
I bet ‘is’ sounds better to you in this context, though ‘my team’ and ‘The Patriots’ are similar noun phrases that could refer to exactly the same thing.
The difference is that Patriots is a plural. Replace it with Manchester United and ‘is’ sounds good again.
This seems like a strange discrepancy. Why is this the case? Maybe it is because "favorite team" is clearly singular, and is closer in the sentence to the "is"/"are" then the plural indicating sound in "Red Sox". Or maybe it is just whichever comes first which determines how the "to be" is conjugated?
Hm, but what if instead of connecting a noun phrase (determiner phrase?) like "The Red Sox" to another noun phrase (determiner phrase) "John's favorite team", we instead connect it to an adjective?
"The Red Sox are singular.", "The Red Sox is singular.", "Singular is The Red Sox." "Singular are The Red Sox." . Well, the "[Adjective] is [noun]" is kind of an unusual thing to say unless one is trying to sound like one is quoting poetry or yoda or something, but to the degree that either of them sound ok, I think "Singular are The Red Sox." sounds better than "Singular is The Red Sox." . Though, in this case, there doesn't seem to be anything grammatically suggested by the adjective that the thing be in the singular case (maybe I shouldn't have used "singular" as the adjective..) .
Hm, what if instead of "John's favorite team [is/are] the Red Sox." , we instead look at "John's favorite [is/are] the Red Sox." ? In this case, it seems, less clear which is more natural? They seem about the same to me (but that might just be me, idk.) .
Anyway : Weird!
I actually think this could extend to a lot of situations where the object is referring to a single group, not just plural-sounding proper nouns. Like if asked "what was your favorite zoo exhibit?", I would probably respond "my favorite was the giraffes" not "my favorite were the giraffes". I'm actually not even sure what the correct response would be technically though. "My favorites were the giraffes" implies multiple favorites, and "my favorite was the giraffe" makes it sound like the exhibit had a single giraffe. So it feels like subject/object have to mismatch then.
In English: It's only five minutes to the bus stop.
In German: Es sind nur fünf Minuten zur Bushaltestelle. (It are only five minutes to the bus stop.)
I think it's a question of whether the verb is supposed to agree with the subject or the complement.
interesting. to my (non-native) ears, the second sounds more natural. Wonder how common preference for each of those is.
1. Hopfield Networks are also known as "associative memory networks", a neural network model developed decades ago by a guy named Hopfield.
2. It's useful to plug these in somehow as layers in Deep Neural Networks today (particularly, in PyTorch).
I hate non-informative titles!