Also, Tomas says he came up with the encoder-decoder (seq-to-seq) idea, and then Ilya and Quoc took over the idea after Tomas moved on to Facebook.

However, there is another statement by Quoc, saying this is not true: https://twitter.com/quocleix/status/1736523075943125029

> We congratulate Tomas on winning the award. Regarding seq2seq, there are inaccuracies in his account. In particular, we all recall very specifically that he did not suggest the idea to us, and was in fact highly skeptical when we shared the end-to-end translation idea with him. Indeed, we worked very hard to make it work despite his skepticism.

So, word against word. I'm not accusing anyone of lying here, one of them probably just misremembers, but this leaves also a somewhat bad taste.

I think this must happen all the time. As they say, ideas are cheap. It's likely that ALL of them had the seq-to-seq idea cross their mind at some point before it was acted on, so if credit is assigned to whoever said it out loud first, there's going to be disagreement, since most people don't remember the full details of every conversation. It's also possible for someone to be skeptical of their own idea, so that argument isn't compelling to me either. Ultimately credit usually goes to the people who do the hard work to prove out the idea, so it seems like the system worked as intended in this case.

This is what Tomas Mikolov said on Facebook:

> I wanted to popularize neural language models by improving Google Translate. I did start collaboration with Franz Och and his team, during which time I proposed a couple of models that could either complement the phrase-based machine translation, or even replace it. I came up (actually even before joining Google) with a really simple idea to do end-to-end translation by training a neural language model on pairs of sentences (say French - English), and then use the generation mode to produce translation after seeing the first sentence. It worked great on short sentences, but not so much on the longer ones. I discussed this project many times with others in Google Brain - mainly Quoc and Ilya - who took over this project after I moved to Facebook AI. I was quite negatively surprised when they ended up publishing my idea under now famous name "sequence to sequence" where not only I was not mentioned as a co-author, but in fact my former friends forgot to mention me also in the long Acknowledgement section, where they thanked personally pretty much every single person in Google Brain except me. This was the time when money started flowing massively into AI and every idea was worth gold. It was sad to see the deep learning community quickly turn into some sort of Game of Thrones. Money and power certainly corrupts people...

Reddit post: "Tomas Mikolov is the true father of sequence-to-sequence" https://www.reddit.com/r/MachineLearning/comments/18jzxpf/d_...

As another small hint of Mikolov-vs-Le divergence: they're the coauthors of the 'Paragraph Vector' paper (https://arxiv.org/abs/1405.4053) applying a slightly-modified version of word2vec to vectorize longer texts, still in a very shallow way. (This technique often goes by the name 'doc2vec', but other things also sometimes get called that, too.)

There are some results in that paper, supposedly from exactly the technique described on an open dataset, that no one has ever been able to reproduce – & you can see the effort has frustrated a lot of people, over the years, in different forums.

When asked, Mikolov has said, essentially: "I can't reproduce that either – those tests were run & reported by Le, you'lll have to ask him."

This is interesting. I went off and searched for paragraph vector code and indeed find doc2vec stuff, including tutorials referring to the paper such as https://radimrehurek.com/gensim/auto_examples/howtos/run_doc.... It’s not obvious that the results aren’t reproducible (and I realise code is not the same as published results), but I wonder if you could steer us more specifically.

As I understand it, no one has come close to the claimed results in s3.1 ("Sentiment Analysis with the Stanford Sentiment Treebank Dataset") and people have come closer but still not matched those in s3.2 ("Beyond One Sentence: Sentiment Analysis with IMDB dataset").

A thread where Mikolov is trying to help other with his patch to `word2vec.c` demoing a tiny bit of 'Paragraph Vector' – but reaches the limits of what he understands Le to have done – is: https://groups.google.com/g/word2vec-toolkit/c/Q49FIrNOQRo/m...

My own frustration (& reco to avoid the thin stanfordSentimentTreebank/RottenTomatoes data/results) is mentioned at: https://groups.google.com/g/word2vec-toolkit/c/ubFrO0a9Pe8/m...

I'd say that "concatenating PV-DBOW and (plain averaging) PV-DM" never seems to offer much lift compared the favorable way it's described in the paper and other Le comments. And after spending a bunch of time implementing the "PV-DM with concatenation (rather than sum/average) of many word-vectors as the context", as I interpret the paper's description "To predict the 8-th word, we concatenate the paragraph vectors and 7 word vectors", I've only seen it massively increase model size & training time for very little advantage.

Reddit searches on related topics turn up other anecdotes/resentments; eg:

https://www.reddit.com/r/MachineLearning/comments/18jzxpf/co...

https://www.reddit.com/r/MachineLearning/comments/hkiyir/com...

At a certain level, with so much "gold in them thar hills" in followup work – from both academic & commercial perspectives, I don't blame Le for rushing forward to other related fertile ideas, & ignoring the requests-for-explanation.

But there's something sloppy or fishy (hiding secret tweaks?) in the originally-claimed PV results, which has wasted a lot of time among those trying to understand & reproduce.

Typical, saying they had the idea first without putting it on the blockchain to prove the time stamp!

"Success has a thousand mothers, but failure is an orphan"

> Indeed, we worked very hard to make it work despite his skepticism.

In this case why the authors of the seq2seq paper did not mention him in their massive acknowledgement section?

To be fair I have some memories of the papers and surrounding tech being really bad. The popular implementations didn’t actually do what the papers said and the tech wasn’t great for anything beyond word level comparisons. You got some juice doing tf idf weighting of specific words but then tf idf weighted bag of words was similarly powerful.

Cosine similarity of the sum of different word vectors sounds soooo dumb nowadays imo

Cosine similarity is like a very primitive version of word2vec. They both work off the idea of context being the key to the derivation of a token's semantic meaning. Cosine similarity is not very useful in itself but it's helpful for understanding the progression in the space

I tried asking on another thread what Goodfellow rage he's referring to since all I could find was this: https://twitter.com/goodfellow_ian/status/113352818965167718...

If so, frankly I think it makes Mikolov sound pretty insecure.

Twitter no longer shows threads to people who aren't logged in. https://nitter.net/goodfellow_ian/status/1133528189651677184

That post sounds like a rant TBH with too many stabs at various people. It could have been a lot more graceful. OTOH I can believe most researchers are human and do not put the progress of shared knowledge first but are very much influenced by ego and money *cough* OpenAI *cough*

To err is human, to seek profit is common to all lifeforms

I wrote a detailed proof years before Mikolov on Twitter, but the 280 characters were too small to contain it

Did you now?? I'll have you know that I wrote the full word2vec paper on a roll of shabby two-ply tissue paper during my time in a Taco Bell stall. Sadly, it was then used to mop up my dietary regrets and was subsequently lost to foul wretches of the sewage system. Left with nothing but the memories of my groundbreaking thoughts and the lingering aroma of liquid feces, I texted Mikolov an idea I had about using neural nets to map sequences of text tokens from one language to another only for him to reply "lol thx" and ghost me. I was quite negatively surprised when he decided to take this to the public courts of Facebook and failed to mention the "brainest boi alive™" who gave him this idea in the first place.

The post would have found a more fitting home on Twitter rather than the 'forgotten' realms of FB. The very individuals and entities mentioned (or hinted at) might have had the chance to see the story and share their perspectives. Otherwise it is just sounds like a rant.