Came here expecting a Haiku.

The authors who wrote

"Attention is all you need" -

Turing candidates?

The people behind

"Attention is all you need"

Are often cited

Attention.

Attention.

Attention.

- Ikkyū

Neural nets advance,

Attention is all you need,

Computing ascends.

#by chatgpt

The Adam optimizer is another possibility. It's unbelievably good and everyone uses it.

>Will receive Turing Award

This is the weird thing - hopefully not! Hopefully there's even better NN models coming out every 5-10 years and we look back on transformers as 'just a phase' sort of like how we look back at RNN's (which were no less of an amazing achievement - look at the proliferation of voice assistants), as potentially obsolete technology today.

Fore example, attention is great and does a really good job of simulating context in language, but what if we come up with a clever way to simulate symbology? Then we actually are back on the path to AGI and transformers will look like child's play.

> symbology

Off-topic, but now I have William Dafoe going "What's the 'symbology' here? The symbolism ..." in my head (from Boondock Saints).

Even thou I watched that movie 20 years ago. I will never forget that scene.

I remember an interview with one of the founders of openAI, saying that if it wasn't the transformer architecture it would be something else. What really matters is the scale of the model. The transformer is only one of the possible configurations that work well with text. It seems they stuck to it because it is really so good so why break things.

The authors did not really expect it to be such a huge influence. You could also argue, it is a somewhat natural next step. This paper did not invent self-attention nor attention. Attention was already very popular, specifically for machine translation, and a few other papers already did use self-attention at that point in time. It was just the first paper which solely used attention and self-attention and nothing else.

Guy who said - “I don’t understand all of this, can we just throw more machines?” should get the award.

Attention existed before that paper and was incorporated to LSTMs until that point in time.