News for hackers. Stuff that matters.
Re:Rethinking Attention with Performers by mrfusion on Monday October 26, 2020 @10:19AM (#24896411)
Every time I research transformers it seems so hand wavy. Is there a simple description, maybe a bit of pseudo code?

Or at the other extreme they dump me into formula land without exposing what all the letters in the formula represent.

Re:Rethinking Attention with Performers by silveraxe93 on Monday October 26, 2020 @10:27AM (#24896490)
This is quite a good explanation of transformers that gets shared a lot. [link](http://jalammar.github.io/illustrated-transformer/)

And here's a super simple implementation of GPT by Andrej Karpathy. [link](https://github.com/karpathy/minGPT/blob/master/mingpt/model....)

Re:Rethinking Attention with Performers by sjg007 on Monday October 26, 2020 @06:47PM (#24902405)
Transformers are kinda similar to state vectors. They are tracking the current state of the world. The input becomes the output which is the input to the next iteration. The transformer transform the input to the output ad infinitum until a stop token is reached.