News for hackers. Stuff that matters.

A comment on Understanding and Coding the Self-Attention Mechanism of Large Language Models. Parent comment

Re:Understanding and Coding the Self-Attention Mechanism of ... by sooheon on Friday February 10, 2023 @11:28PM (#34749579)
Good explanation. Which is why the success of transformers, LLMs etc. is still not the final word in Rich Sutton's "The Bitter Lesson" -- no learning method is free of inductive biases.
[ Parent | on HN ]
Re:Understanding and Coding the Self-Attention Mechanism of ... by candiodari on Saturday February 11, 2023 @05:47PM (#34757227)
Inductive biases can work even if they're wrong, because they allow for simple and quick action, simpler reasoning. They don't need to be correct for that to pay off, they just need a positive expected value.
[ Parent | on HN ]