News for hackers. Stuff that matters.

A comment on Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper. Parent comment

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by meowkit on Wednesday May 17, 2023 @02:49PM (#35979557)
Going to go out on a limb and say they are probably referring to the gradient calculus required for updating the model.

https://en.wikipedia.org/wiki/Differentiable_programming

See automatic differentiation.

[ Parent | on HN ]
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by throwawaymaths on Wednesday May 17, 2023 @02:58PM (#35979662)
Correct, but note that if you subject a standard hash table algo to AD it won't magically become a transformer. (Hashes in the "normal construction" are discrete functions and thus aren't really continuous or differentiable, neither are lookup tables)
[ Parent | on HN ]