Formal Algorithms for Transformers

Re:Formal Algorithms for Transformers by tartakovsky on Thursday July 21, 2022 @10:09AM (#32179602)

Zero diagrams, but maybe they wouldn’t be helpful to clarify the concept? Guess it depends on the types of learners, I’m not sure.

[ Parent | on HN ]

Re:Formal Algorithms for Transformers by nh23423fefe on Thursday July 21, 2022 @12:33PM (#32181415)

paper explicitly rejects diagrams as unhelpful

> Some 100+ page papers contain only a few lines of prose informally describing the model [RBC+21]. At best there are some high-level diagrams

[ Parent | on HN ]

Re:Formal Algorithms for Transformers by godelski on Thursday July 21, 2022 @02:26PM (#32182936)

> only a few lines of prose informally describing the model

This is ironic considering they use more words to describe chunking (splitting along a dimension, x,y = a[0,:,:,:], a[1,:,:,:]) than multi-heads.

[ Parent | on HN ]