News for hackers. Stuff that matters.

A comment on Formal Algorithms for Transformers. Parent comment

Re:Formal Algorithms for Transformers by nh23423fefe on Thursday July 21, 2022 @12:33PM (#32181415)
paper explicitly rejects diagrams as unhelpful

> Some 100+ page papers contain only a few lines of prose informally describing the model [RBC+21]. At best there are some high-level diagrams

[ Parent | on HN ]
Re:Formal Algorithms for Transformers by godelski on Thursday July 21, 2022 @02:26PM (#32182936)
> only a few lines of prose informally describing the model

This is ironic considering they use more words to describe chunking (splitting along a dimension, x,y = a[0,:,:,:], a[1,:,:,:]) than multi-heads.

[ Parent | on HN ]