News for hackers. Stuff that matters.
Re:Understanding Transformers via N-gram Statistics by pona-a on Sunday May 18, 2025 @03:35PM (#44023795)
I don't have a list, but another popular one was this [0]. They trained a one layer attention-only transformer and could extract its weights as bigrams and skip-trigrams ("A… B C").

[0] https://transformer-circuits.pub/2021/framework/index.html

There are no comments on this comment yet.