I made a transformer by hand (no training)

Re:I made a transformer by hand (no training) by teddykoker on Friday September 22, 2023 @01:43PM (#37615122)

A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components, and demonstrate how different programs can be written with it, e.g. histograms, sorting. Sasha Rush and Gail Weiss have an excellent blog post on it as well [2]. Follow on work actually demonstrated how RASP-like programs could actually be compiled into model weights without training [3].

[1] https://arxiv.org/abs/2106.06981

[2] https://srush.github.io/raspy/

[3] https://arxiv.org/abs/2301.05062

[ Parent | on HN ]

Re:I made a transformer by hand (no training) by newhouseb on Friday September 22, 2023 @06:04PM (#37618386)

Huge fan of RASP et al. If you enjoy this space, might be fun to take a glance at some of my work on HandCrafted Transformers [1] wherein I hand-pick the weights in a transformer model to do long-handed addition similar to how humans learn to do it in gradeshcool.

[1] https://colab.research.google.com/github/newhouseb/handcraft...

[ Parent | on HN ]

Re:I made a transformer by hand (no training) by theGnuMe on Friday September 22, 2023 @09:13PM (#37619761)

It seems like a functional language like Haskell would be the right tool for this.

Also going from a net to code would be super interesting in terms of explain-ability.

[ Parent | on HN ]