News for hackers. Stuff that matters.
Re:Memorizing Transformers by blackbear_ on Friday May 20, 2022 @12:02PM (#31449259)
> On benchmarks including code and mathematics, we find that the model is capable of making use of newly defined functions and theorems during test time.

Train on test, improved performance on test. Wow.

Re:Memorizing Transformers by visarga on Friday May 20, 2022 @12:58PM (#31450020)
> Wow.

Transformers are very limited in the size of the attention window. They can take a few thousand tokens at maximum. But your data might not fit into the window, and you also don't want to have to fine-tune the model. This paper offers a solution.

Re:Memorizing Transformers by spullara on Friday May 20, 2022 @01:45PM (#31450501)
It isn't being trained on test. Kind of the point of memory is that you can change the memory at will and don't need to train on new information you have never seen before.