Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Differential Transformer. Parent comment
Hah. Yes. It looks like they only show up in models with 6.7B parameters or more.
The problem can start at 125M. Small enough to test on a whim.
So train a model that exhibits these behaviours, then try it out.
There are no comments on this comment yet.
Hah. Yes. It looks like they only show up in models with 6.7B parameters or more.
The problem can start at 125M. Small enough to test on a whim.
So train a model that exhibits these behaviours, then try it out.