News for hackers. Stuff that matters.
Re:Pre-Trained Large Language Models Use Fourier Features to... by wongarsu on Thursday February 06, 2025 @10:45AM (#42963461)
Curious that they chose to use GPT-2-XL, given the age of that model. I guess they wanted a small model (1.5B) and started work on this quite a while ago. Today there is a decent selection of much more capable 1.5B models (Quen2-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, TinySwallow 1.5B, Stella 1.5B, Qwen2.5-Math-1.5B, etc). But they are all derived from the Qwen series of models, which wasn't available when they started this research.
Re:Pre-Trained Large Language Models Use Fourier Features to... by scoresmoke on Thursday February 06, 2025 @11:11AM (#42963766)
GPT-2 follows the very well-studied architecture of Transformer decoder, so the outcomes of this study might be applicable to the more complicated models.
Re:Pre-Trained Large Language Models Use Fourier Features to... by imjonse on Thursday February 06, 2025 @11:27AM (#42963962)
The paper predates Qwen2 and R1, this work is probably a year old.
Re:Pre-Trained Large Language Models Use Fourier Features to... by ImHereToVote on Thursday February 06, 2025 @11:07AM (#42963711)
I understand GPT-2 has been somewhat mapped to a certain extent.
Re:Pre-Trained Large Language Models Use Fourier Features to... by nickpsecurity on Thursday February 06, 2025 @11:05AM (#42963687)
I was collecting examples of models trained on single GPU or with very low cost. A number of projects used Bert or GPT to since the implementations were very simple with with some components optimized. There’s also a lot of projects that have trained Bert in GPT two models which make for more scientific comparisons.

With no other information, those would be my guesses as to why one would use a GPT2 model.

Re:Pre-Trained Large Language Models Use Fourier Features to... by littlestymaar on Thursday February 06, 2025 @11:04AM (#42963681)
TinyLlama would have worked too and is older than the Qwen family.
Re:Pre-Trained Large Language Models Use Fourier Features to... by Sharlin on Thursday February 06, 2025 @12:47PM (#42964688)
You can think of GPT-2 as the D. melanogaster of language models.