News for hackers. Stuff that matters.

A comment on Phi-2: Self-Extend Boosts Performance, Extends Context to 8k Without Training. Parent comment

Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by eightysixfour on Friday January 12, 2024 @11:33AM (#38970133)
I feel like Phi-2 is all about the context of the fact that it is a <3b parameter model. It is incredible in that context, it isn't incredible in others.
[ Parent | on HN ]
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by make3 on Friday January 12, 2024 @12:02PM (#38970654)
it's definitely what it's about. it's insanely strong for its size
[ Parent | on HN ]
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by yufeng66 on Friday January 12, 2024 @01:17PM (#38971749)
Phi-2 basically demonstrated that you don't need a very large model to figure out language. It not very smart but speaks perfect English. it's not obvious the best way to gain IQ is to have a larger language model. some other structure might be needed.
[ Parent | on HN ]
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by behnamoh on Friday January 12, 2024 @01:52PM (#38972248)
But isn't that something that even smaller GPT-2 models demonstrated already?
[ Parent | on HN ]
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by erichocean on Saturday January 13, 2024 @01:25AM (#38977781)
No, and this is covered in the various Phi papers, as well as TinyStories [0]:

> Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training.

[0] https://arxiv.org/abs/2305.07759

[ Parent | on HN ]