News for hackers. Stuff that matters.

A comment on It looks like GPT-4-32k is rolling out. Parent comment

Re:It looks like GPT-4-32k is rolling out by cubefox on Saturday May 06, 2023 @11:15AM (#35842149)
O(n^2) seems unlikely:

https://cognitiverevolution.substack.com/p/openais-foundry-l....

https://news.ycombinator.com/item?id=34977194#:~:text=Sparse...

[ Parent | on HN ]
Re:It looks like GPT-4-32k is rolling out by MichaelZuo on Saturday May 06, 2023 @12:55PM (#35843132)
Your second link has the immediate comment "Gpt3 includes dense attention layers that are n^2". So it's not at all unlikely.
[ Parent | on HN ]
Re:It looks like GPT-4-32k is rolling out by space_fountain on Saturday May 06, 2023 @01:46PM (#35843638)
GPT3 was released 3 years ago now. There have been major advancements in scaling attention so it would be strange if they didn't use some of them
[ Parent | on HN ]
Re:It looks like GPT-4-32k is rolling out by MichaelZuo on Monday May 08, 2023 @03:47PM (#35866181)
It doesn't matter how many major advancements they made in scaling, as long as one component is O(n^2) or higher.
[ Parent | on HN ]
Re:It looks like GPT-4-32k is rolling out by cubefox on Monday May 08, 2023 @06:16PM (#35867671)
It's not the scale itself, it's the scaling architecture.
[ Parent | on HN ]
Re:It looks like GPT-4-32k is rolling out by MichaelZuo on Monday May 08, 2023 @08:01PM (#35868583)
The same applies.
[ Parent | on HN ]