Re:Philosophers on GPT-3
by Baeocystin on Thursday July 30, 2020 @09:16PM
(#24004653)
Do we know what the cost is for increasing the window size? I'd imagine that each additional token of range requires ever more power to properly integrate, but that is only a hunch.
[ Parent
| on HN ]
I believe it's quadratic, but that still means making it 10x larger is perfectly possible just by brute force with a little motivation.
Re:Philosophers on GPT-3
by Baeocystin on Thursday July 30, 2020 @09:29PM
(#24004723)
Interesting. That's less steep than I would have naively guessed. And you're right, if it's that 'easy', the next iteration will be (1) soon and (2) awesome, in both the positive and negative sense of the word.