News for hackers. Stuff that matters.
Re:Philosophers on GPT-3 by Baeocystin on Thursday July 30, 2020 @09:16PM (#24004653)
Do we know what the cost is for increasing the window size? I'd imagine that each additional token of range requires ever more power to properly integrate, but that is only a hunch.
Re:Philosophers on GPT-3 by orost on Thursday July 30, 2020 @09:20PM (#24004671)
I believe it's quadratic, but that still means making it 10x larger is perfectly possible just by brute force with a little motivation.
Re:Philosophers on GPT-3 by Baeocystin on Thursday July 30, 2020 @09:29PM (#24004723)
Interesting. That's less steep than I would have naively guessed. And you're right, if it's that 'easy', the next iteration will be (1) soon and (2) awesome, in both the positive and negative sense of the word.