News for hackers. Stuff that matters.
Re:GPT-4 by bitL on Tuesday March 14, 2023 @06:21PM (#35159892)
I am surprised they allow only 32k tokens when Reformer can have context length of 1M on 16GB VRAM. It seems like they have some ways to optimize it further.
Re:GPT-4 by geysersam on Tuesday March 14, 2023 @07:32PM (#35161090)
Is the Reformer as capable as this model? It's a trade-off.
Re:GPT-4 by bitL on Tuesday March 14, 2023 @10:35PM (#35163023)
It's not, it uses locality-sensitive hashing to reduce attention complexity from O(n^2) to O(nlogn) while maintaining the same performance in 16GB as a best model that could fit into 100GB but nobody scaled it up to 1000 GPUs as its purpose was the opposite.