News for hackers. Stuff that matters.
Re:Perplexity Labs Playground by swyx on Tuesday December 19, 2023 @03:37PM (#38701169)
whats the intuition for 2/3 of RAM?
Re:Perplexity Labs Playground by avereveard on Tuesday December 19, 2023 @04:02PM (#38701495)
you need some leftover for holding the context
Re:Perplexity Labs Playground by M4v3R on Wednesday December 20, 2023 @01:03AM (#38705911)
Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.
Re:Perplexity Labs Playground by swyx on Wednesday December 20, 2023 @02:52PM (#38712822)
so why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption