News for hackers. Stuff that matters.

A comment on Stability AI: Emad Mostaque resigned from his role as CEO. Parent comment

Re:Stability AI: Emad Mostaque resigned from his role as CEO by bawana on Saturday March 23, 2024 @09:25AM (#39799766)
Why would increasing memory bandwidth reduce performance? You said "You can get a decent approximation for LLM performance in tokens/second by dividing the model size in GB by the system's memory bandwidth"
[ Parent | on HN ]
Re:Stability AI: Emad Mostaque resigned from his role as CEO by AnthonyMouse on Sunday March 24, 2024 @03:43AM (#39805674)
Memory bandwidth is in the denominator. Making it bigger makes tokens/second go up.
[ Parent | on HN ]
Re:Stability AI: Emad Mostaque resigned from his role as CEO by AnthonyMouse on Sunday March 24, 2024 @04:27AM (#39805819)
Yeah the sentence is backwards, you divide the system's memory bandwidth by the size of the model.
[ Parent | on HN ]