News for hackers. Stuff that matters.

A comment on Introducing DBRX: A New State-of-the-Art Open LLM by Databricks. Parent comment

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by swalsh on Wednesday March 27, 2024 @12:02PM (#39840995)
> The model requires ~264GB of RAM

This feels as crazy as Grok. Was there a generation of models recently where we decided to just crank on the parameter count?

[ Parent | on HN ]
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by wrs on Wednesday March 27, 2024 @01:03PM (#39841704)
Isn’t that pretty much the last 12 months?
[ Parent | on HN ]
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by Jackson__ on Wednesday March 27, 2024 @02:16PM (#39842696)
If you read their blog post, they mention it was pretrained on 12 Trillion tokens of text. That is ~5x the amount of the llama2 training runs.

From that, it seems somewhat likely we've hit the wall on improving <X B parameter LLMs by simply scaling up the training data, which basically forces everyone to continue scaling up if they want to keep up with SOTA.

[ Parent | on HN ]
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by breezeTrowel on Wednesday March 27, 2024 @03:32PM (#39843551)
Cranking up the parameter count is literally how the current LLM craze got started. Hence the "large" in "large language model".
[ Parent | on HN ]
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by espadrine on Wednesday March 27, 2024 @06:22PM (#39845309)
Not recently. GPT-3 from 2020 requires even more RAM; the open-source BLOOM from 2022 did too.

In my view, the main value of larger models is distillation (which we particularly witness, for instance, with how Claude Haiku matches release-day GPT-4 despite being less than a tenth of the cost). Hopefully the distilled models will be easier to run.

[ Parent | on HN ]