News for hackers. Stuff that matters.

A comment on Introducing DBRX: A New State-of-the-Art Open LLM by Databricks. Parent comment

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by irusensei on Thursday March 28, 2024 @10:40AM (#39852144)
I can run a certain 120b on my M3 max with 128GB memory. However I found that while it “fits” Q5 was extremely slow. The story was different with Q4 though which ran just fine around ~3.5-4 t/s.

Now this model is ~134B right? It could be bog slow but on the other hand its a MoE so there might be a chance it could have satisfactory results.

[ Parent | on HN ]
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by marci on Thursday March 28, 2024 @05:08PM (#39857322)
From the article, should have the speed of a ~36b.
[ Parent | on HN ]