News for hackers. Stuff that matters.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @11:04AM (#39840222)
> No, it can't run at all.

https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg

EDIT: This was ran on a 1080ti + 5900x. Initial generation takes around 10-30seconds (like it has to upload the model to GPU), but then it starts answering immediately, at around 3 words per second.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by wokwokwok on Wednesday March 27, 2024 @11:43AM (#39840734)
Did you check your GPU utilization?

Typically when it runs that way it runs on the CPU, not the GPU.

Are you sure you're actually offloading any work to the GPU?

At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all.

The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/...

It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @12:01PM (#39840989)
I have to check, something does indeed seem weird, especially with the PC freezing like that. Maybe it runs on the CPU.

> quantized version Yes, it is 4bit quantized, but still has 24.6GB

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by spxneo on Wednesday March 27, 2024 @03:04PM (#39843275)
this is some new flex to debate online: copying and pasting the other sides argument and waiting for your local LLM to explain why they are wrong.

how much is your hardware at today's value? what are the specs? that is impressive even though its 3 words per second. if you want to bump it up to 30, do you then 10x your current hardware cost?

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @03:43PM (#39843663)
That question was just an example (Lorem ipsum), it was easy to copy paste to demo the local LLM, I didn't intend to provide more context to the discussion.

I ordered a 2nd 3090, which has 24GB VRAM. Funny how it was $2.6k 3 years ago and now is $600.

You can probuild a decent AI local machine for around $1000.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by spxneo on Wednesday March 27, 2024 @03:49PM (#39843742)
https://howmuch.one/product/average-nvidia-geforce-rtx-3090-... you are right there is a huge drop in price
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @07:29PM (#39845896)
New it's hard to find, but the 2nd hand market is filled with them.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by taneq on Thursday March 28, 2024 @12:44AM (#39847865)
Where are you seeing 24GB 3090s for $600?
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Thursday March 28, 2024 @06:31AM (#39849492)
2nd hand market
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by llm_trw on Wednesday March 27, 2024 @06:47PM (#39845543)
Congratulations on using CPU inference.