News for hackers. Stuff that matters.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by PheonixPharts on Wednesday March 27, 2024 @10:27AM (#39839606)
While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio.

Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup.

It really boils down to what type of local development you want to do. I'm mostly experimenting with things where the time to response isn't that big of a deal, and not fine-tuning the models locally (which I also believe GPUs are still superior for). But if your concern is "how big of a model can I run" vs "Can I have close to real time chat", the unified memory approach is superior.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by bee_rider on Wednesday March 27, 2024 @11:25AM (#39840513)
I know the M?-pro and ultra variants are multiple standard M?’s in a single package. But so the CPUs and GPUs share a die (like a single 4 p-core CPU 10 GPU core is what come in the die, and the more exotic variants are just a result of LEGO-ing out those guys and disabling some cores for market segmentation or because they had defects?)

I guess I’m wondering if they technically could throw in their gauntlet and compete with nvidia by doing something like a 4 CPU/80 GPU/256 GB chip, if they wanted to. Seems like it’d be a really appealing ML machine. (I could also see it being technically possible but Apple just deciding that’s pointlessly niche for them).

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by astrange on Wednesday March 27, 2024 @06:07PM (#39845198)
Ultra is the only one that's made from two smaller SoCs.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @11:08AM (#39840276)
I already have 128GB of RAM (DDR4), and was wondering if upgrading from a 1080ti (12GB) to a 4070ti super (16GB), would make a big difference.

I assume the FP32 and FP16 operations are already a huge improvement, but also the 33% increased VRAM might lead to fewer swaps between VRAM and RAM.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by loudmax on Wednesday March 27, 2024 @11:47AM (#39840786)
I have an RTX 3080 with 10GB of VRAM. I'm able to run models larger than 10GB using llama.cpp and offloading to the GPU as much as can fit into VRAM. The remainder of the model runs on CPU + regular RAM.

The `nvtop` command displays a nice graph of how much GPU processing and VRAM is being consumed. When I run a model that fits entirely into VRAM, say Mistral 7B, nvtop shows the GPU processing running at full tilt. When I run a model bigger than 10GB, say Mixtral or Llama 70B with GPU offloading, my CPU will run full tilt and the VRAM is full, but the GPU processor itself will operate far below full capacity.

I think what is happening here is that the model layers that are offloaded to the GPU do their processing, then the GPU spends most of the time waiting for the much slower CPU to do its thing. So in my case, I think upgrading to a faster GPU would make little to no difference when running the bigger models, so long as the VRAM is capped at the same level. But upgrading to a GPU with more VRAM, even a slower GPU, should make the overall speed faster for bigger models because the GPU would spend less time waiting for the CPU. (Of course, models that fit entirely into VRAM will run faster on a faster GPU).

In my case, the amount of VRAM absolutely seems to be the performance bottleneck. If I do upgrade, it will be for a GPU with more VRAM, not necessarily a GPU with more processing power. That has been my experience running llama.cpp. YMMV.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by htrp on Wednesday March 27, 2024 @02:49PM (#39843097)
How's your performance on the 70b parameter llama series?

Any good writeups of the offloading that you found?

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by loudmax on Wednesday March 27, 2024 @04:36PM (#39844293)
Performance of 70b models is like 1 token every few seconds. And that's fitting the whole model into system RAM, not swap. It's interesting because some of the larger models are quite good, but too annoyingly slow to be practical for most use cases.

The Mixtral models run surprisingly well. They can run better than 1 token per second, depending on quantization. Still slow, but approaching a more practical level of usefulness.

Though if you're planning on accomplishing real work with LLMs, the practical solution for most people is probably to rent a GPU in the cloud.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by zozbot234 on Wednesday March 27, 2024 @11:23AM (#39840494)
That's system memory, not unified memory. Unified means that all or most of it is going to be directly available to the Apple Silicon GPU.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by giancarlostoro on Wednesday March 27, 2024 @02:43PM (#39843023)
This is the key factor here. I have a 3080, with 16GB of Memory, but still have to run some models on CPU since the memory is not unified at all.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by bevekspldnw on Wednesday March 27, 2024 @01:34PM (#39842151)
I had gone the Mac Studio route initially, but I ended up with getting an A6000 for about the same price as a Mac and putting that in a Linux server onder my desk. Ollama makes it dead simple to serve it over my local network, so I can be on my M1 Air and using it no differently than if on my laptop. The difference is that the A6000 absolutely smokes the Mac.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by starik36 on Wednesday March 27, 2024 @02:44PM (#39843040)
Wow, that is a lot of money ($4400 on Amazon) to throw at this problem. I am curious, what was the purpose that compelled you to spend this (for the home network, I assume) amount of money.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by bevekspldnw on Wednesday March 27, 2024 @04:43PM (#39844365)
Large scale document classification tasks in very ambiguous contexts. A lot of my work goes into using big models to generate training data for smaller models.

I have multiple millions of documents so GPT is cost prohibitive, and too slow. My tools of choice tend to be a first pass with Mistral to check task performance and if lacking using Mixtral.

Often I find with a good prompt Mistral will work as well as Mixtral and is about 10x faster.

I’m on my “home” network, but it’s a “home office” for my startup.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by Datagenerator on Thursday March 28, 2024 @01:14AM (#39848003)
Interesting I have the same task, can you share your tools? My goal is to detect if documents contain GDPR sensitive parts or are copies of official documents like ID's and driving licenses etc - would be great to reuse your work!
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by bevekspldnw on Thursday March 28, 2024 @04:12PM (#39856732)
Working in the same sector, we’ll license it out soon.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by c1b on Wednesday March 27, 2024 @06:08PM (#39845214)
> The difference is that the A6000 absolutely smokes the Mac.

Memory Bandwidth : Mac Studio wins (about the same @ ~800)

VRAM : Mac Studio wins (4x more)

TFLOPs: A6000 wins (32 vs 38)

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by bevekspldnw on Wednesday March 27, 2024 @06:23PM (#39845326)
VRAM in excess of the model one is using isn’t useful per se. My use cases require high throughput, and on many tasks the A6000 executes inference at 2x speed.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by rldjbpin on Thursday March 28, 2024 @06:42AM (#39849545)
this. if you can afford m3 level of money, a6000 is definitely worth it and provides you long-term access to a level of compute even hard to find in the cloud (for the price and waiting period).

it is only dwarfed by other options if your workload can use multi-gpu, which is not a granted for most cases.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by purpleblue on Wednesday March 27, 2024 @02:12PM (#39842644)
Aren't the Macs good for inference but not for training or fine tuning?
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by spxneo on Wednesday March 27, 2024 @03:01PM (#39843244)
Aren't quantized models different models outright requiring a new evaluation to know the deviation in performance? Or are they "good enough" in that the benefits outweigh the deviation?

I'm on the fence about whether to spend 5 digits or 4 digits. Do I go the Mac Studio route or GPUs? What are the pros and cons?

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by brandall10 on Wednesday March 27, 2024 @03:25PM (#39843482)
Wait for the M3 Ultra and it will be 256GB and markedly faster.