News for hackers. Stuff that matters.
Re:New iMac with M4 by traceroute66 on Monday October 28, 2024 @12:15PM (#41972796)
> As someone who mucks around with running LLMs and other large models on my computer the 32GB maximum RAM is a show-stopper for me.

To be fair, I don't think people mucking around with large LLMs is the primary target market for the iMacs.

The sort of people who muck around with LLMs almost certainly already have a monitor, keyboard and mouse. And so are more likely to pick up a Mac Studio which will no doubt be coming soon with M4 Ultra.

Re:New iMac with M4 by whiplash451 on Monday October 28, 2024 @12:25PM (#41972946)
Or separate concerns and shell out on a Studio display with the-mac-you-like on usb-c
Re:New iMac with M4 by talldayo on Monday October 28, 2024 @12:45PM (#41973224)
Or buy a real computer, plug in an Nvidia GPU, and save your DRAM for compute while using your VRAM for inference.
Re:New iMac with M4 by lynguist on Monday October 28, 2024 @01:02PM (#41973432)
16 GB or 24 GB for inference doesn’t cut it for large models…

Mac hardware offers up to 128 GB shared RAM.

Re:New iMac with M4 by diffeomorphism on Monday October 28, 2024 @03:22PM (#41975023)
It is in a weird middle ground. It is much worse ram and much slower hardware (factor 4 or so IIRC) and only an option if you need more than 24 but less than 200. Also, only if you think that 9000€ is pocket change but 20000€ is cost prohibitive. Also, you need all that power but no server, no ECC,... And you only ever want to do inference but no training or tuning.

If you want a Mac anyway, sure. But if you don't care, this seems like a very, very specific Venn diagram.

Re:New iMac with M4 by int_19h on Monday October 28, 2024 @03:51PM (#41975420)
It's not much worse RAM, though. RTX 4090 has memory bandwidth of 1050 Gb/s. M2 Ultra is 800 Gb/s. And you can get a Mac Studio with Ultra and 128Gb of RAM for $3K or less. It's great for 70-150B models.

You're correct that it's only good for inference, but most people running local LLMs only do inference.

Re:New iMac with M4 by keybits on Monday October 28, 2024 @06:05PM (#41976939)
That config Mac Studio costs 5,800 Euros (minimum). Where do you get it for 3,000 USD?
Re:New iMac with M4 by int_19h on Monday October 28, 2024 @07:08PM (#41977533)
You don't need an M2 for this, an M1 will do just fine. Nor do you need the one with maxed-out SSD, which jacks up the price considerably. Finding a brand new M1 Ultra for $4K or less right now is pretty easy. When I got mine, about a year ago, $3K was the best deal I could find.
Re:New iMac with M4 by nickthegreek on Monday October 28, 2024 @12:53PM (#41973333)
So now you just limited to 24gb unless you are running dual 3090s or leave the consumer market for a gpu.
Re:New iMac with M4 by talldayo on Monday October 28, 2024 @12:55PM (#41973363)
I mean, model layering has been around for several years now: https://huggingface.co/blog/lyogavin/airllm
Re:New iMac with M4 by Philpax on Monday October 28, 2024 @01:12PM (#41973540)
Moving the weights between the CPU and the GPU significantly limits performance. It's not comparable to having the entire model resident within video/unified memory.
Re:New iMac with M4 by sroussey on Monday October 28, 2024 @09:09PM (#41978340)
Yet it gets done for really big models. I can’t find the description for how Cerebras is doing Llama 405b for reference, but they are splitting by layers for that one (and why it’s not available right now).

https://cerebras.ai/blog/introducing-cerebras-inference-ai-a...

Re:New iMac with M4 by Philpax on Tuesday October 29, 2024 @07:07AM (#41982595)
That's not quite the same thing. There are three layers in the memory hierarchy (there are more, but for the purpose of this discussion, three is sufficient)

- CPU RAM

- GPU RAM

- GPU SRAM

The grandposter was talking about moving layers between CPU and GPU RAM, while Cerebras store models entirely in SRAM.

CPU <-> GPU is _much_ slower than GPU RAM <-> GPU SRAM: the former will never be competitive with keeping the weights entirely on accelerator. Comparing the bandwidth numbers given in that article with the bandwidth of PCI-E should demonstrate the problem.

Larger models are split between multiple accelerators, but even then they avoid going through the CPU as much as possible and instead use a direct interconnect (NVLink, etc). The weights are uploaded ahead of time to ensure they don't need to be reloaded during active inference.

The considerations for training are different, but the general principle of avoiding CPU traffic still holds.

Re:New iMac with M4 by diggan on Monday October 28, 2024 @12:47PM (#41973246)
> And so are more likely to pick up a Mac Studio which

Although price-conscious LLM muckers are most likely to pick any Apple-hardware. You can easily build rigs that are twice as powerful for half the price, assuming we compare desktops.

Re:New iMac with M4 by DrPhish on Monday October 28, 2024 @01:03PM (#41973448)
True. eBay dual-socket Epyc systems ala https://rentry.org/miqumaxx are a great case in point.