News for hackers. Stuff that matters.
Re:New iMac with M4 by lynguist on Monday October 28, 2024 @01:02PM (#41973432)
16 GB or 24 GB for inference doesn’t cut it for large models…

Mac hardware offers up to 128 GB shared RAM.

Re:New iMac with M4 by diffeomorphism on Monday October 28, 2024 @03:22PM (#41975023)
It is in a weird middle ground. It is much worse ram and much slower hardware (factor 4 or so IIRC) and only an option if you need more than 24 but less than 200. Also, only if you think that 9000€ is pocket change but 20000€ is cost prohibitive. Also, you need all that power but no server, no ECC,... And you only ever want to do inference but no training or tuning.

If you want a Mac anyway, sure. But if you don't care, this seems like a very, very specific Venn diagram.

Re:New iMac with M4 by int_19h on Monday October 28, 2024 @03:51PM (#41975420)
It's not much worse RAM, though. RTX 4090 has memory bandwidth of 1050 Gb/s. M2 Ultra is 800 Gb/s. And you can get a Mac Studio with Ultra and 128Gb of RAM for $3K or less. It's great for 70-150B models.

You're correct that it's only good for inference, but most people running local LLMs only do inference.

Re:New iMac with M4 by keybits on Monday October 28, 2024 @06:05PM (#41976939)
That config Mac Studio costs 5,800 Euros (minimum). Where do you get it for 3,000 USD?
Re:New iMac with M4 by int_19h on Monday October 28, 2024 @07:08PM (#41977533)
You don't need an M2 for this, an M1 will do just fine. Nor do you need the one with maxed-out SSD, which jacks up the price considerably. Finding a brand new M1 Ultra for $4K or less right now is pretty easy. When I got mine, about a year ago, $3K was the best deal I could find.