Coding with LLMs in the summer of 2025 (an update)

Re:Coding with LLMs in the summer of 2025 (an update) by jmb99 on Sunday July 20, 2025 @10:52PM (#44631352)

What’s your budget and speed requirement? A quad-CPU Xeon E7 v4 server (Supermicro X10QBI, for example) with 1TB of RAM gives you ~340GB/s memory bandwidth and enough actual memory to host a full DeepSeek instance, but it will be relatively slow (a few tokens/s max in my experience). Up front cost a bit under $1k, less if you can source cheap 32GB DDR3 RAM. Power consumption is relatively high, ~1kW under load. But I don’t think you can self host a large model cheaper than that.

(If you need even more memory you could equip one of those servers with 6TB of DDR3 but you’ll lose a bit of bandwidth if you go over 2TB. DDR4 is also a slightly faster option but you’re spending 4x as much for the same capacity.)

[ Parent | on HN ]

Re:Coding with LLMs in the summer of 2025 (an update) by pmarreck on Sunday July 20, 2025 @11:12PM (#44631449)

This would require massively more power than the Mac Studios.

[ Parent | on HN ]

Re:Coding with LLMs in the summer of 2025 (an update) by jmb99 on Monday July 21, 2025 @12:53PM (#44637427)

Yep, ~1kW as mentioned. Depending on your electrical rate, break even might be years down the line. And obviously the Mac Studios would perform substantially better.

Edit: And also, to get even half as much memory, you need to spend $10k. If you want to host actually-large LLMs (not quantized/distilled versions), you'll need to spend close to that much. Maybe you can get away with 256GB for now, but that won't even host full Deepseek now (and I don't know if 512GB either, with OS/etc overhead, and a large context window).

[ Parent | on HN ]

Re:Coding with LLMs in the summer of 2025 (an update) by theshrike79 on Tuesday July 22, 2025 @04:18AM (#44644474)

I think we're at early 2000's bitcoin markets here.

People were buying stores empty of GPUs to mine for BTC.

Then people built custom ASICs that couldn't do anything but mine BTC, but did it a lot cheaper and with a lot less electricity required -> nobody GPU mines anymore pretty much.

I'm waiting for a similar thing to happen to local AI.

[ Parent | on HN ]