News for hackers. Stuff that matters.
Re:Unsloth GLM-5.2 – How to Run Locally by eventualcomp on Monday June 22, 2026 @07:11PM (#48637750)
Where is $50k coming from again?
Re:Unsloth GLM-5.2 – How to Run Locally by stingraycharles on Monday June 22, 2026 @07:15PM (#48637790)
That’s less than the monthly salary of 10 software engineers, and assuming they pay API prices, probably earns itself back in about a year.

Having said that, I don’t think it’s all that tempting for companies at all, considering this whole market is developing rapidly and it’s nearly impossible to predict where we’ll be at in a year or two.

Re:Unsloth GLM-5.2 – How to Run Locally by cogman10 on Monday June 22, 2026 @07:25PM (#48637905)
The hardware requirements aren't evolving and the local models have only been improving.

It's not like you'd lose capabilities, if anything this solution just gets better with time.

Re:Unsloth GLM-5.2 – How to Run Locally by chatmasta on Monday June 22, 2026 @07:58PM (#48638258)
If the newer models require more/better hardware then you’ll lose capabilities.

I think you’re better off renting GPU instances and running all the software on those. It’ll be cheaper than Anthropic and OpenRouter but slightly more expensive than electricity and depreciation of hardware.

Re:Unsloth GLM-5.2 – How to Run Locally by cogman10 on Monday June 22, 2026 @09:10PM (#48638845)
The newer models don't require more/better hardware. There's a small army of local llm enthusiasts who are running LLMs using 3090s and H100s because they have lots of memory. Them being old isn't really that big of an issue as the compute power needed is relatively low all things considered.

The number of parameters needed for these open weight models has mostly stabilized so the actual memory requirements aren't likely to change all that much.

Re:Unsloth GLM-5.2 – How to Run Locally by dannyw on Monday June 22, 2026 @11:27PM (#48639874)
Correct. The main bottleneck with LLM inference is, and have always been, memory bandwidth.

TPS = active weights in GB / your memory bandwidth.

That’s it for decode. That’s all.

Re:Unsloth GLM-5.2 – How to Run Locally by cogman10 on Monday June 22, 2026 @07:19PM (#48637840)
As in who pays for it or how did I arrive at that number?

For who pays for it, obviously the employer would.

For "how did I arrive at this number" Ballpark estimate from what I know about part cost. Most of that money will go towards AI cards about $5k for the mb, cpu, power supply, etc. $45k would be for as much ram and as big/expensive nVidia cards as you can get your hands on. The B300 has 288GB of VRAM in it. Probably what you'd be after.

Re:Unsloth GLM-5.2 – How to Run Locally by Tepix on Tuesday June 23, 2026 @09:42AM (#48644869)
$50K seems low if you want to run, say, GLM 5.2 4bit fast enough for a team for devs.

You need something like 6x RTX Pro 6000 at $11800 each plus a nice server (add $10000) = $80800 and then quite a bit of electricity.

Re:Unsloth GLM-5.2 – How to Run Locally by theYipster on Tuesday June 23, 2026 @01:43PM (#48648529)
You don't need all of the model in VRAM. 1 or 2 RTX Pro 6000s will do. $50K will get you there very nicely, and on a 1600 watt PSU if you go for the MAX-Q versions. (The same wattage PSU I'm typing this on, and have been using over the last 5 years.)
Re:Unsloth GLM-5.2 – How to Run Locally by Tepix on Wednesday June 24, 2026 @02:23AM (#48655904)
If you want decent performance (more than say 20 tokens/s) for your dev team, you absolutely do need all of the model in VRAM.