News for hackers. Stuff that matters.
Re:Experimenting with Local LLMs on macOS by data-ottawa on Monday September 08, 2025 @01:59PM (#45171596)
This is what I’m doing with my amd 395+.

I’m running docker containers with different apps and it works well enough for a lot of my use cases.

I mostly use Qwen Code and GPT OSS 120b right now.

When the next generation of this tech comes through I will probably upgrade despite the price, the value is worth it to me.

Re:Experimenting with Local LLMs on macOS by milgrum on Monday September 08, 2025 @08:34PM (#45175985)
How many TPS do you get running GPT OSS 120b on the 395+? Considering a Framework desktop for a similar use case, but I’ve been reading mixed things about performance (specifically with regards to memory bandwidth, but I’m not sure if that’s really the underlying issue)
Re:Experimenting with Local LLMs on macOS by data-ottawa on Tuesday September 09, 2025 @09:56AM (#45181976)
30-40 at 64k context, but it's a mixture of experts model.

A 70b dense model is slower

Qwen coder 30b Q4 runs 40+.