News for hackers. Stuff that matters.
Re:Codex for almost everything by frank_nitti on Thursday April 16, 2026 @04:41PM (#47799219)
Interesting to see your observation where I have observed the opposite: posts that share big news about open-weight local models have many upvoted comments arguing local models shouldn’t be taken seriously and promoting the SOTA commercial models as the only viable options for serious developers.

Here and on AI tech subreddits (ones that aren’t specifically about local or FOSS) seem to have this dynamic, to the degree I’ve suspected astroturfing.

So it’s refreshing to see maybe that’s just a coincidence or confirmation bias on my end.

Re:Codex for almost everything by lxgr on Thursday April 16, 2026 @05:27PM (#47799717)
Both can be true at the same time. I currently wouldn't waste my time with open models for almost all use cases, but they're crucial from a data privacy and competitive perspective, and I can't wait for them to catch up enough to be as useful as the current frontier models.
Re:Codex for almost everything by organsnyder on Thursday April 16, 2026 @06:03PM (#47800097)
I've found qwen3 to be very usable on my local machine (a Framework Desktop with 128gb RAM). I doubt it could handle the complex tasks I throw at Claude Opus at work, but it's more than capable of doing a surprising number of tasks, with good performance.
Re:Codex for almost everything by dotancohen on Thursday April 16, 2026 @06:14PM (#47800214)
What tasks do you use qwen3 for? Coding? Are you running it on CPU or GPU? What GPU does that Framework have?

Thanks!

Re:Codex for almost everything by girvo on Thursday April 16, 2026 @06:44PM (#47800457)
I have an Asus GX10 that I run Qwen3.5 122B A10B on, and I use it for coding through the Pi coding agent (and my own); I have to put more work in to ensure that the model verifies what it does, but if you do so its quite capable.

It makes using my Claude Pro sub actually feasible: write a plan with it, pick it up with my local model and implement it, now I'm not running out of tokens haha.

Is it worth it from a unit economics POV? Probably not, but I bought this thing to learn how to deploy and serve models with vLLM and SGLang, and to learn how to fine tune and train models with the 128GB of memory it gets to work with. Adding up two 40GB vectors in CUDA was quite fun :)

I also use Z.ai's Lite plan for the moment for GLM-5.1 which is very capable in my experience.

I was using Alibaba's Lite Coding Plan... but they killed it entirely after two months haha, too cheap obviously. Or all the *claw users killed it.

Re:Codex for almost everything by dotancohen on Thursday April 16, 2026 @06:56PM (#47800564)
Thank you. I've been using ollama for a much more modest local inference system. I'll research some of the things you've mentioned.
Re:Codex for almost everything by jeremyjh on Thursday April 16, 2026 @09:06PM (#47801463)
GLM 5.1 is extremely good, and ridiculously cheap on their coding plan. Its far better than Sonnet, and a fifth of the cost at API rates. I don't know if the American providers can compete long-term; what good is it to be more innovative it only buys them a six month lead andthey can't build the data center capacity fast enough for demand? Chinese providers have a huge advantage in electrical grid capacity.
Re:Codex for almost everything by girvo on Thursday April 16, 2026 @10:07PM (#47801784)
True but Z.ai also just silently raised the price, and the entire Chinese frontier set is having to make profit now... hence Alibaba killing the Lite plan and not letting people sign up to their Pro one either; and why MiniMax has their non-commercial license, etc. etc.

So I agree with you, its better than Sonnet but way cheaper. I do wonder how long that will last though

Re:Codex for almost everything by fragmede on Friday April 17, 2026 @01:00AM (#47802574)
Z.ai does really well at the carwash question!
Re:Codex for almost everything by organsnyder on Thursday April 16, 2026 @06:21PM (#47800276)
The Framework Desktop has a Ryzen 395 chip that is able to allocate memory to either the CPU or GPU. I've been able to allocate 100+gb to the GPU, so even big models can run there.

Most recently I used it to develop a script to help me manage email. The implementation included interacting with my provider over JMAP, taking various actions, and implementing an automated unsubscribe flow. It was greenfield, and quite trivial compared to the codebases I normally interact with, but it was definitely useful.

Re:Codex for almost everything by dotancohen on Thursday April 16, 2026 @06:58PM (#47800595)
That's great. Ostensibly my system could also allocate some of the 32 GB of system memory to argument the 12 GB VRAM, but I've not been able to get it to load models over 20B. I should spend some more time on it.
Re:Codex for almost everything by bloppe on Thursday April 16, 2026 @05:10PM (#47799532)
I'm just waiting till I can afford a GPU again
Re:Codex for almost everything by nl on Thursday April 16, 2026 @09:47PM (#47801686)
I've invested significant time into getting open models to work, and investigating what works well.

The TL;DR is that unless you are doing it as a hobby or working in an environment where none of the data privacy options supported by Anthropic/OpenAI (including running on Azure/Bedrock with ZDR) work for you then it's not worth it.

The best open models are around the Sonnet 4.6 level. That's excellent, but the level of tasks you can give to GPT 5.4 or Opus 4.6 is just so much higher it doesn't compare (and Opus 4.7 seems noticeably better in my few hours of testing too).

I have my own benchmarks, but I like this much under-publicized OpenHands page: https://index.openhands.dev/home

It shows for every task they test closed models do the best. The closest and open model gets is Minmax 2.7 on issue resolution where it's ~1% worse than the leaders.

That matches my experience - fine for small problems, but well behind has the task gets bigger.

Re:Codex for almost everything by echelon on Thursday April 16, 2026 @09:48PM (#47801688)
> Interesting to see your observation where I have observed the opposite: posts that share big news about open-weight local models have many upvoted comments arguing local models shouldn’t be taken seriously and promoting the SOTA commercial models as the only viable options for serious developers.

When I argue this, my point is that FOSS shouldn't target the desktop with open weights - it should target H200s. Really big parameter models with big VRAM requirements.

Those can always be distilled down, but you can't really go the other way.

Re:Codex for almost everything by adamsmark on Friday April 17, 2026 @01:54PM (#47808654)
Local isn't viable yet on an economic basis, API costs are so low that you're better off taking advantage of the bonanza. As local models become more performant, so too will the ability of providers via Openrouter be able to offer them cheaper than your likely payoff period for a $4K Mac Studio 128GB. e.g Gemma 4 31B is impressive, but it costs practically nothing via Openrouter. Given that there are a ton of providers for open models, I doubt there's any subsidy going on because the providers are faceless and interchangeable.

At least, that's my theory.

The big advantages of local on a business level are:

- Freezing your model's exact settings once you've locked in some kind of workflow that works just fine. - Guarding against insane token usage from LLMs who have been told to never stop until they figure out the solution OR setting up an LLM run incorrectly. (The last one happened to me with Gemini 3.1 Pro) - PII or some need for on-premise only LLMs.