News for hackers. Stuff that matters.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by soccerdave on Tuesday November 03, 2026 @03:18PM (#50357606)
Can you tell a little more about your software setup? Been playing with local AI models but struggle to get anything as easy as ChatGPT or Claude. I have a folder with skills, and I can switch between ChatGPT and Claude, and either will do the same work. I’m struggling to find something that can do that locally
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by ActorNightly on Tuesday November 03, 2026 @03:24PM (#50357674)
Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by dr_hooo on Tuesday November 03, 2026 @03:50PM (#50357984)
> need to basically write custom harnesses

Curious, why/how does that help? Or do you have some niche needs?

Re:Qwen 3.8 27B on my laptop can interpret model weights, too by ActorNightly on Tuesday November 03, 2026 @04:19PM (#50358340)
The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.

As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.

Re:Qwen 3.8 27B on my laptop can interpret model weights, too by dr_hooo on Tuesday November 03, 2026 @04:45PM (#50358636)
But the question remains: isn't something like pi.dev good enough for this? Or are you using custom harness to describe the scripts/skills/etc that you create yourself?
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by trog on Tuesday November 03, 2026 @06:48PM (#50359833)
I am cheerfully using Opencode and pi.dev on a local model without thinking about any of that stuff, just running it straight out of the box and it's working great. I have access to an RTX6000 on a remote machine so using it for 'free' and it's replaced Claude for my personal projects. It gets stuck occasionally and gets things wrong from time to time, but so does everything.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by ActorNightly on Tuesday November 03, 2026 @10:21PM (#50361159)
No. All agents out there basically operate on skills, mempry, and tools. There are almost none that do context injection with every prompt. You basically need an external system that modifies every submitted prompt with additional context
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by dr_hooo on Wednesday November 04, 2026 @09:11PM (#50374222)
Okay, just started to read up an Agentic Context Engineering (ACE) & Co,: really interesting. Thank you for the explanation!
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by nickspacek on Wednesday November 04, 2026 @05:47AM (#50363745)
Yes, this is a straightforward extension for Pi to have it continually steer the model at every turn. You could write your own with Pi itself or use an existing extension.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by bityard on Tuesday November 03, 2026 @03:52PM (#50358014)
> You then need to basically write custom harnesses.

Why? Most existing harnesses let you point them at whatever API you want. I'm currently have a great time with Juggler.

Re:Qwen 3.8 27B on my laptop can interpret model weights, too by dcrazy on Tuesday November 03, 2026 @04:14PM (#50358283)
I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by ActorNightly on Tuesday November 03, 2026 @04:15PM (#50358293)
For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by irthomasthomas on Wednesday November 04, 2026 @04:30AM (#50363169)
Why use llamacpp instead of something faster like exllama?
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by zkmon on Tuesday November 03, 2026 @04:35PM (#50358522)
One RTX4090 with 24G gives me about 200k KV cache with Q4_K_M quant for typica coding tasks, via opencode. I use Q6_K with 60k cache if the work doesn't require scanning too many files. Q6_K is really good.
Re:Qwen 3.8 27B on my laptop can interpret model weights, too by netsroht on Wednesday November 04, 2026 @03:35AM (#50362841)
Could you shed some more light on the differences when you switch from Q4 to Q6_K? Tbh, I prefer having larger context windows since it looks like qwen just figures it out eventually. Since I noticed that it really wants to fill its context by, e.g., reading many files etc. I'd be open to experiment with smaller context windows and less quantization in my ninfer fork [0] if there's a benefit to it. But sometimes I think it's just more art that science :)

[0] https://github.com/tensorninja/ninfer-4090

Re:Qwen 3.8 27B on my laptop can interpret model weights, too by codazoda on Tuesday November 03, 2026 @07:37PM (#50360178)
I had the same trouble but I also had some success. I did a little write up on what I got to work.

https://joeldare.com/a-local-open-weight-model-builds-its-fi...

I think it can be made to work if you’re willing to tweak the prompts.

But, to be honest, I find my time to be worth more than the $20 I spend on a light weight frontier plan that can do this without the hand holding.

I want local to succeed, however, because of things like Anthropic stopping games based on other games.

Re:Qwen 3.8 27B on my laptop can interpret model weights, too by netsroht on Wednesday November 04, 2026 @03:28AM (#50362789)
Shameless plug here: I'm building a coding agent that kicks smaller models on track again if they misbehave. E.g. I noticed that qwen-3.8-27b often writes an intend but then stops and hands the turn back to the user. This is why I implemented lots of "nudges" and paired with a goal mode (and maybe autopilot) does wonders.

Here are some examples: https://caudra.ai/gallery/#nudges