Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.

[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.

The swarm strategy can’t overcome the dumbed down model problem. You just end up with your swarm of agents all going in the same dumbed down loops.

5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B

The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.

Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.

> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.

I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.

In the fall and winter (right now) I’m heating the house anyway so running LLMs 24/7 is basically free to me. So I try it. The models have are limited in their intelligence. I have tasks where I could let Qwen run for days with concurrent workers and never get anywhere, but 15 minutes with Opus solves the problem.

I need to try this... it's on my list anyway.

https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...

You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?

Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...

Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though

I'm not sure I understand. Are you saying we could put a dinner-plate sized chip that costs millions of dollars and draws tens of killowats of power into every single computer on the planet? Or that a single of these chips could provide inference as a service for every computer on the planet? Because neither of those are even remotely feasible.

if the approach works, you have now a viable path to use this complex/hybrid structures where you can pair those chips with gpus and offload whatever is optimum to the model on chip.

Just like you can go and buy a threadripper now (the ram maybe not so much), how much was worth that amount of computing 10 years ago?

Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.

These 3bit quants of qwen are supposed to be 95% accurate as the full model and runs moothly on laptops:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

There are multiple independent tests of this on YouTube.

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...

I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.

IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.

https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4?hl=en...

DeepSeek and GLM are useful, though. Even the flash variants.

Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.

It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.

Wow, that sounds like a lot! Do you use/mean the regular models or flash variants here?

  5257408 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
  49215492960 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00002-of-00005.gguf
  49700372160 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00003-of-00005.gguf
  49466495968 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00004-of-00005.gguf
  13481997024 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00005-of-00005.gguf

  49842093280 GLM-5.3-Flash-Q3_XL-3.86bpw-00001-of-00004.gguf
  49644428576 GLM-5.3-Flash-Q3_XL-3.86bpw-00002-of-00004.gguf
  49653319264 GLM-5.3-Flash-Q3_XL-3.86bpw-00003-of-00004.gguf
  5694702080  GLM-5.3-Flash-Q3_XL-3.86bpw-00004-of-00004.gguf

  20437336576 qwen3_8_27b.ninfer

Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.

More details here:

https://jacquesmattheij.com/local-inference-with-large-langu...

I haven’t been able to be faster with Qwen than doing it myself. Even with GLM & DS Flash variants it’s not always worth my time.

GLM 5.3, Minimax, Sonnet, etc. I find are definitely worth using. They need handholding and prodding and you have to carefully review the output, but I almost always end up faster than doing it manually.

It uses all if my 6 rtx3090s but runs 7 sessions at combined 200tok/s and prefill of 7k on a cheap pc (no threadripper, all gpus via usb4).

DeepSeek v4.1 Flash 1017 is better but it is also 3x the size... And I find running quant below 3.8bits unusable.

But flash next is good enough (at 4.5bits).

Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.

...so we really do need open models to keep pace, not just be statically competitive at a given point in time

So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.

cecli.dev

I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.

and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.

If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.

It’s a moving target thus we need open weights to move

Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.

The key thing is tool use.

The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.

New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.

Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.

Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.

At the very least we need _current_ models, though. The weights have all sorts of information baked into them, like "how do Python unit test drivers work" or "what does idiomatic C# look like," that will just naturally go stale over time as the frameworks improve. In five years, Qwen 3.8 will still be targeting the Python of 2026.

Why do we need Rust, for instance, when LLMs know how to use C++ without shooting themselves in the foot? If that's not entirely true today, it certainly will be tomorrow.

See also the lack of new instruction-set architectures in the CPU business. Why do we need more, in the absence of fundamental and revolutionary hardware innovations? Even RISC-V is just a rehash of MIPS.

I.e. I think you can bake certain properties into the language such that various kinds of static checks can prevent certain types of errors. This should make the language nicer for coding without reading any of the code.

An LLM doesn't need a memory-safe language. It needs one it knows like the back of its metal hand.

I also switched to local qwen recently, but have to block such outgoing requests.

The model itself is just dead weights, which cannot do any action by themselves.

All actions are done by your inference engine and harness, and you should take care about what you allow those to do.

What it did: https://cohere.com/blog/concerning-behaviour-popular-open-we...

On current GPU hardware/firmware... The cynic in me imagines in the future GPU vendors wanting to implement only loading signed weights to your card, aligned with your "subscription level", for your "safety".

This is why it's so important to have more than one hardware vendor. The bigger the ecosystem, the harder it is to cartel. I cant believe it, but i am pinning my hopes on china to break through with commoditized chips and hardware.

We've been through this kind of thing before. Like copy restrictions in DVDs. You'd argue the drive vendors had no incentive to collude, yet they did.

those didnt come from vendors, but from the media companies, who have more power in their relationship.

Will nvidia become like that? It currently isn't, but who knows?

Unless the GPU vendors start selling end products? It's much more lucrative to sell locked down end-user devices after all. Shareholders will demand it.

I do not buy the sheeple argument of “companies force people to buy things they don’t want”. Demand always wins.

And that net neutrality is a good thing to enforce?

Enshittification is another shifting who value is delivered to, and I can make heads nor tails of what you’re trying to imply with net neutrality.

In comments it’s usually best to just directly say what you mean, there’s not enough signal or shared context for people to get innuendo.

(No offense or underhanded arguments.)

https://en.wikipedia.org/wiki/Enshittification

https://en.wikipedia.org/wiki/Net_neutrality

People came up with the word "enshittification" for a reason ... and the word is quite popular on HN, so make of that what you want.

Here I thought I was just a niche market that’s not profitable for manufacturers to serve, but apparently I’m a victim of tyranny and didn’t even know it!

Those products are a sick joke played on us. About the only thing they have that is better than tjose ancient gpus is energy efficiency.

And what good are the weights when all the rtx3090s fail (they were pretty crappy to begin with) and every ddr3 Xeon system fails too?

I'm fully prepared to be running on my hardware that is 6 and 8 years old for next decade. Last 3 months saw rtx3090 going up by 20% each month.

Unless it is locked down by the vendor, or perhaps even governments. Which at some point will start happening more and more ...

It strikes me that if it were up to Sam Altman, Eliezer Yudkowsky, Bill Gates and Dario Amodei, this crisis would persist indefinitely. If it were up to them I wouldn't be surprised to see them destroy the GPUs just them to keep them from the plebs.

I picked those names not to be hyperbolic, but because of very specific things I've seen them say.

In fact I fail to see any other end game where they make profit on the huge liabilities that now have.

Additionally, any capable local model can be reclassified as “cybersecurity risk”, not enough “safety guardrails” to run locally, etc

https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...

Which is literally the coolest thing to come out of this...

Did they get a letter from Nintendo's legal team or something?

Banning them would just invite competition which is why I think HF is holding out as long as they legally can.

Example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-fo...

Previously, I recall HuggingFace only took models down if someone complained / sent a DMCA or cease and desist.

Punish the consumer, ala the copyright war since the 90s

This is the kind of bet where I would take the opposite position without hesitation.

The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.

I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.

It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.

... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.

They're not small but they're hardly out of reach. I have backups on my home NAS.

> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017

It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.

Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)

To use an old analogy, it's not rocket science:

https://github.com/p-e-w/heretic

The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.

Especially as people grow more dependent on agents doing all of their computer work.

Not as bad as you might think. This costs about as much as a decent new car nowadays.

The abliterated 1017 release I've seen is crazy good.

I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.

...which is still tooo much for most people to spend on "AI hardware"....

It is the driver/user of the car who is liable, and possibly the manufacturer if it's defective.

Corn always wins in the end.

https://gnosh.ai/head-to-head.html

AFAIK the timeline was:

  - AI was shit, models could fit in your GPU
  - AI was shit, models could no longer fit in your GPU
  - AI got good

Is $2k for a whole machine not "consumer" ?

llama-ui on unsloth/Llama-3.2 3B Instruct:

  > hello

  Runtime info
  Current time is 2026-11-02T13:00:18+01:00
  Runtime info
  Current time is 2026-11-02T13:00:19+01:00
  Runtime info
  Current time is 2026-11-02T13:00:21+01:00
  Runtime info
  Current time is 2026-11-02T13:00:23+01:00

  There is not enough information about the functions to properly answer the prompt.

I did do the yak-shaving for the longest time. llama in a second-hand RTX3070 8GB was a useful tool. When gpt-oss came out, I upgraded to a second-hand 16GB card. It was slower (didn't seem to one-shot as much) but it gave smarter overall answers. At some point I bought a 24GB card which was basically dead on arrival. I kept my 16GB card in a server and maintained my VPN infra so I could access it from wherever.

I was still using this setup even when my workplace was pushing windsurf/copilot harder and harder. For me they were just broken. I'd ask for trivial tasks like "copy this unit test into a second unit test which tests something slightly different" and it would just get confused and delete shit, or leave random UUID strings throughout the codebase.

Eventually I tried out cline/openrouter/deepseekv4flash, and that was actually useful enough for me to tell it to do something to my code base, and as long as I supervised carefully enough it was useful, and priced OK.

At some point in the process I tried Claude Code because of all the hype, but I couldn't get any useful value out of it. I tried working with it all morning on something and hit my token limit without achieving anything worthwhile, so I cancelled Claude and went back to Deepseekv4flash.

These days every job application requires proficiency with claude code (or similar), so I was forced to get it again. And this time it just worked better. I haven't hit any limits, and it works like a better version of my cline/deepseek setup.

So where does that leave me?

Deepseek-v4-pro is 1.6T params, the cut-down version is 284B params, and I wouldn't trust them to do something on their own.

I'm very skeptical of a model being both small enough to run fast on my hardware and somewhere in the v4-flash <-> claude quality range.

If I did put down $2k, I wouldn't want it in such an old chip (M1) or a second-hand device that could die tomorrow with no chance of refund, especially to just 'see if it's good enough' (compared to the effectively-free v4-flash token spend).

who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.

Who's keeping you beholden to them now?

We achieved this with opus 4.6 last year, what those providers give you is great front end on top of it, which allows users to delegate even greater part of their process to model.

With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests simultaneously have total tok/s throughout much higher than when handling a single request (unlike game streaming, where the main cost benefit of putting GPUs in the cloud is just keeping them active 24/7). Cloud inference is/will be multiple orders of magnitude cheaper than owning a GPU for any reasonable amount of usage over the life of the GPU.

With mainframes, network latency hardly matters compared to overall batch processing time, and mainframes processing multiple requests simultaneously have total throughput much higher than when handling a single request

The vast majority of people spend their screen time on battery powered smartphones (akin to dumb terminals) backed by cloud servers (akin to the mainframes).

There are a few exceptions for gaming, programming*, and a few other specialized productivity tools. But their total userbase is a tiny fraction of mobile users.

*even for programming, some companies are starting to embrace cloud dev boxes.

At current GPU pricing you're right, unless you highly value running uncensored models, data ownership, and/or privacy.

This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.

Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.

My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.

The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.

I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).

It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”

There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)

He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.

Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.

[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>

[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)

Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.

I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.

edit: I should point out that it's also starting to accelerate now in that you can get your agents to help you with making them better at these tasks. So, it matters significantly less if you yourself don't have the skills to progress things, so long as you can figure out just enough to get the models to do it themselves. I suspect we'll see this dynamic ramp up quite a bit in 2027 with Qwen4.

Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.

I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.

    > Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. 

In the Emory paper:

    > We reduce this loss by expanding Φ with a stateful four-phase multi-agent evolver, temporal-reveal feedback, and cross-cycle memory. This design targets the unbounded-stream failure where a single-agent evolver must absorb growing trajectories, delayed labels, and prior research within one context window

    > The experiments further show that...multi-agent evolution constructs benchmark-specific capabilities

4 weeks back I switched over to Paseo and started using the agent profiles on a whim and have observed similar outcomes and behavior (seat of the pants): the win in my use case is that it feels like I can get higher autonomy without sacrificing quality (less input from me required to get quality output). The Emory paper and this team-solve seem to reinforce that role-based, multi-agent setups can improve solve performance.

For the curious, I shared a bit of my setup in a blog post[1]

[0] https://arxiv.org/pdf/2606.01770

[1] https://chrlschn.dev/blog/2026/10/larping-with-agents-and-wh...

Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models: haiku, sonnet, terra, sol, grok, muse spark,

Then take the updated plan in opus and forward to Astra and Fable, once you have the final plan in place execute.

Then have the same batch review code after created.

I am surprised how often one of the smaller models catch things. Despite so many things having eyes on.

The benefit of Claude is purely team collaboration and some basic QoL features.

I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.

I've been running something similar for few weeks, but with Qwen3.8-Flash-Next. I started with opus 4.8 limited to 200k context as the supervisor and 5 Qwen3.8-Flash-Next agents(262k context each, autocompacted). A very simple way of calling agents via a command line using pi.

Telling agents to maintain task status in files.

This has worked very well. I tried Qwen3.8-Flash-Next as the supervisor too and as long as the plan is written by a frontier model all the implementation can be done by cheap/local models.

When I have lots of code to write I use DeepSeek v4.1 Flash 1017, because my entire local Qwen3.8-Flash-Next throughput is 200tok/s across 7 sessions, while a single DeepSeek session runs at hundreds. 7 DeepSeek agents and one Qwen supervisor can complete substantial tasks fable needed 12h in 2h.

I haven't tried Qwen3.8-27b for coding. I've been using it mainly for interactive chat with Web tools via librechat. My benchmarks show it as weaker than Qwen3.8-Flash-Next by few points.