I’m not buying this argument, the apps of the LLM providers are some of the most wasteful apps that exist. Have you ever looked at how much memory / CPU Claude and Codex use? They ship with entire virtual machine hypervisors for sandboxing.

I think it’s simply a matter of dogfooding and new feature development, they do have much higher velocity than companies of equivalent size and an unlimited token budget helps as well.

This is completely orthogonal to the inference and training stack that the parent comment mentions

If it’s written by AI it’s gonna be the same overcomplicated overfit slop in one way or another. Performance pressure doesn’t fix that.

Assuming it’s entirely unsupervised, yes, which I can only assume to be the case at these volumes (hence the dogfooding).

I do believe this will force the AI labs to eventually solve the bloat problem with AI, but we’re definitely not there yet.

Depends on what the goal is and the skill of the engineer right?

Give a verifiable benchmark for the AI agent to hit such as reduce token processing time by 5% without affecting output.

Different metrics. You measure how many users you have, whether they’re closing the app and etc. when it comes to consumer level apps. ChatGPT/Claude is heavily used, not a problem for an average user, so no need to improve that.

Kernels, performance-required stacks etc. is different, so you optimise them for that.

I think Deepseek did this in the most focussed way.. their own optimization (key value caches, downscaled images for vision support etc) in their inference stack to push the boundaries in favor at a time of hardware scarcity.

It feels like OpenAI and Anthropic have a ton of room to optimizing that regard, but they choose to rather focus on moving the frontier forward than doing things that are already possible in a more efficient way.

Looking at their desktop apps, I don't think they're into optimizing.

They certainly are spewing forth large amounts of code, though.

The cost of running the apps to you is much less of a concern compared the cost for them running their own training.

But shouldn't it be easy for their great models to fix it?

I'm doing that now with gemini argon.

It's better than the ductape and glue pipelines that we had previous to this.

The software is the definition of plumbing no one cares about and testing it is trivial.