The capabilities of all models increasing so much all the time means there are simply less and less tasks you need a frontier model for.
Even if Opus 5.5 is 500x better than Deepseek, if deepseek can solve all my problems, why do I need to pay for more?
Obviously any model will do if you use it as a better autocomplete.
I believe that there is a large gap in expectations between different workflows.
Until the AI like reads my mind and produces perfectly production ready apps with minimal intervention from my side, there is still going to be room for improvement.
Uhm, yes?
It's about 12000 lines of code. I could probably understand every line of it if I spent a couple of months on it, although I'd need to learn some things about WebGL, numerical computation, and solid-state physics. But I elicited it in four days. Probably I'd be better off spending the next couple of months doing something else instead.
So, I was faced with the question of how I could keep the artifact thus created from being completely valueless. My solution was to export the heat-equation solution produced by the solver as (gzipped!) CSV, so that I can use my own code (that I do understand every line of!) to check that the solutions found by the solver are, in fact, solutions.
If that check checks out, the backward Euler solver may be of some value even if I don't understand every line of it.
Now my provider serves DeepSeek around 300 tokens a second. The code is shit but I can have a few more rounds of corrections.
DeepSeek was maybe a dollar for the full PR. Opus/Astra over 50 dollars. And double that if you use their fast variants which matches the DeepSeek speeds on certain providers.
Yes, it is so good I rarely test new models anymore. Or think about cost.
Actually I felt since 4.5, it was “good enough” for basic coding (and competing models have reached that level as well). The big improvements since then have been longer-horizon tasks. Giving it a vague description of a problem, then come back hours later and find it did a better job than I would have. And since 5.5 I haven’t encountered any more load bearing seams.
Maybe it’s slow, but I’m not interacting with the models in a live chat anymore for most work. I’m giving them tasks to do async. The limiting factor is actually my bandwidth reviewing their output.
I am eyeing a huge codebase built with Opus. Did multiple passes of refactoring for things like using traits not just free functions. And using the Rust types like PathBuf from the damn standard library not just strings everywhere.
Oh and it is so hard to use a crate. Let's just vibe an inferior version and push another 5k line PR. Nobody reads them anyway right? Logos schlogos, a lexer is easy to write, right?
Maybe I am just getting old, but I remember a time when we were proud on the code we wrote. Now I read some of the old stuff and get a bit sad for what we lost.
Don't get me wrong DeepSeek also does this. But if the model is 100x more expensive I expect better. Of course if you never read the code it's ok. But it does the same mistakes and cannot design great architecture...
This morning I elicited a microkernel operating system from Opus 5.5. Well, mostly. It doesn't implement task switching yet; we'll see if it runs into a wall at some point. But it boots in QEMU, and it's running a user process in ring 3 and serving web pages.
But if on the other hand, I mostly use my human intelligence and just need a dumb model to complement my human intelligence at low cost and high speed (say review every commit to catch obvious bugs), I have a much better chance of building an actual moat than you do.
But outside of coding, it’s even more clear that you don’t need frontier intelligence. My customer service agent is very happy with a 100B param Deepseek flash model, thank you!
I’m using subscription models for exactly that, better models catch more subtle bugs, and they catch them faster. It works out far better in terms of work-hours saved.
Also A/ then OAI slashed token pricing by 2x~5x on their latest models
I have enough real problems in life. I don't need to invent new ones just because a new technology is available.
Many of my problems in life are fully solved far past my satiation point by a 3b model that costs me nothing to run.
Many others are not.
But in either case, when I am acting and living wisely, almost all of my problems exist prior to the existence of technological solutions to those problems.
This is also true for the customers and employers that I care to work with. This has changed in me over time, but I now try my best to avoid inventing new problems. The world has enough big, important problems already.
> This morning I elicited a microkernel operating system from Opus 5.5.
This is cool but also a good example. I don't need a personalized microkernel just because it's possible to have one.
Maybe I need one and I don't know it, but the problem statement definitely isn't "I have inherent desire for a personalized microkernel".
You're talking about definition 1 in https://en.wiktionary.org/wiki/problem, "A difficulty that has to be resolved or dealt with," with the examples given being racism, addictions, and lack of access to health care. Those aren't the kind of problems AI can help with.
The kind of "problem" that AI can help with is definition 2, "A question to be answered, schoolwork exercise." That is the character of engineering "problems", although they are more open-ended than schoolwork exercises, because there are many defensible tradeoffs. "How can I build a bridge here?" or "How can I improve the fuel efficiency of this vehicle?" is a "problem" in the sense of a question to be answered with knowledge and effort, not in the sense of being similar to racism or addiction.
If the questions you're thinking of are so easy to answer that they can be easily answered by a hypothetical AI model 1/500th as good as Opus 5.5 — well, think up some harder questions.
> well, think up some harder questions.
There's nothing wrong with this mindset, but your insistence that other people must have your values is off-putting.
To wit: at the end of my phd I was offered several tenure-track positions where "work on problems that are hard" would've been a big part of my job.
I turned those offers down due to my distaste with "because it's hard" as a justification for working on problems.
I'm much more interested in helping people and solving (def'n 1) problems. Those problems are almost always too large and messy to one-shot, but very often contain many components that can be solved with non-sota models, and make sense to solve with non-sota models for any number of economic/scaling reasons. E.g.: "can I serve this thing that costs $0.25 per run on even c.2024 sota models to 100,000 K12 students who will use it dozens of times a day on a non-profit's budget of ~$0.00."
P.S. The "problem" definition thing is very confusing to your readers because the person you responded to was clearly using the first definition! It would have been helpful for you to sign-post that you are using the same word as the person you are replying to but changing the meaning out from under your reader.
I feel like the "good enough" argument isn't about how big the gap between models is but about how good they are at solving the tasks at hand.
The capabilities of all models increasing so much all the time means there are simply less and less tasks you need a frontier model for.
Even if Opus 5.5 is 500x better than Deepseek, if deepseek can solve all my problems, why do I need to pay for more?
They are clearly talking about "problems" that DeepSeek can solve, and they seem to be using the term "task" as a synonym for "problem", even using the problem verb "solve" with "tasks" as its object. To me, that makes it clear that they are not talking about problem₁s such as racism, addiction, and lack of access to health insurance; while you could imagine "solving" such problems, they are not tasks, and DeepSeek certainly cannot solve them. Rather, I think they are talking about "engineering problems", which are examples of problem₂, more analogous to math questions (to be diligently sought out) than to troubles (to be avoided).
It is certainly the case that the proposition that such problems should be diligently sought out is a value judgment, and therefore it derives from the value system employed by me, the writer. Would you prefer that I refrain from making normative statements in general, or just those with which you might disagree? In either case, is your preference specific to me personally, or do you consider it a duty incumbent upon everyone?
— ⁂ —
The 01913 edition of Webster's dictionary offers:
1. A question proposed for solution; a matter stated for examination or proof; hence, a matter difficult of solution or settlement; a doubtful case; a question involving doubt. --Bacon.
2. (Math.) Anything which is required to be done; as, in geometry, to bisect a line, to draw a perpendicular; or, in algebra, to find an unknown quantity.
Note: Problem differs from theorem in this, that a problem is something to be done, as to bisect a triangle, to describe a circle, etc.; a theorem is something to be proved, as that all the angles of a triangle are equal to two right angles.
The absence of the "trouble" definition suggests that perhaps it is derived from the "question" definition, and, surprisingly, very recently.
You gotta problem wit that?
Returning to the substantive matter, I'm happy to try to re-state the topic at hand using your preferred symbols.
Suppose:
1. Someone has a type of task to be done.
2. Finding a systematic way to complete that type of task has immediate value.
3. That type of task can be solved at lower per-unit cost without paying for tokens from the latest Anthropic or OpenAI model release.
Then why would you tell that person that they need to "think of better problems"? They have a task to be done, and they can do that task. Assuming the task has sufficient value to justify the time invested in crafting a solution, I'm not sure I see the need for "better" problems.
There is nothing wrong with being the type of consumer that adopts a product early and then works on things to which that product is congruent.
But there is also nothing wrong with using a plain old nail gun to drive a plain old nail into two pieces of plain old wood that someone with money needs fastened together.
Stick framing may be a "solved" "problem". The task of building individual houses remains at hand. The carpenters framing those houses do not need "better problems" to "solve". They would probably benefit from lower per-unit cost of nails or lumber, though.
I hope this makes the meaning of my position clear.
Look at the context in which I used that term 'good enough'.
What i was saying is that there are tasks for which a dumber model can be good enough, and for organizations with sovereignty/ privacy concerns, those concerns can be strong enough to incentivize the use of a dumber model.
I had the exact same experience. And unlike Fable, it doesn't gobble up your entire usage limit in a few hours.
I always wonder what the "good enough" people are actually using it for.
If Opus can’t 1-shot it, then it must rely on our human intelligence which can be complemented well enough with a dumb model as a frontier model.
As for Mistral - I got really excited when they said Large 4 was focusing on being #1 in cybersecurity, because that's somewhere that they genuinely could edge out Anthropic & OpenAI. Have it actually solve problems, instead of Anthropic flagging "you tried to find a null pointer exception bug in your own code, we're now reporting you to the US government". But on the Mistral benchmarks I'm seeing, this looks very disappointing, but at least they haven't entirely given up. I genuinely thought Mistral had given up on new general models. They need to learn the bitter lesson all over again.