They're also getting closer to IPO and have a growing user base. They can't justify losing a very large number of billions of other people's money in their IPO prospectus.

So there's a push for them to increase revenue per user, which brings us closer to the real cost of running these models.

I agree, and I'm also quite skeptical that Anthropic will be able to remain true to its initial, noble mission statement of acting for the global good once they IPO.

At that point you are beholden to your shareholders and no longer can eschew profit in favor of ethics.

Unfortunately, I think this is the beginning of the end of Anthropic and Modei being a company and CEO you could actually get behind and believe that they were trying to do "the right thing".

It will become an increasingly more cutthroat competition between Anthropic and OpenAI (and perhaps Google eventually if they can close the gap between their frontier models and Claude/GPT) to win market share and revenue.

Perhaps Amodei will eventually leave Anthropic too and start yet another AI startup because of Anthropic's seemingly inevitable prioritization of profit over safety.

I think the pivot to profit over good has been happening for a long time. See Dario hyping and salivating over all programming jobs disappearing in N months. He doesn't care at all if it's true or not. In fact he's in a terrible position to even understand if this is possible or not (probably hasn't coded for 10+ years). He's just in the business of selling tokens.

And worse, he (eventually) has to sell tokens above cost - which may have so much "baggage" (read: debt to pay Nvidia) that it'll be nearly impossible; or a new company will come to play with the latest and greatest hardware and undercut them.

Just how if Boeing was able to release a supersonic plane that was also twice as efficient tomorrow; it'd destroy any airline that was deep in debt for its current "now worthless" planes.

That's why open models are going to win in the long run.

I think the key question is “when”? In a highly competitive business environment, companies are going to naturally be attracted to the most capable model if it leads to a competitive advantage and the switching costs are low. This suggests that “open” (giving away inference despite ever-higher training costs) may not win for a very long time, if ever.

When frontier models plateau and efficiency increases sufficiently that it becomes a commodity like other cloud compute.

One driver of open models might be foreign actors. With the entire US economy being held up by AI, it's a crucial vulnerability for a capable foreign actor (guess who) to exploit if they wanted to.

Skeptical is a light way to put it. It is essentially a forgone conclusion that once a company IPOs, any veil that they might be working for the global good is entirely lifted.

A publicly traded company is legally obligated to go against the global good.

It’s not really, companies like GM used to boast about how well they treated their employees and communities. It was Jack Welch and a legion of like-minded arseholes who decided they should be increasingly richer no matter who or what paid for it.

See also HP. Pretty much only Costco left.

This is where PBCs (Public Benefit Companies) and B-Corps may have a role to play. Something like that seems necessary to enable both (A) sufficient profitability to support innovation and viability in a capitalist society and (B) consideration of the public good. Traditional public companies aren't just disincentivized from caring about externalities, they're legally required to maximize shareholder profits, full stop. Which IMHO is a big part of the reason companies ~always become "evil".

The company I currently work for is both a B-Corp and an employee-owned trust. The difference in culture, attitude and behaviour to the previous place I worked at, which only cared about quarterly results is stark.

Costco is such a strange and stark case standing in opposition to this general rule. From everything I hear, I can only gather that the reason is because of extremely experienced and level-headed executive staff.

The previous deal was due to (a) a lower level of development of capitalism (b) a higher profit margin that collapsed in the 70s (c) a communist movement that threatened capital into behaving

"Is your washroom breeding Bolsheviks?"

Middle class productive population produces commons goods and resources which gets exploited by Elites. Tragedy of the Commons applied to wealth generation process itself.

It’s funny how corporations get a bar wrap. Have you ever worked with private equity? Bad to worse.

Most PE is ironically ultimately owned by publicly traded funds. If you have a 401k that you’re not personally managing odds are that PE is where most of your gains come from.

The problem is that people equate money to power and power to evil.

So no matter what, if you do something lots of people like (and hence compensate you for), you will be evil.

It's a very interesting quirk of human intuition.

A reasonable conclusion, considering that money and power seem to have their own gravity, so people with more of both end up getting even more of both, and vice versa.

Can't blame someone who comes to such a conclusion about money and power.

The unreasonable part automatically labeling power as evil.

It’s a sane default to label power as evil in a society driven by greed, usury, and capital gain. Power tends to corrupt, particularly when the incentives driving its pursuit or sustenance undermine scruples or conscientiousness. It is difficult to see how power is not corrupting when it becomes an end in itself, rather than a means directed toward a worthy or noble purpose.

Labeling power evil is not automatic, its just making an observation of the common case. Money-backed power almost never works for the forces of good, and the people who claim they're gonna be good almost always end up being evil when they're rich and powerful enough. See also: Google.

Google is the company that created a class-less non-hierarchical internet. Everyone can get the same access to the same services regardless of wealth or personhood. Google is probably the most progressive company to ever exist, because money stops no one from being able to leverage google's products. Born in the bush of the Congo or high rise of Manhatten, you are granted the same google account with the same services. The cost of entry is just to be a human, one of the most sacrosanct pillars of progressive ideology.

Yet here they are, often considered on of the most evil companies on Earth. That's the interesting quirk.

Lot of people and companies were responsible for that. Anyway, that says nothing about what Google has become.

> Google is the company that created a class-less non-hierarchical internet.

Can you explain what you mean by this? I disagree but I don't understand how you think Google did this so I am very curious.

For my part, I started using the internet before Google, and I strongly hold the opinion that Google's greatest contribution to the internet was utterly destroying its peer to peer, free, open exchange model by being the largest proponent of centralizing and corporatizing the web.

The alternative was a teleco AOL style internet with pay tiers for access to select websites. The free web of the 90's would remain, but would be about as culturally relevant as Linux.

Surely you have to recognize the inconsistency of saying that Google "corporatized" the web, while the vast majority of people using google have never paid them anything. In fact many don't even load their ads or trackers, and still main a gmail account.

If we put on balance good things and evil things google has done, with honest intention, I struggle very hard to counter "gave the third world a full suite of computer programs and access to endless video knowledge for free with nothing more than dumpy hardware", while the evil is "conspired with credit card companies to find out what you are buying".

This might come off like I am just glazing google. But the point I am trying to illuminate is that when there is big money at play, people knee-jerk associate it with evil, and throw all nuance out the window.

Besides, IRC still exists for you and anyone else to use. Totally google free.

No I actually do understand where your opinion comes from now and I partially agree. I had forgotten about how badly the ISPs wanted the internet to mirror Cable TV plans.

There’s several subjects to go into here and HN probably isn’t the best place for the amount of detail this discussion requires but I will just note the amount of people blocking Google’s ads and trackers is negligible and has significantly shrunk in the mobile first era.

The wave is shifting to other corporations now but for a good while most of the internet was architected to give Google money. Remember SEO? An entire practice of web publishing centered around Google’s profit share. That hasn’t disappeared- it’s just evolved and transformed into more ingrained rent-seeking.

*Was

Money and power are good when used democratically to clearly benefit the majority of the people. They are bad otherwise. It is hard to see this because we live in such a regime that exists in the negative space seemingly without beginning or end. Other countries have different relationships to their population.

Fair point.

Call me an optimist, but I'm still holding out hope that Amodei is and still can do the right thing. That hope is fading fast though.

« Don’t be evil »

If no one can buy your soul, what's its value? Every Management Consulting Firm

> At that point you are beholden to your shareholders

No not really, you can issue two types of shares, the company founders can control a type of shares which has more voting power while other shareholders can get a different type of shares with less voting power.

Facebook, Google has something similar.

No, they still have to act in the interest of shareholders even if they have no voting power.

As a PBC, the intent of the company is not only profit, but it's hard to analyze the counterfactuals of if Anthropic were a pure for-profit or a non-profit

thats the benefit of a pbc

What will happen if they don't because the founders control the voting powe

They're also getting into cloud compute given you can use the desktop app to work in a temporary sandbox that they provision for you.

I was about to call it reselling but so many startups with their fingers in the tech startup pie offer containerised cloud compute akin to a loss leader. Harking back to the old days of buying clock time on a mainframe except you're getting it for free for a while.

> Harking back to the old days of buying clock time on a mainframe except you're getting it for free for a while.

I submitted this yesterday but it got no traction (I did not write it): https://www.mjeggleton.com/blog/AIs-mainframe-moment

The "real cost" of running near-SOTA models is not a secret: you can run local models on your own infrastructure. When you do, you quickly find out that typical agentic coding incurs outsized costs by literal orders of magnitude compared to the simple Q&A chat most people use AI for. All tokens are very much not created equal, and the typical coding token (large model, large noisy context) costs a lot even under best-case caching scenarios.

I meant reference Toby Ord's work here. I think his framing of the performance/cost frontier hasn't gotten enough attention https://www.tobyord.com/writing/hourly-costs-for-ai-agents

That post doesn't address the human factor of cost, and I don't mean that in a good way. Even if AI costs more than a human, it's tireless, doesn't need holidays, is never going to have to go to HR for sexual harassment issues, won't show up hungover or need an advance to pay for a dying relative's surgery. It can be turned on and off with the flip of a switch. Hire 30 today, fire 25 of them next week. Spin another 5 up just before the trade show demo needs to go out and fire them with no remorse afterwards.

That was a great promise before the models starting becoming "moody" due to their proprietors arbitrarily modifying their performance capabilities and defaults without transparency or recourse.

I still haven't seen any statistically sound data supporting that this is happening on the API (per-token pricing.)

If you've got something to share I'd love to see it.

There's an interesting analysis here: https://github.com/anthropics/claude-code/issues/42796

>The most striking row is user prompts: 5,608 in February vs 5,701 in March. The human put in the same effort. But the model consumed 80x more API requests and 64x more output tokens to produce demonstrably worse results.

Sorry, "this" referred to the parent comment's claim.

> models starting becoming "moody" due to their proprietors arbitrarily modifying their performance capabilities

The tokenizer changes are measurable, the above is quite difficult to quantify.

There are a few sites floating around that purport to, but all of them have fatal flaws in their methodology.

Unfortunately, LLM performance isn't an exact science and some observations are going to be subjective. Observations like ChatGPT being "lazy" in the Winter. Wanting to form opinions based on hard data, aka science, and not vibes is entirely reasonable but doesn't make the vibes a figment of imagination. Or as Jeff Bezos put it, "When the data and the anecdotes disagree, the anecdotes are usually right." And while he's not a scientist, his success does put some weight behind that quote. (as does digging deeper in what he meant by that.)

I think it's difficult to say agentic and human developer labor are fungible in the real world at this point. Agents may succeed in discrete tasks, like those in a benchmark assessment, but those requiring a larger context window (i.e. working in brownfield systems, which is arguably the bulk of development work) favor developers for now. Not to mention that at this point a lot of necessary context is not encoded in an enterprise system, but lives in people's heads.

I'd also flip your framing on its head. One of the advantages of human labor over agents is accountability. Someone needs to own the work at the end of the day, and the incentive alignment is stronger for humans given that there is a real cost to being fired.

For some the appeal of agent over human is the lack of accountability. “Agent, find me ten targets in iran to blow up” - “Okay, great idea! This military strike isn’t just innovative - it’s game changing! A reddit comment from ten years ago says that military often uses schools to hide weapons, so here is a list of the ten most crowded schools in Iran”

It must be wild to actually go through life believing the things written in this post and also thinking you have a rational worldview.

I think the word you're looking for is contractors. But yes, you still have to treat those with _some_ human decency.

Ah-ha, the perfect slave.

it just will delete production database when flustered. no biggie. we learning how to socialize again. cant let all that history go to waste.

More importantly it collapses mythical-man-month communication overhead.

This is an amazing frame /reframe.

Hang on, tell me how, because I am not picking up what you are putting down. At a minimum, wouldn’t this require working from a perfectly written spec that has already accounted for the discovery of changes that would need to be made from the original perfect spec?

The way I see it, because you can spin up additional AI employees at will (and spin them back down), when the problem with the spec is found, it's no big deal to redo all of that work from before, adjusting for that change.

Ironically, people keep saying this, but then gloss over the core problem of coordinated between these agents... For completely independent codebases with no dependencies, sure thing right on...go for it. But the vast majority of F500 companies I work with have wild and undocumented dependencies between almost every system that will take years to "agentify" (assuming they ever figure out that it's an organization and governance problem.... which they might not ever realize)

So we have two things here:

1. "Mythical Man Month" which is the shorthand for a whole book + concept that you can't just throw more people at a software development project and get linear productivity improvements as the communications overhead (meetings, emails, mistakes due to poor assumptions, etc.) deeply eat into the raw number of productive hours that a new person added to the team brings.

2. AI automation tools (Claude Code) are often described as a "junior developer" which is an imperfect comparison as while you could potentially sort of set them up that way many people use them as more of a singular force multiplier.

I use them to work on many more projects in many more ways and ship far more than I could even if I had a "junior developer" sitting alongside of me as there's not the same level of communication needed.

The cost to hire a human is highly predictable. The cost of AI isn't. I, as a human, need food and shelter, which puts a ceiling to my bargaining power. I can't withdraw my labour indefinitely.

The power dynamics are also vastly against me. I represent a fraction of my employer's labour, but my employer represents 100% of my income.

That dynamic is totally inverted with AI. You are a rounding error on their revenue sheet, they have a monopoly on your work throughput. How do you budget an workforce that could turn 20% more expensive overnight?

By continuously testing competitors and local LLMs? The reason for rising prices is that they (Anthropic) probably realized that they have reached a ceiling of what LLMs are capable of, and while it's a lot, it is still not a big moat and it's definitely not intelligence.

Anything but the simplest tooling is not transferable between model generations, let alone completely different families.

> Anything but the simplest tooling is not transferable between model generations, let alone completely different families.

It is transferable-yes, you will get issues if you take prompts and workflows tuned for one model and send them to another unchanged. But, most of the time, fixing it is just tinkering with some prompt templates

People port solutions between models all the time. It takes some work, but the amount of work involved is tractable

Plus: this is absolutely the kind of task a coding agent can accelerate

The biggest risk is if your solution is at the frontier of capability, and a competing model (even another frontier model) just can’t do it. But a lot of use cases, that isn’t the case. And even if that is the case today, decent odds in a few more months it won’t be

Ha. Sounds a lot like the one 10x vs. predictable mediocre guys with a scaffolding of processes. Aim high and hit or miss or try to grind predictably and continuously. Same with humans and depends on the loss you can afford.

Yep. My approach has been, if I can’t reliably get something to 90+% with a flash / nano / haiku, then it’s not viable for any accuracy critical work. (I don’t know of or have the luck of having any other work.) Starting out with the pro / opus for any production classification work has always been a trick.

If you're talking about APIs and SDKs, whether direct API calls or driving tools like Claude code or codex with human out of the loop, I think that's actually fairly straightforward to switch between the various tools.

If you're talking about output quality, then yeah, that's not as easy. But for product outputs (building a customer service agent or something like that), having a well-designed eval harness and doing testing and iteration can get you some degree of convergence between the models of similar generations. Coding is similar (iterate, measure), but less easy to eval.

It's not that hard to make it generic. It does take a little work, but really it boils down to figuring out how to make things work with the "dumbest" model in your set.

For most tasks, at some future date, isn't there going to be some ambient baseline of capabilities you can get per $/tok, starting at ~0 for OSS models, such that eventually all tooling gets trivially transferable?

Note that it is very likely this market can't sustain this level of competition for long. We are all still chasing the carrot of AGI, while hardware costs skyrocket.

The same way companies already deal with any cost.

> The cost of AI isn't.

This is why there are a ton of corps running the open source models in house... Known costs, known performance, upgrade as you see fit. The consumer backlash against 4o was noted by a few orgs, and they saw the writing on the wall... they didnt want to develop against a platform built on quicksand (see openweb, apps on Facebook and a host of other examples).

There are people out there making smart AI business decisions, to have control over performance and costs.

> How do you budget an workforce that could turn 20% more expensive overnight?

Like, say, oil or DRAMs?

Exactly. Big headaches. It doesn't happen to the salaries of the employees of the companies affected by those price hikes. That's the point.

Its countered by competitors for inference. You could locally host a model and have your cost be fixed by your infra costs

Why do you think it can't sexually harass someone or drive people to suicide. There are already lawsuits coming in on it causing suicides.

This is an architecture that people are increasing begging to give network connectivity that can't differentiate its system prompt from user input

I thought it already was used to sexually harass people by creating naked versions of them.

As the NRA puts it, guns don't kill people, people kill people. Yes, a harasser now has access to more powerful tools to sexual harass somebody, but Claude, the AI, isn't going to randomly grab the secretary's ass and have to be fired for having done so at the holiday Christmas party.

Let's give that one a SCP* re-up: https://news.ycombinator.com/item?id=47778922

(* explained at https://news.ycombinator.com/item?id=26998308)

> The fact that Anthropic is rapidly trying to increase price may betray the fact that their recent lead is at the cost of dramatically higher operating costs.

Or they are just not willing to burn obscene levels of capital like OpenAI.

I mean, the signs have been there that the costs to run and operate these models wasn't as simple as inference costs. And the signs were there (and, arguably, are still there) that it costs way, way more than many people like to claim on the part of Anthropic. So to me this price hike is not at all surprising. It was going to come eventually, and I suspect it's nowhere near over. It wouldn't surprise me if in 2-3 years the "max" plan is $800 or $2000 even.

> It wouldn't surprise me if in 2-3 years the "max" plan is $800 or $2000 even.

I'd rather be surprised if they are still doing business by then.

I would not be surprised at all, a $1,000/mo tool that makes your $20,000/mo engineer a lot more productive is an easy sell.

I’m guessing we’re gonna have a world like working on cars - most people won’t have expensive tools (ex a full hydraulic lift) for personal stuff, they are gonna have to make do with lesser tools.

noway.

i bought a $3k AMD395+ under the Sam Altman price hike and its got a local model that readily accomplishes medial tasks.

theres a ceiling to these price hikes because open weights will keep popping up as competitors tey to advertise their wares.

sure, we POV different capabilities but theres definitely not that much cash in propfietary models for their indererminance

No engineer will cost 20.000 bucks a month at this point in time. Offshoring is still happening aggressively.

What about when there is a $100/month tool that makes your engineer 90% as productive as they were on the $1000/mo tool?

What if that tool is something you can run on prem, and over time make the investment back?

It's not so simple.

If your company is making $1 mil per employee per year, then 10% is 100k. Even at 500k employee or lesseer numbers it's almost always better to buy the $1000/month tool (break even is a measly $108k revenue per employee per year)

It's not just about cost, it's about having the control, stability, and autonomy of on-prem. Plus you can probably repurpose that compute when employees are out of the office.

Anyways, I'm just saying it's not so simple ;)

> It's not really clear whether Opus 4.5+ represent a level shift on this frontier or just inhabits place on that curve which delivers higher performance, but at rapidly diminishing returns to inference cost.

I think we're reaching the point where more developers need to start right-sizing the model and effort level to the task. It was easy to get comfortable with using the best model at the highest setting for everything for a while, but as the models continue to scale and reasoning token budgets grow, that's no longer a safe default unless you have unlimited budgets.

I welcome the idea of having multiple points on this curve that I can choose from. depending on the task. I'd welcome an option to have an even larger model that I could pull out for complex and important tasks, even if I had to let it run for 60 minutes in the background and made my entire 5-hour token quota disappear in one question.

I know not everyone wants this mental overhead, though. I predict we'll see more attempts at smart routing to different models depending on the task, along with the predictable complaints from everyone when the results are less than predictable.

> It was easy to get comfortable with using the best model at the highest setting for everything for a while, but as the models continue to scale and reasoning token budgets grow, that's no longer a safe default unless you have unlimited budgets.

For a while I used Cerebras Code for 50 USD a month with them running a GLM model and giving you millions of tokens per day. It did a lot of heavy lifting in a software migration I was doing at the time (and made it DOABLE in the first place), BUT there were about 10 different places where the migration got fucked up and had to manually be fixed - files left over after refactoring (what's worse, duplicated ones basically), some constants and routes that are dead code, some development pages that weren't removed when they were superseded by others and so on.

I would say that Claude Code with throwing Opus at most problems (and it using Sonnet or Haiku for sub-agents for simple and well specified tasks) is actually way better, simply because it fucks things up less often and review iterations at least catch when things are going wrong like that. Worse models (and pretty much every one that I can afford to launch locally, even ones that need around ~80 GB of VRAM in the context of an org wanting to self-host stuff) will be confidently wrong and place time bombs in your codebases that you won't even be aware of if you don't pay enough attention to everything - even when the task was rote bullshit that any model worth its salt should have resolved with 0 issues.

My fear is that models that would let me truly be as productive as I want with any degree of confidence might be Mythos tier and the economics of that just wouldn't work out.

The GLM-4.7 model isn't that great. I was on their $200/month plan for a while. It was really hard to keep up with how fast it works. Going back to Claude seems like everything takes forever. GLM got much better in 5.1 but Cerebras still doesn't offer that yet (it's a bit heavier). I have a year of Z.ai that I got as a bargain and I use GLM-5.1 for some open source stuff but I am a bit nervous about sending data into their API.

The new one is quite a bit heavier!

GLM 4.7 is 358B parameters: https://huggingface.co/zai-org/GLM-4.7

GLM 5.1 is 754B parameters: https://huggingface.co/zai-org/GLM-5.1

That said, 5.1 is indeed a bunch better and I could definitely see myself using it for some tasks! Sadly all of the stuff I can actually run locally is still trash (I appreciate the effort behind Qwen 3.6, Gemma 4 and Mistral Small 4 though, alongside others).

Good points. I was speaking from a position of using an LLM in a pair programming style where I'm interactive with each request.

For handing work off to an LLM in large chunks, picking the best model available is the only way to go right now.

I have this exact same fear as an IC.

I wonder if Engineering Managers have this same fear, or they’re used to having to distribute complex tasks to senior engineers and gamble with seeming less risky tasks to juniors that may leave ticking time bombs in their code. Just the nature of code written by agents or humans?

Yes, that is absolutely a dynamic in managing an engineering team, and I'd argue that knowing the right person to give a particular task to, and how much detail they're going to need to get it done, is what separates good engineering managers from bad ones.

Yes, that definitely happens as an EM. You want your Senior/Staff engineers to architect out the new high-risk functionality into a doc for review. Then that Staff engineer either implements or has a junior/senior under their wing helping implement some of the scaffolding.

In this [common] paradigm the Staff Engineer acts as a architect/programmer and project manager in one. The EM should be there to guide and unblock.

> I know not everyone wants this mental overhead, though.

I’m curious how to even do it. I have no idea how to choose which model to use in advance of a given task, regardless of the mental overhead.

And unless you can predict perfectly what you need, there’s going to be some overuse due to choosing the wrong model and having to redo some work with a better model, I assume?

Human dev labor cost is still the high pole in the tent, even multiplying today's subsidized subscription cost by 10x. If the capability improvement trajectory continues, developers should prepare for a new economy where more productivity is achieved by fewer devs by shifting substantial labor budget to AI.

I'm getting a lot more done by handing off the code writing parts of my tasks to many agents running simultaneously. But my attention still has its limits.

Your employer doesn’t pay the subscription cost, they pay per token. So it’s already way more than 10x the cost.

Depends on the type of subscription. We have Codex Team and have a monthly subscription, no per-token costs.

Isn't that essentially GPT Pro Extended Thinking?

Except developers can’t even do that. Estimation of any not-small task that hasn’t been done before is essentially a random guess.

That's why you split tasks and do project management 101.

That's how things worked pre-AI, and old problems are new problems again.

When you run any bigger project, you have senior folks who tackle hardest parts of it, experienced folks who can churn out massive amounts of code, junior folks who target smaller/simpler/better scoped problems, etc.

We don't default to tell the most senior engineer "you solve all of those problems". But they're often involved in evaluation/scoping down/breakdown of problem/supervising/correcting/etc.

There's tons of analogies and decades of industry experience to apply here.

Yeah... you split tasks into consecutively smaller tasks until it's estimateable.

I'm not saying that can't be done, but taking a large task that hasn't been broken down needs, you guessed it, a powerful agent. that's your senior engineer who can figure out the rote parts, the medium parts, and the thorny parts.

the goal isn't to have an engineer do that. we should still be throwing powerful agents at a problem, they should just be delegating the work more efficiently.

throwing either an engineer or an agent at any unexplored work means you just have to delegate the most experienced resource to, or suffer the consequences.

I don't completely agree. Estimation is nontrivial, but not necessarily a random guess. Teams of human engineers have been doing this for decades -- not always with great success, but better than random. Deciding whether to put an intern or your best staff engineer on a problem is a challenge known to any engineering manager and TPM.

or tech lead. or whoever. the point is, someone has to do the sizing. I think applying an underpowered agent to a task of unknown size is about as good as getting the intern to do it.

Even EMs and TPMs are assigning people based on their previous experience, which generally boils down to "i've seen this task before and I know what's involved," "this task is small, and I know what's involved," or "this task is too big and needs to be understood better."

We should lean on the LLMs to do estimation. Claude loves to estimate that a plan will take two weeks, then implements the whole thing in 45 minutes.

The problem is half the time you don't know you need the better model until the lesser model has made a massive mess. Then you have to do it again on the good model, wasting money. The "auto" modes don't seem to do a good job at picking a model IME.

The problem is that without a lot of human involvement (which is even more expensive), even the best still make a lot of mistakes.

I'd be fine with right-sizing the model and effort if there was any semblance of reasoning stability. I thought we were getting there a couple months ago, but last night Opus couldn't even insert a new item in a numbered list without getting confused and ending up with a duplicate item and the wrong number order.

heh adaptive thinking is letting the meter run itself. they make more when it runs longer.

That sounds very plausible. But it implies they could offer even higher performance models at much higher costs if they chose to; and presumably they would if there were customers willing to pay. Is that the case? Surely there are a decent number of customers who’d be willing to pay more, much more, to get the very best LLMs possible.

Like, Apple computers are already quite pricey -- $1000 or $2000 or so for a decent one. But you can spec up one that’s a bit better (not really that much better) and they’ll charge you $10K, $20K, $30K. Some customers want that and many are willing to pay for it.

Is there an equivalent ultra-high-end LLM you can have if you’re willing to pay? Or does it not exist because it would cost too much to train?

> Is there an equivalent ultra-high-end LLM you can have if you’re willing to pay? Or does it not exist because it would cost too much to train?

I guess at the time that was GPT-4.5. I don't think people used it a lot because it was crazy expensive, and not that much better than the rest of the crop.

The issue is not better - it’s better _AND_ fast enough. An agentic loop is essentially [think,verify] in a loop - i.e. [t1,v1,t2,v2,t3,v3,…] A model that does [t1,t2,t3,t4] in 40 minutes, if verify takes 10 min, will most likely do MUCH worse that a model that does t1 (decently worse) in 10 mins, v1 in 10 mins, t2 now based on t1 and v1 in 10 mins, v2 in 10 mins, etc..

So, for agentic workflows - ones where the model gets feedback from tools, etc…, fast enough is important.

> As of right now that's basically Mythos. If the rumors are true, it's a 10 trillion parameter model which is QUITE a bit larger than other models. If so, I'd suspect the main reason it has a super limited roll out is that they can't afford to run it at scale yet.

Yeah. Combine this with much of Corpos right now using a “burn as many tokens as you need” policy on AI, the incentive is there for them to raise price and find an equilibrium point or at least reduce the bleed.

Once they implement their models directly in silicon, the cost will come down and the speed will go up. See Taalas.

taalas is amazing. id gladly spend 5-15k on something that matched that performance with opus 4.6 quality

This is a bad take. It's not really wrong in the sense that yes higher performance does cost more.

But it ignores completely the fact that the same intelligence is dropping by an order of magnitude (at least) every 12 months.

GPT o1 launched at $600/M output tokens and GPT4.5 launched at $150/M.

Opus 4.7 is $25/M for more intelligence

What a well thought and written comment. I totally agree.

Also appreciate the author for doing the analysis and sharing the results

[delayed]

FWIW, Artificial Analysis has a "Intelligence vs Cost" plot on their front page that shows models' score vs cost to run the benchmark, which should be more fair in this sense. According to that one, Opus 4.7 (max) is slightly cheaper than 4.6 (though still very expensive).