The way Antropic continues to get away with testing these nerfing "experiments" on paying customers without their consent or a discount is wrong.

They'd probably claim "well, it's not an A/B test then, because we are pre selecting for a specific user type who signs up for this".

I don't think that's an excuse, but it is likely the reasoning.

Probably won't work for Anthropic because they likely want a larger assortment of users and use cases.

Then they should suffer the consequences of not delivering what users paid for.

You would prefer they experiment on everyone at the same time without a control group? Think about your answer.

No, they should not change the model at all when you select the model.

No improvements to prompts, tooling, harness between model releases? Is this really what you want? I don't think you've thought this through at all.

[delayed]

Please make your substantive points without crossing into swipes or putdowns.

https://news.ycombinator.com/newsguidelines.html

I don’t think I said anything about prompts, tooling, or harnesses.

The model itself should not be substituted(for example, quantized) without an explicit SKU change.

Oh brother, do I have news for you...

They can get away with it because they are so far ahead of OpenAI models for coding, it's not like you can switch to anything else if you don't like it. Astra is even worse than nerfed Opus 5.5.

Pretty much the only alternative is loading up some credits and using Fable.

[delayed]

My experience is that 6.1 Sol is a little behind Opus 5.5 but it’s in spitting range. Their cost per task is generally higher, but if you’re on a quota plan the difference in cost doesn’t really matter until you’re paying API rates.

Both get the job done these days. I still hate Opus 5.5’s output, 6.1 Sol does an acceptable job of doing a second cleanup pass to it.

Astra when it hit the ground felt like “A Fable monent”: it was so good and concise and just was done with whatever task in a short time, where opus and fable sometimes can take forever. I think it lasted a week or two until it started to deteriorate to where we are now. Coincided nicely with the stopping of codex20 subscriptions etc.

Sol/Astra are sweet but they also have pretty much no discipline or executive function, which is where Opus 5.5 comes in, and I've not seen that capability in it nerfed too bad in my experience.

I have high hopes for Carbon here -- Argon is not quite Opus 5.5 but it's pretty darn good.

How can you even tell? Like, what's an experiment I could run right now which would would prove it to myself.

Yes this is the experiment I have been running for a while, that shows the constant comments on Claude models here have to be push from Claude team friends...The experience of output of OpenAI models is consistently better.

1)- Get a coding task and get the output of Opus and Astra.

2)- Open a new session and ask each other to review in detail the solution from the other model without mentioning where is it coming from.

3)- After the review, ask the original model to comment on it and state which solution is better and if the criticisms are valid...all in a new session.

4)- At the end Opus and others from Anthropic always end up agreeing the OpenAI model solution is better. My human review agrees.

After doing this for a few weeks...and consistently having their models rating higher the OpenAI solutions...I cancelled the Anthropic subscription...

Isn't this inherently task/domain/programming language/even prompting style specific? Why can't other people have genuinely different experience than you?

Nice try Amodei, but buying openai or anthropic credits is equivalent to just setting your money on fire.

DeepSeek, Qwen.

Yep, just one hour ago I again reached my weekly limit on claude code (3 days till reset btw), using Fable as Head-Orchestrator + Codex sub-agents via custom mcp.

I tried $20 of usage credits as a test once the limit was reached.

I shit you not, it lasted for less than 5 minutes.

Yeah it's a compete joke/scam. I bought $200 of credits once... gone in an hour. Never again. Part of me is thinking they decreased usage partly to force users to buy credits before the IPO... but at least in this case they ended up losing a user.

I get 5x more done (of equivalent quality) on my opencode go + DeepSeek flash sub for $40, than $200 anthropic sub.

Have you had a chance to try DeepSeek as a Orchestration/Manager for sub agents?

I have been using DeepSeek flash for a few projects and it is actually crazy capable, And I don't think I have spent $20 via api in a month yet.

Yeah it works great, maybe not quite as good as opus 5, but it's literally 20x-50x cheaper, and several times faster, an extra review/refinement loop per task brings the work up to opus level while still being much faster and cheaper overall. I have my own setup/tool (which I hope to open source soon) for planning/tasking/tracking which I found helpful to keep things on track an not forget/redo things - regardless of model - essentially tasks, dependencies/hierarchy, priorities, requirements, and task completion evidence tracked in the repo in structured plaintext/markdown files. DeepSeek was able to very easily understand and follow the workflow right off the bat.

Ok, thanks, I'm going to try to setup the pi harness and add on some needed extensions for my workflow. I'd appreaciate any advice in regards to any addons you would recommend if you have them.

If you aren't pay per token, then they can do whatever they want.

A bit like "You can order the steak for $75, or you can have whatever the chef is making for $30"

Eh sometimes it cuts the other way, like when Opus 5.5 was being shadow-routed on Opus 5. I am on a team plan and seem to not get any of these A/B tests.

You know what will never ab test your usage? Local ai. Just saying.

Yep, this.

And with Strata running at 50tok/s on my desktop is all I need. Would I like faster? Sure! But it's super useful in all sorts of tasks now.

I tried it. Nothing local or open weights can touch Fable/Opus/Gemini for my use cases.

What it did you try, and what are your use cases? Genuinely curious.

I am also curious what exactly you tried. Depending on your hardware it might also be worthwhile to stay in-tune with Qwen 4 and GLM 5.5/6.0 leaks.

Realistically it can't match Fable, let alone Opus today. It can however beat basically all of the models from 1 year ago. If last years state of the art is "good enough" for you today then local is great.

What open-weight models can match Opus 4.x or GPT 5.4 performance at a reasonable throughput?

Reasonable throughput depends a lot on your hardware. I'm running Qwen3.8-Flash-Next on a 128GB Strix Halo box at 50t/s gen (using the gufo inference engine) and it feels like Opus4.6/4.8 level performance. It's an amazingly good model. I've got it coding a C->microcode compiler in OCaml and it's doing great (there's also verilog code generation involved as well so this is a pretty good stress-test, I think).

Agreed.

Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.

Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.

If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.

Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.

From experience, qwen3.8 27b with gsq/rco at iq3_xxs has good performance with little intelligence loss.

Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.

Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.

That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.

I'd argue that Qwen3.8-Flash-Next is basically at Opus 4.6/4.8 level from about 6 months ago.

Maybe at Opus 4.6 level for the "easier" tasks. But honestly if you were given a choice between a computer with Qwen 3.8 or one with even older Sonnet 5, both local, same tok/s, window size, etc... what are you picking?

Last year's state was not good enough for me, but this year's state next year would be great

Interesting, mine freezes the whole laptop:-)

100% agree. A local model might not have the same stretch goal capabilities, but at least I don't have to constantly second guess myself in terms of the inconsistencies I think I'm noticing.

No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.

[delayed]

Assuming you built a top of the line PC a year or two ago or have a huge amount of money to drop on one now

And here I thought it was them taking away datacenter usage so they could drop a new model soon.

Do they ever test by giving the model more resources to see if they can give users more for the same price ? Just saying

They have lowered the price several times; and often released more for the same price. But it is normal to consider the best ever price to performance the benchmark and anything below that disappointing.

IPO is coming so they will have to maximize profits going forward which will mean constraining the compute they give everyone to ensure the subscriptions make money.

I don't think the subscriptions will ever make money.

I think the subscriptions are already making money.

But are they profitable?

That's generally what "making money" means. Profit, not revenue.

Based on API pricing per tokens, I'm getting anywhere from $10-20k/month in surplus value from a single $200 subscription.

How are they making money in your view?

By selling subscription tokens for less than they cost to generate, of course.

What you said doesn't yield any information whatsoever about whether or not subscriptions are profitable, all that says is that subscriptions are less profitable than API.

Things are able to be less profitable than other things while still being profitable.

If something like the API is very very profitable, it's even possible for profitable things to be much less profitable!

The subscriptions are a farm for generating training data.

This account claims Sonnet and Opus 5.6 are in A/B testing now. There is nerfing when they release a new version, for reallocating resources.

https://x.com/tokengremlin/status/2119783808773841101

how reliable is this twitter account?

i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof

They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...