Implicated

About 6 months ago I embarked upon building a little mac native app to monitor all my claude code sessions and alert me/surface sessions that were going to go stale so I could go ping them if I wanted to keep things cached. I named it CREAM.

That eventually grew into what is now my 'software factory' and I've realized, eventually, it's still C.R.E.A.M. (I named it flower) but it's _context_ rules everything around me.

> Even then, 170GB/s is a bump up from 153GB/s of the M5. So we see a 11% uplift already.

My M2 Pro does 200GB/s.

wtf are we even talking about here.

I literally launched my development and web career on the back of this service as a broke early-20s person. Just seeing that link on HN gave me goosebumps. <3

Ask claude to do it end to end, then write a solution that fits your needs from it's experience with you driving it.

I did exactly this - probably the single most valuable piece of software I've "written". It's the backbone of every piece of code coming out of any agent under my control (via MCP toolset).

Knives are dangerous.

> But, my biggest annoyance with most non-flash model providers - including GLM is they will dumb down the models based on load (like switching a non-flash model to flash model).

Where are you seeing this? This would be sufficient for me to stop using that provider entirely, immediately.

seems like maybe we're all just not as mature as you, unfortunate for you.

> not to clone someone else's work and release it for less money.

... You're saying this about _deepseek_?

When I read things like this all I can imagine is a farmer responding in a similar fashion to people talking about using tractors.

I picked up a pre-ai-price-insanity AX162-R at Hetzner a while back and loaded it up on memory to max out the 12 channels the 48c EPYC 9454P as a "this will be the last mysql box I'll need" and have I been _wildly_ impressed with it's performance. The things I throw at it are honestly laughable at times, wildly irresponsible queries against a rather large database, the redis qps metrics are ridiculous and I just load it up with random ggufs since the memory bandwidth is... not terrible and 384GB of it is... useful.

The web app that's hosted on it deals with lots of images and text - over 100mil of each deduplicated, embedded, simhashed - it does the hashing, and the embedding in real time during ingest. Just handles it.

These things are absolutely insane.

> I did not personally test the open weight models beyond the old Qwen 3.6 27B, which produced unusably bad results for me.

So you don't have much perspective on things, it seems. Let me introduce you to the GLM 5.2 and then 5.3/5.3 flash series of... "oh, wow, I should have bought some RTX PRO 6000's while they were 'cheap'" stage of progression.

As someone carrying multiple max subscriptions to both claude and codex - primary workhorse is glm 5.3 flash running on rented GPUs for less than a latte/hr.

I also found qwen 3.6 27B nearly useless for my own needs. DS4 flash 0731 and then 4.1 have been nearly as eye opening as glm 5.3 flash, but have their own warts.

> it seems like for law specifically all relevant facts will be cited and checked easily by humans.

Don't be too sure about that. [0]

0: https://www.damiencharlotin.com/hallucinations/

I don't have a horse in this race but I would question the idea that a municipal service has a better reliability record than x/twitter.

> Sol is so much better than Fable 5.

... looks around ...

I think you're arguing the same general point that the person you're responding to is. But you're saying he's not understanding - he understands that they report a cache hit % but you can't look at that public metric with any level of accuracy _because_ most people aren't pinning their providers and they _are_ getting juggled around which is bringing that metric down. That's not to say that specific providers might have issues or worse cache implementations - but it stands that if openrouter is juggling the requests back and forth by default then _that alone_ is breaking caches on those requests in huge numbers.

> OpenRouter randomizes which provider gets your request by default right?

I'm not sure it's wholey accurate to say they "randomize" the provider, rather my assumption based on usage is that it's something like cheapest-ish/responded to the request within some reasonable-ish time/etc algorithm that chooses the provider on each request - which seems, remarkably questionable in terms of optimizing for user experience or hidden user costs.

> This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.

I so very much recommend this approach. My avenues that automate llm calls to openrouter are setup to make api reqs to openrouter to determine best price/response/etc and then pin the request to that (and, preferably, a fallback if there's reasonable difference between #1 and #2) provider for that session. Otherwise you're going to have a bad time.

I'd imagine this could make things interesting in cases where one provider is offering different quants than the others and openrouter is just swapping you back and forth on a long agentic session.

> it ran like shit. Very slow (~20tps, VERY high latency) and it would timeout all the time.

Boy, do you have a rude awakening in store.

> and it was running very slowly

... I'm at a loss for words here. It was being served for free. To the entire world.

> All the American companies you mentioned still follow American law and regulation. Skirting that blatantly has big consequences. > Chinese companies do not follow American laws and there are absolutely no consequences for violating it.

... lmk when anthropic/openai/spacex/xai are held accountable for anything. Anything at all. Hard to be when you're _writing_ the rules.

At the current point in time I'd argue it's more about opportunity cost/value.

If I'm a professional photographer chasing the best possible end product, I'm not buying cameras because they're economical. I'm buying the best camera I can get my hands on to get the best product I can produce within reason under the understanding that it doesn't have to equate to the best economic decision to be the _right_ decision.

If you're in a position to be able to take advantage of the local inference - it's a no brainer. If you're not sure how that would be done, then it's not a good move.

If your usage wouldn't change with local inference and you don't have security/privacy concerns then at the currently heavily subsidized pricing, sure.. not economical.

But things change real fast when you're no longer bound by costs/apis/rate limits. All of a sudden it's not about "how can I do this right and efficiently" and more about "I can poke at and test _all the things_ that might make this better".

I think most people who can't see this value in the local inference approach are likely still copy/pasting from their web LLM ui's or don't even come close to subscription quotas. Meanwhile, 1b tokens a day is a light day for me with 3 $200/m subscriptions + some level of sub at basically every frontier level provider. Had I been less frugal and ponied up for the hardware before things got crazy I wouldn't need 80% of that - just the frontier models for the most complex tasks, the open weight models would handle the rest easily _and_ I'd get to do a lot more exploratory work without concern about quotas.

Use Opus 4.8. 5 is absolute garbage.

Don't use DS4 Flash in max effort mode. It's just spinning its wheels, in my experience (I have a harness for testing models with 25 real bugs/features/etc from my real projects that I measure outcomes against) DS4 flash does _worse_ with max effort. It will literally have the right approach and reason itself away from it.

You can't possibly think that it's going to get cheaper and cheaper to pay for tokens though. Right? Have you seen what's happening with Codex/Claude subscriptions? Deepseek raising API prices.. We've been getting subsidized tokens for some time now and as the hardware costs skyrocket these labs/people with inference compute are going to continue to clamp down.

As a counter to that - I've tried various flavors/quants/full weights and Qwen 3.8 27B has been entirely useless at anything non-trivial. Sure - it can do some boilerplate work (though, even armed with a well written spec and working within a very well known framework it went off the rails and did things in a way that were... um... questionable at best) but I don't see it as anything more than a personal assistant style model. Zero chance I'd "work" with it, I spent days trying to get it to do something for me that was usable that I didn't have to have reviewed and refined by a frontier level model or myself. Couldn't do it. The idea that qwen 3.8 27b is _anywhere near_ Opus 4.8 is laughable. Pure benchmaxxing.

DS4 Flash 0731, on the other hand, wildly opposite experience. Would recommend.

GLM 5.2 - even quanted down to a hybrid 4/3 bit setup is amazing for everything but the hardest/most complex stuff in the same projects/realm.

> JpegXL will be the best of all worlds choice in a few years when software support is good.

Where is the software support lacking other than the browsers at this point?

Whether or not they can be trusted isn't all that relevant when it's still something along the lines of "Insert $1 get $25 in return" even if it's their own rates you're using to measure the value. I'm at ~2.2b Fable 5 tokens in the last 7 days (I ingest/index every session) and napkin math says that's ~$2,700 in usage. I have two max accounts, so $400 a month, divide by 4 to get $100 for this same 7 day period across those two accounts (neither are maxed out for the week, so this isn't even full utilization). I put $100 into the machine and got back $2,700 in fable bucks. Deepinfra would have to have quite the discounted rate to beat that.

> Why not just use that other vendor's model for everything?

Because it's not an either or thing. Neither is sufficient. I'd argue that, expenses aside, you should have every model you have access to cross reviewing the work of the others.

Outside of super trivial things that I should have just done myself, I have a cross-model review of _everything_ these days. The tokens are too cheap not to.

> and often described as a match with Opus in overall quality

It's not. Idk about who has more T's but, unfortunately, DS4 pro is not a match to Opus, at least not Opus 4.8.

July 27th

> So your argument is now RNG plays a bigger role so eventually they'll score a win by luck?

Uh, no. His argument is that the ~10% or so superiority you're fighting so hard to claim is no longer enough to have dominate sway over the overall outcome. Other factors, not random.