News for hackers. Stuff that matters.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by jstummbillig on Monday December 01, 2025 @02:31PM (#46111955)
How could we judge if anyone is "winning" on cost-effectiveness, when we don't know what everyones profits/losses are?
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by ericskiff on Monday December 01, 2025 @02:55PM (#46112288)
I believe this was a statement on cost per token to us as consumers of the service
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by moffkalast on Tuesday December 02, 2025 @04:32AM (#46119367)
Training cost-effectiveness doesn't matter for open models since someone else ate the cost. In this case, Chinese taxpayers.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by KvanteKat on Tuesday December 02, 2025 @06:05AM (#46120008)
Deepseek is a private corporation funded by a hedge fund (High-Flyer). I doubt much public money was spent by the Chinese state on this. Like with LLMs in the US, the people paying for it so far are mainly investors who are betting on a return in the long to medium term.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by boringg on Tuesday December 02, 2025 @12:51PM (#46124098)
Do you actually believe what you just wrote or are you trolling? One version at least has a foot planted in reality. The other one well...
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by rowanG077 on Monday December 01, 2025 @03:54PM (#46113064)
Well consumers care about the cost to them, and those we know. And deepseek is destroying everything in that department.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by eru on Monday December 01, 2025 @10:57PM (#46117367)
Yes. Though we don't know for sure whether that's because they actually have lower costs, or whether it's just the Chinese taxpayer being forced to serve us a treat.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by chronogram on Monday December 01, 2025 @11:42PM (#46117624)
Third party providers are still cheap though. The closed models are the ones where you can't see the real cost to running them.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by eru on Tuesday December 02, 2025 @01:37AM (#46118299)
Oh, I was mostly talking about the Chinese taxpayer footing the training bill.

You are right that we can directly observe the cost of inference for open models.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by rescbr on Tuesday December 02, 2025 @08:44AM (#46121115)
Not sure the Chinese taxpayer is footing the bill though - of course, it might not be net zero, there might be secondary effects, etc.

A few days ago I read an article saying the Chinese utilities have a pricing structure that favors high-tech industries (say, an AI data center), making the difference by charging more the energy-intensive but less sophisticated industries (an aluminium smelter, for example).

Admittedly, there are some advantages when you do central and long-term economic planning.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by tedivm on Monday December 01, 2025 @05:29PM (#46114227)
If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware.

On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing where you are able to, which will save you money on both methods) and you have a basic idea of how much it would cost per token.

Then you compare that to the cost of something like GPT5, which is a bit simpler because the cost per (million) token is something you can grab off of a website.

You'd be surprised how much money running something like DeepSeek (or if you prefer a more established company, Qwen3) will save you over the cloud systems.

That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by Muromec on Monday December 01, 2025 @06:37PM (#46115136)
>That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on.

What's cheap nowdays? I'm out of the loop. Does anything ever run on integrated AMD that is Ryzen AI that comes in framework motherboards? Is under 1k americans cheap?

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by GTP on Tuesday December 02, 2025 @03:37AM (#46119004)
Not really in the loop either, but when Deepseek R1 was released, I sumbled upon this YouTube channel [1] that made local AI PC builds in the 1000-2000$ range. But he doesn't always use GPUs, maybe the cheaper builds were CPU plus a lot of RAM, I don't remember.

[1] https://youtube.com/@digitalspaceport?si=NrZL7MNu80vvAshx

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by baq on Tuesday December 02, 2025 @07:31AM (#46120566)
'lots of RAM' got expensive lately -_-
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by District5524 on Tuesday December 02, 2025 @08:01AM (#46120775)
Digital Spaceport is a really good channel, I second that - the author is not sparing any detail. The cheaper options always use CPU only, or sharding between different cheap GPUs (without SLI/switching) - which is not good for all use cases (he also highlights this). But some his prices are one-off bargains for used stuff. And RAM prices doubled this year, so you won't buy 2x256 GB DDR4 for $336, no matter what: https://digitalspaceport.com/500-deepseek-r1-671b-local-ai-s...
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by qeternity on Monday December 01, 2025 @05:58PM (#46114639)
> DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on.

Uh, Deepseek will not (unless you are referring to one of their older R1 finetuned variants). But any flagship Deepseek model will require 16x A100/H100+ with NVL in FP8.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by AlexCoventry on Monday December 01, 2025 @09:27PM (#46116668)
Mixture-of-Expert models benefit from economies of scale, because they can process queries in parallel, and expect different queries to hit different experts at a given layer. This leads to higher utilization of GPU resources. So unless your application is already getting a lot of use, you're probably under-utilizing your hardware.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by chazeon on Monday December 01, 2025 @10:12PM (#46117011)
Well the seemingly cheap comes with significantly degraded performance, particular for agentic use. Have you tried replacing Claude Code with some locally deployed model, say, on 4090 or 5090? I have. It is not usable.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by estsauver on Monday December 01, 2025 @10:55PM (#46117353)
Well, those are also extremely limited vram areas that wouldn't be able to run anything in the ~70b parameter space. (Can you run 30b even?)

Things get a lot more easier at lower quantisation, higher parameter space, and there's a lot of people's whose jobs for AI are "Extract sentiment from text" or "bin into one of these 5 categories" where that's probably fine.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by nylonstrung on Tuesday December 02, 2025 @01:19AM (#46118192)
Deepseek and Kimi both have great agentic performance

When used with crush/opencode they are close to Claude performance.

Nothing that runs on a 4090 would compete but Deepseek on openrouter is still 25x cheaper than claude

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by Aeolun on Tuesday December 02, 2025 @06:44AM (#46120294)
> Deepseek on openrouter is still 25x cheaper than claude

Is it? Or only when you don’t factor in Claude cached context? I’ve consistently found it pointless to use open models because the price of the good ones is so close to cached context on Claude that I don’t need them.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by joefourier on Tuesday December 02, 2025 @07:06AM (#46120442)
Deepseek via their API also has cached context, although the tokens/s was much lower than Claude when I tried it. But for background agents the price difference makes it absolutely worth it.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by ewoodrich on Tuesday December 02, 2025 @11:43AM (#46123135)
Yes, if you try using Kilo Code/Cline via Openrouter the cost will be much cheaper using Deepseek/Kimi vs Claude Sonnet 4.5.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by elif on Tuesday December 02, 2025 @09:34AM (#46121578)
Strictly speaking, you have not deployed any model on a 5090 because a 5090 card has never been produced.

And without specifying your quantization level it's hard to know what you mean by "not usable"

Anyway if you really wanted to try cheap distilled/quantized models locally you would be using used v100 Teslas and not 4 year old single chip gaming GPUs.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by __alexs on Tuesday December 02, 2025 @12:16PM (#46123621)
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by matthewmacleod on Tuesday December 02, 2025 @10:42AM (#46122338)
You can just buy a 5090 now for $3k. Have you confused it with something else?
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by JosephjackJR on Tuesday December 02, 2025 @01:09PM (#46124316)
they took the already ridiculous v3.1 terminus model, added this new deepseek sparse attention thing, and suddenly it’s doing 128k context at basically half the inference cost of the old version with no measurable drop in reasoning or multilingual quality. like, imo gold medal level math and code, 100+ languages, all while sipping tokens at 14 cents per million input. that’s stupid cheap. the rl recipe they used this time also seems way more stable. no more endless repetition loops or random language switching you sometimes got with the earlier open models. it just works. what really got me is how fast the community moved. vllm support landed the same day, huggingface space was up in hours, and people are already fine-tuning it for agent stuff and long document reasoning. i’ve been playing with it locally and the speed jump on long prompts is night and day. feels like the gap to the closed frontier models just shrank again. anyone else tried it yet?
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by miki123211 on Tuesday December 02, 2025 @12:17AM (#46117843)
> with your own hardware

Or with somebody else's.

If you don't have strict data residency requirements, and if you aren't doing this at an extremely large scale, doing it on somebody else's hardware makes much more economic sense.

If you use MoE models (al modern >70B models are MoE), GPU utilization increases with batch size. If you don't have enough requests to keep GPUs properly fed 24/7, those GPUs will end up underutilized.

Sometimes underutilization is okay, if your system needs to be airgapped for example, but that's not an economics discussion any more.

Unlike e.g. video streaming workloads, LLMs can be hosted on the other side of the world from where the user is, and the difference is barely going to be noticeable. This means you can keep GPUs fed by bringing in workloads from other timezones when your cluster would otherwise be idle. Unless you're a large, worldwide organization, that is difficult to do if you're using your own hardware.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by embedding-shape on Tuesday December 02, 2025 @12:44PM (#46124005)
> If you use MoE models (al modern >70B models are MoE), GPU utilization increases with batch size

Isn't that true for any LLM, MoE or not? In fact, doesn't that apply to most concepts within ML, as long as it's possible to do batching at all, you can scale it up and utilize more of the GPU, until you saturate some part of the process.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by kmacdough on Tuesday December 02, 2025 @12:51PM (#46124100)
Furthermore, paid models are heavily subsidized by bullish investors playing for monopoly. So that tips the scales further towards Deepseek.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by deaux on Monday December 01, 2025 @08:05PM (#46115974)
We can judge on inference cost because we do know what those are for open-weights models as there are a dozen independent providers that host these models and price them according to respective inference cost.

We can't judge on training cost, that's true.

Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by stingraycharles on Monday December 01, 2025 @08:17PM (#46116078)
You can use tokens/sec on something like AWS Bedrock (which hosts both open and closed models) as a proxy for “costs per token” for the closed providers.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by mzl on Tuesday December 02, 2025 @01:55AM (#46118395)
Apart from measuring prices from venture-backed providers which might or might not correlate with cost-effectiveness, I think the measures of intelligence per watt and intelligence per joule from https://arxiv.org/abs/2511.07885 is very interesting.
Re:DeepSeek-v3.2: Pushing the Frontier of Open Large Languag... by badmonster on Tuesday December 02, 2025 @04:08AM (#46119192)
Good point. Could usage patterns + inference costs give us proxy metrics? What would be a fair baseline?