News for hackers. Stuff that matters.
Re:Deploying DeepSeek on 96 H100 GPUs by 34679 on Friday August 29, 2025 @10:44AM (#45064819)
"By deploying this implementation locally, it translates to a cost of $0.20/1M output tokens"

Is that just the cost of electricity, or does it include the cost of the GPUs spread out over their predicted lifetime?

Re:Deploying DeepSeek on 96 H100 GPUs by zipy124 on Friday August 29, 2025 @12:17PM (#45066023)
This is all costs included. Thats 22k tokens per second per node, so per 8 h100's. With 12 nodes they get 264k tokens per second, or 950 million an hour. This get's you to roughly $0.2021 per million at $2 an hour for an h100, which is what they go for on services such as runpod.io . (cheaper if not paying spot-price + volume discounts).
Re:Deploying DeepSeek on 96 H100 GPUs by dragonslayer56 on Friday August 29, 2025 @10:54AM (#45064954)
” Our implementation, shown in the figure above, runs on 12 nodes in the Atlas Cloud, each equipped with 8 H100 GPUs.”

Maybe the cost of renting?

Re:Deploying DeepSeek on 96 H100 GPUs by 34679 on Friday August 29, 2025 @11:09AM (#45065147)
I'm confused because I wouldn't consider a cloud implementation to be local.
Re:Deploying DeepSeek on 96 H100 GPUs by randomjoe2 on Friday August 29, 2025 @11:39AM (#45065518)
Local doesn't refer to "on metal" anymore to many people
Re:Deploying DeepSeek on 96 H100 GPUs by mwcz on Friday August 29, 2025 @11:51AM (#45065663)
"On metal" is muddied too. I've heard people refer to web apps running in an OCI container as being "bare metal" deployment, as opposed to AWS or whatever hosting platform.

That's silly, but the idea that "local" is not the opposite of remote is even sillier.

Re:Deploying DeepSeek on 96 H100 GPUs by dtech on Friday August 29, 2025 @12:06PM (#45065883)
If you do bare metal as not being under a VM it fits. OCI on linux is cgroup so that counts as not a VM I'd say. Or at least it's a layer closer to the metal than a typical VM running OCI images.

I a Java app running on Linux bare metal?

Re:Deploying DeepSeek on 96 H100 GPUs by ffsm8 on Friday August 29, 2025 @11:56AM (#45065742)
You can run an OCI container on bare metal though. It doesn't stop being run on bare metal just because you're running in kernel namespaces, aka docker container

Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers

Now wherever that's applied to your conversation... I've no clue, too little context ( 。 ŏ ﹏ ŏ )

Re:Deploying DeepSeek on 96 H100 GPUs by okasaki on Friday August 29, 2025 @12:20PM (#45066076)
In my opinion, if you're running k8s on bare metal, that's "k8s on bare metal" but still "<your app> on kubernetes", not "<your app> on bare metal".
Re:Deploying DeepSeek on 96 H100 GPUs by ffsm8 on Friday August 29, 2025 @01:44PM (#45067194)
Sorry, but then your opinion is just plain wrong

Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware.

As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines

Re:Deploying DeepSeek on 96 H100 GPUs by mystifyingpoi on Friday August 29, 2025 @02:37PM (#45067809)
I think both of you are correct.

Of course, a process running inside Kubernetes Pod, on a baremetal node will show up in `top` if I run it on the node directly. In such terms, it is running directly on hardware.

But when I deploy this Pod, I'm not interacting with the OS in any way. I'm interacting with Kubernetes apiserver, telling it what to run, not really caring about the operating system underneath. In such terms, the application is running "in k8s".

Re:Deploying DeepSeek on 96 H100 GPUs by bee_rider on Friday August 29, 2025 @01:50PM (#45067264)
Bare metal as in, no operating system? Does Linux really get in the way of these LLM inference engines?
Re:Deploying DeepSeek on 96 H100 GPUs by ffsm8 on Friday August 29, 2025 @01:56PM (#45067336)
No, as I said in my previous comment: bare metal as in not a virtual machine

https://en.m.wikipedia.org/wiki/Bare-metal_server

Re:Deploying DeepSeek on 96 H100 GPUs by pessimizer on Friday August 29, 2025 @04:20PM (#45068930)
Note that this is a term whose meaning has been expanded to refer to non-VPS servers very recently. Bare-metal has traditionally meant "without an operating system." It did not mean "a server that is an actual server," because that was the default.

It also does not always "clearly" have this new meaning. Somebody who is used to running programs directly (with no intermediate OS) on hardware might not understand what you're saying, or might ask you to clarify, and you probably shouldn't feel put upon by a totally understandable misinterpretation.

edit: Especially when you keep repeating "directly on hardware" when you mean "not on a VM." VMs also run on hardware. You're saying that you're only running on one OS instead an OS in your OS.

Re:Deploying DeepSeek on 96 H100 GPUs by mwcz on Sunday August 31, 2025 @10:20AM (#45083341)
This discussion made me realize that I have a head canon definition of "bare metal" that applies more to the programming environment than the deployment environment. It would exclude any runtime translation to the native instruction set, such as a VM, bytecode VM, language interpreter, etc. Basically identical in meaning to "static compilation", so I'll update my brain to the conventional meaning.
Re:Deploying DeepSeek on 96 H100 GPUs by monsieurbanana on Friday August 29, 2025 @11:50AM (#45065653)
I missed that train
Re:Deploying DeepSeek on 96 H100 GPUs by vFunct on Friday August 29, 2025 @12:03PM (#45065843)
My basement server really confused by all this...
Re:Deploying DeepSeek on 96 H100 GPUs by bee_rider on Friday August 29, 2025 @01:45PM (#45067202)
Local doesn’t need to be “on metal,” but I’m still confused as to what they are saying. Are they running some local cloud system?
Re:Deploying DeepSeek on 96 H100 GPUs by DSingularity on Friday August 29, 2025 @11:42AM (#45065549)
I guess local for him is independent/private.
Re:Deploying DeepSeek on 96 H100 GPUs by ollybee on Friday August 29, 2025 @11:38AM (#45065503)
H100's can be $2 and hour, so $192 an hour for the full cluster. They report 22k tokens per second, so ~ 80 million an hour, thats $16 an hour at $0.2 per million. Maybe a bit more for input tokens, but it seems a long way off.
Re:Deploying DeepSeek on 96 H100 GPUs by zipy124 on Friday August 29, 2025 @12:15PM (#45066003)
I think you mis-read. Thats 22k tokens per second per node, so per 8 h100's. With 12 nodes they get 264k tokens per second, or 950 million an hour. This get's you to roughly $0.2021 per million at $2 an hour.
Re:Deploying DeepSeek on 96 H100 GPUs by adam_arthur on Friday August 29, 2025 @11:39PM (#45071720)
I'm curious as well.

Depreciation and GPU failure rate over time must be considered, which I don't see mentioned in the article.