News for hackers. Stuff that matters.

A comment on Nvidia Unveils Updated GH200 Grace Hopper with HBM3e Memory, Shipping in Q2'2024. Parent comment

Re:Nvidia Unveils Updated GH200 Grace Hopper with HBM3e Memo... by refulgentis on Monday October 02, 2023 @12:14PM (#37740162)
It's not really clear what you mean, maybe there's gaps in my ML knowledge, but generally the answer depends on "what throughput do you need for your use case?"

Generally I agree that if you only need to deliver mail once daily on a mile long route, a car that takes 24 hours to travel a mile is fine.

[ Parent | on HN ]
Re:Nvidia Unveils Updated GH200 Grace Hopper with HBM3e Memo... by upbeat_general on Monday October 02, 2023 @06:53PM (#37745553)
Lots of ML applications don't generally care about latency, and throughput only matters per dollar (as you can just scale horizontally).

To my knowledge, most inferencing (at least for simpler models) happens on cheaper, slower GPUs that have better throughput/dollar.

[ Parent | on HN ]