H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark

Re:H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark by usaar333 on Tuesday June 27, 2023 @09:08PM (#36500986)

Good call on power draw, but it seems dwarfed by capital cost. Assuming 3 year life (this tech gets outdated..), 300W constant consumption costs under $1k at $0.3/kwh.

You need to be an order of magnitude higher in power usage before it really starts mattering. e.g. a Tesla3 consumes around 15 KW (20x the H100) while driving.

That said it looks like flops/watt dropped by only 3.4x, which is also sub Moore's Law (3 years to halve power consumption)

[ Parent | on HN ]

Re:H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark by TOMDM on Tuesday June 27, 2023 @09:40PM (#36501227)

I don't think flops/$ is enough to really capture the difference here.

You couldn't replicate the scale of compute this allows no matter how many V100's you had.

A huge amount of cost here is embodied in networking and memory.

If one were to design a chip that cared only for flop/$ without caring for all of the interconnect and memory, then the 4090 is a much fairer comparison, and even then that card isn't designed for a flop/$ optimisation.

[ Parent | on HN ]