2023 was the year that GPUs stood still

Re:2023 was the year that GPUs stood still by brucethemoose2 on Friday December 29, 2023 @10:48AM (#38806169)

The thing about ML is that memory helps everything.

Take almost any given model that fits in X memory pool for some task. Double that, and everything gets better. Training quality goes up, inference quality potentially goes up, you gain the capacity for more batching which dramatically increases performance. Maybe more caching, depending on the app. You can add more models to the pipeline without much fuss.

[ Parent | on HN ]

Re:2023 was the year that GPUs stood still by eachro on Friday December 29, 2023 @05:06PM (#38810819)

why is this? I would have thought that more memory would help up to some point (wherever the entire model can fit in RAM) and then no gains after that. I'd also have thought that things like accuracy wouldnt be affected.

[ Parent | on HN ]

Re:2023 was the year that GPUs stood still by brucethemoose2 on Saturday December 30, 2023 @12:53AM (#38813080)

Every model and ecosystem has different compromises, and obviously small models hit diminishing returns quick.

But the big models that are in fashion now... GPUs are basically always memory capacity limited to some extent.

[ Parent | on HN ]