Are GPUs Worth It for ML?

Re:Are GPUs Worth It for ML? by rfrey on Monday August 29, 2022 @05:25PM (#32643770)

Not related to the article, but how would one begin to become smart on optimizing GPU workloads? I've been charged with deploying an application that is a mixture of heuristic search and inference, that has been exclusively single-user to this point.

I'm sure every little thing I've discovered (e.g. measuring cpu/gpu workloads, trying to multiplex access to the gpu, etc) was probably covered in somebody's grad school notes 12 years ago, but I haven't found a source of info on the topic.

[ Parent | on HN ]

Re:Are GPUs Worth It for ML? by pqn on Monday August 29, 2022 @05:36PM (#32643884)

Let's just take the topic of measuring GPU usage. This alone is quite tricky -- tools like nvidia-smi will show full GPU utilization even if not all SMs are running. And also the workload may change behavior over time, if for instance inputs to transformers got longer over time. And then it gets even more complicated to measure when considering optimizations like dynamic batching. I think if you peek into some ML Ops communities you can get a flavor of these nuances, but not sure if there are good exhaustive guides around right now.

[ Parent | on HN ]