News for hackers. Stuff that matters.

A comment on Nvidia Hopper GPU Architecture and H100 Accelerator. Parent comment

Re:Nvidia Hopper GPU Architecture and H100 Accelerator by algo_trader on Tuesday March 22, 2022 @03:07PM (#30770135)
Is it difficult/desirable to squeeze/compress an open-sourced 200B parameter model to fit into 40GB?

Are these techniques for specific architectures or can they be made generic ?

[ Parent | on HN ]
Re:Nvidia Hopper GPU Architecture and H100 Accelerator by ml_hardware on Tuesday March 22, 2022 @03:53PM (#30770658)
I think it depends what downstream task you're trying to do... DeepMind tried distilling big language models into smaller ones (think 7B -> 1B) but it didn't work too well... it definitely lost a lot of quality (for general language modeling) relative to the original model.

See the paper here, Figure A28: https://kstatic.googleusercontent.com/files/b068c6c0e64d6f93...

But if your downstream task is simple, like sequence classification, then it may be possible to compress the model without losing much quality.

[ Parent | on HN ]
Re:Nvidia Hopper GPU Architecture and H100 Accelerator by algo_trader on Tuesday March 22, 2022 @03:12PM (#30770189)
Ah, found some stuff already

https://www.tensorflow.org/model_optimization/guide/pruning

https://www.tensorflow.org/model_optimization/guide/pruning/...

[ Parent | on HN ]