News for hackers. Stuff that matters.

A comment on GPUs for Deep Learning in 2023 – An In-depth Analysis. Parent comment

Re:GPUs for Deep Learning in 2023 – An In-depth Analysis by nl on Wednesday January 18, 2023 @06:03PM (#34434384)
The lack of ECC memory is almost certainly not a factor. If you can train at FP8 your model will recover from a single flipped bit somewhere.
[ Parent | on HN ]
Re:GPUs for Deep Learning in 2023 – An In-depth Analysis by Loranubi on Wednesday January 18, 2023 @10:50PM (#34436494)
I mean you could even view bit flips as a regularization technique like dropout...
[ Parent | on HN ]
Re:GPUs for Deep Learning in 2023 – An In-depth Analysis by dahart on Thursday January 19, 2023 @12:31AM (#34437040)
Yeah I hear it’s common practice now to avoid synchronizing GPU training kernels in order to speed things up, and it has positive regularization benefits and little downside.
[ Parent | on HN ]