News for hackers. Stuff that matters.
Re:Show HN: I made a GPU VRAM calculator for transformer-bas... by chrsig on Tuesday December 26, 2023 @10:10PM (#38778679)
How does one rawdog a 16-bit model?
Re:Show HN: I made a GPU VRAM calculator for transformer-bas... by kkzz99 on Wednesday December 27, 2023 @06:36AM (#38781036)
Usually, for efficiency, you use quantized models. Quantized models reduce the number of bits available for each parameter, saving space and reduce RAM usage.