Re:Mistral NeMo
by hislaziness on Thursday July 18, 2024 @12:03PM
(#40997003)
isn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?
[ Parent
| on HN ]
Re:Mistral NeMo
by ancientworldnow on Thursday July 18, 2024 @12:06PM
(#40997036)
This was trained to be run at FP8 with no quality loss.
Re:Mistral NeMo
by hislaziness on Thursday July 18, 2024 @08:04PM
(#41000896)
The model description on huggingface says - Model size - 12.2B params, Tensor type - BF16. Is the Tensor type different from the training param size?
it's very common to run local models in 8 bit int.