News for hackers. Stuff that matters.
Re:Mistral NeMo by hislaziness on Thursday July 18, 2024 @12:03PM (#40997003)
isn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?
Re:Mistral NeMo by ancientworldnow on Thursday July 18, 2024 @12:06PM (#40997036)
This was trained to be run at FP8 with no quality loss.
Re:Mistral NeMo by hislaziness on Thursday July 18, 2024 @08:04PM (#41000896)
The model description on huggingface says - Model size - 12.2B params, Tensor type - BF16. Is the Tensor type different from the training param size?
Re:Mistral NeMo by fzzzy on Thursday July 18, 2024 @01:36PM (#40997949)
it's very common to run local models in 8 bit int.
Re:Mistral NeMo by qwertox on Thursday July 18, 2024 @02:50PM (#40998593)
Yes, but it's not common for the original model to be 8 bit int. The community can downgrade any model to 8 bit int, but it's always linked to quality loss.