Re:Mixtral 8x7B: A sparse Mixture of Experts language model
by eyegor on Tuesday January 09, 2024 @01:33AM
(#38923092)
So you don't see significantly worse performance on 3bit quantized models compared to 4? Every 7/13b model I tried quantized gave much worse responses at 3 bit and below, whereas the differences from 4 bit to 6 or even 8 bit is more subtle.
[ Parent
| on HN ]
Re:Mixtral 8x7B: A sparse Mixture of Experts language model
by coder543 on Tuesday January 09, 2024 @02:04AM
(#38923270)
Mixtral is a larger model, so maybe that makes it more tolerant of that level of quantization? I’ve been impressed with 3-bit Mixtral, but I haven’t done a ton of side by sides against 4-bit because I haven’t felt the need.