News for hackers. Stuff that matters.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by patrakov on Tuesday January 09, 2024 @02:26AM (#38923382)
Just as another data point, a CPU-only setup with Q5_K_M would give you roughly 4 tokens per second on a Ryzen laptop (Dell Inspiron 7415 upgraded to 64 GB of RAM).

There are no comments on this comment yet.