News for hackers. Stuff that matters.
Re:Bfloat16 support coming to Apple's Metal and PyTorch [video] by vmirnv on Tuesday July 04, 2023 @05:53AM (#36584408)
Also I would highly recommend Q5_K_M for both 7B and 13B models.

It has the best balance between quality and weight of the model and almost indistinguishable from original f16: https://www.reddit.com/r/LocalLLaMA/comments/142q5k5/updated...

There are no comments on this comment yet.