News for hackers. Stuff that matters.
Re:Dalai: Automatically install, run, and play with LLaMA on... by Taek on Monday March 13, 2023 @09:27AM (#35135428)
For completeness, there's also another paper that demonstrated you get more power/accuracy per-bit at 4 bits than at any other level of precision (including 2 bits and 3 bits)
Re:Dalai: Automatically install, run, and play with LLaMA on... by MacsHeadroom on Tuesday March 14, 2023 @12:38AM (#35147506)
That's the paper I referenced. But newer research is already challenging it.

'Int-4 llama is not enough [0] - Int-3 and beyond' suggests 3-bit is best for models larger than ~10B parameters when combining binning and GPTQ.

[0] https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...