News for hackers. Stuff that matters.
Re:1-Bit LLMs Could Solve AI's Energy Demands by kir-gadjello on Friday May 31, 2024 @01:59AM (#40531940)
While llama3-8b might be slightly more brittle under quantization, llama3-70b really surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime. It requires one of the most advanced quantization methods (IQ2_XS specifically) to work, but the reward is a SoTA LLM that fits on one 4090 GPU with 8K context (KV-cache uncompressed btw) and allows for advanced usecases such as powering the agent engine I'm working on: https://github.com/kir-gadjello/picoagent-rnd

For me it completely replaced strong models such as Mixtral-8x7B and DeepSeek-Coder-Instruct-33B.

1. https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_...

Re:1-Bit LLMs Could Solve AI's Energy Demands by d13 on Friday May 31, 2024 @02:57AM (#40532196)
How does it compare against unequalised Llama 3 8B at 16fp? I’ve been using that locally and it’s almost replaced GPT4 for me. Runs in about 14GB of VRAM.
Re:1-Bit LLMs Could Solve AI's Energy Demands by iwontberude on Friday May 31, 2024 @10:31AM (#40535697)
llama3 is nowhere near gpt4, though it is cool
Re:1-Bit LLMs Could Solve AI's Energy Demands by LordDragonfang on Friday May 31, 2024 @05:09PM (#40540278)
What is your use case where you find it comparable to gpt4?
Re:1-Bit LLMs Could Solve AI's Energy Demands by stavros on Friday May 31, 2024 @06:52PM (#40541162)
For creative tasks, for example, Llama 3 is much better. GPT-4 is very sterile, Llama is much more whimsical, and has a lot more character.
Re:1-Bit LLMs Could Solve AI's Energy Demands by CaptainOfCoit on Friday May 31, 2024 @07:55PM (#40541491)
> For creative tasks

What, specifically, are you asking of these LLMs? "creative tasks" can be anything from programming to cooking recipes, so a tiny bit more specificality would be appreciated :)

Re:1-Bit LLMs Could Solve AI's Energy Demands by stavros on Friday May 31, 2024 @08:35PM (#40541747)
Sorry, I meant making up stuff, e.g. creative writing.
Re:1-Bit LLMs Could Solve AI's Energy Demands by CaptainOfCoit on Saturday June 01, 2024 @08:02AM (#40545013)
So like making presentations? Composing poems? Writing novellas/novels? Technical articles?
Re:1-Bit LLMs Could Solve AI's Energy Demands by LordDragonfang on Sunday June 02, 2024 @04:19AM (#40552417)
I've used pretty much every major LLM out there for a specific type of creative writing, and none of them are as good at it as GPT4 with the exception of maybe Claude (Opus is actually probably even better regarding the sterility). Llama 3, even 70b, is definitely not better by any measure of actual quality - it's more random, at best.
Re:1-Bit LLMs Could Solve AI's Energy Demands by renewiltord on Friday May 31, 2024 @11:31AM (#40536453)
Holy wow. Thank you for this. Very cool. I’ve been using 8b for things it might be worth using 70b for.
Re:1-Bit LLMs Could Solve AI's Energy Demands by endofreach on Friday May 31, 2024 @12:56PM (#40537486)
> surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime

I am too dumb for all of this ML stuff. Can you explain what exactly that means & why it's surprising?

Re:1-Bit LLMs Could Solve AI's Energy Demands by m1el on Friday May 31, 2024 @01:35PM (#40538014)
Artificial neural networks work the following way: you have a bunch of “neurons” which have inputs and an output. Neuron’s inputs have weights associated with them, the larger the weight, the more influence the input has on the neuron. These weights need to be represented in our computers somehow, usually people use IEEE754 floating point numbers. But these numbers take a lot of space (32 or 16 bits). So one approach people have invented is to use more compact representation of these weights (10, 8, down to 2 bits). This process is called quantisation. Having a smaller representation makes running the model faster because models are currently limited by memory bandwidth (how long it takes to read weights from memory), going from 32 bits to 2 bits potentially leads to 16x speed up. The surprising part is that the models still produce decent results, even when a lot of information from the weights was “thrown away”.
Re:1-Bit LLMs Could Solve AI's Energy Demands by endofreach on Saturday June 01, 2024 @01:35PM (#40547616)
Oh, nice. Thank you for this explanation. Now i think i get quantisation. Very well explained for someone like me. Thank you a lot!