News for hackers. Stuff that matters.
Re:Apple introduces M4 chip by freeqaz on Tuesday May 07, 2024 @02:01PM (#40289265)
There are some crazy 1/1.5 bit quants now. If you're curious I'll try to dig up the papers I was reading.

1.5bit can be done to existing models. The 1 bit (and less than 1 bit iirc) requires training a model from scratch.

Still, the idea that we can have giant models running in tiny amounts of RAM is not completely far fetched at this point.

Re:Apple introduces M4 chip by gpm on Tuesday May 07, 2024 @02:10PM (#40289397)
Yeah, I'm broadly aware and have seen a few of the papers, though I definitely don't try and track the state of the art here closely.

My impression and experience trying low bit quants (which could easily be outdated by now) is that you are/were better off with a smaller model and a less aggressive quantization (provided you have access to said smaller model with otherwise equally good training). If that's changed I'd be interested to hear about it, but definitely don't want to make work for you digging up papers.