News for hackers. Stuff that matters.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by paul_mk1 on Wednesday February 28, 2024 @05:48PM (#39544500)
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there).

Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work).

Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance this will work for LLMs too.

https://arxiv.org/abs/1606.01981

This was a passion project of mine in my last few months at IBM research :).

I am convinced there is a deep connection to understanding why backprop is unreasonably effective, and the result that you can train low precision DNNs; for those note familiar, the technique is to compute the loss wrt to the low precision parameters (eg project to ternary) but apply the gradient to high precision copy of parameters (known as the straight through estimator). This is a biased estimator and there is no theoretical underpinning for why this should work, but in practice it works well.

My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.

Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by WhitneyLand on Wednesday February 28, 2024 @07:56PM (#39545417)
That’s really interesting to see the breadcrumb trail goes back that far.

So what are the most important insights in this paper compared to what was previously done?

I assume there’s more context to the story and it’s not just that no one thought to apply the concepts to LLM’s until now?

Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by paul_mk1 on Wednesday February 28, 2024 @11:40PM (#39546444)
I don't think there is anything conceptually new in this work, other than it is applied to LLMs.

But in fairness, getting these techniques to work at scale is no small feat. In my experience quantization aware training at these low bit depths was always finicky and required a very careful hand. I'd be interested to know if it has become easier to do, now that there are so many more parameters in LLMs.

In any case full kudos to the authors and I'm glad to see people continuing this work.

Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by nxobject on Wednesday February 28, 2024 @07:21PM (#39545209)
As aside, I'm curious: what was it like to work at IBM research, especially as a legacy industrial research org?
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by eru on Wednesday February 28, 2024 @08:41PM (#39545698)
You can probably apply the same techniques 'Deep neural networks are robust to weight binarization and other non-linear distortions' used to get to 0.68 bits / weight to get your ternary weights below one bit; so you can claim they are still one-bit networks.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by antimatter15 on Wednesday February 28, 2024 @10:10PM (#39546049)
They cite straight through estimators in the previous work with many of the same authors on (actual binary) BitNet
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by fabmilo on Wednesday February 28, 2024 @10:13PM (#39546069)
They train using Straight Through Estimator but is cited in the previous BitNet paper. What happen to the TrueNorth Chip? I think investing in specialized hardware for AI is a good bet.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by paul_mk1 on Wednesday February 28, 2024 @11:31PM (#39546409)
Nice to know there is a trail to relevant citations. I missed the BitNet paper and need to catch up.

Btw TrueNorth project evolved into "NorthPole" chip by the same group, and was recently in the press. From afar NorthPole looks like an interesting design point and leverages on-chip memory (SRAM)--so it's targeting speed and efficiency at the expense of memory density (so perhaps like Groq in some respects). Tbh I haven't followed the field closely after leaving the group.

Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by mjcohen on Wednesday February 28, 2024 @11:51PM (#39546491)
IIRC, Hamming's book "Digital Filters" (1989) has a section on FFTs with only the sign of the coefficient being used. It performed surprisingly well.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by thomasahle on Thursday February 29, 2024 @01:22AM (#39546845)
You mean Fast Hadamard Transform?
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by mitthrowaway2 on Thursday February 29, 2024 @02:36AM (#39547168)
What is the sign of a complex number? Do you mean the phase?
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by nine_k on Thursday February 29, 2024 @09:45AM (#39549955)
AFAICT, both the real and imaginary components are from (-1, 0, +1) only. No single sign, but only 8 directions and the center.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by cs702 on Thursday February 29, 2024 @09:19AM (#39549574)
Thank you. Others on this thread have addressed the citation-trail issues you raise. I just want to tell you how helpful I find your comment about why ternary weights ought to work at all without degrading performance:

> My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.

Your guess sounds and feels right to me, even if currently there's no way to express it formally, with the rigor it deserves.

Thank you again for your comment!

Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by WiSaGaN on Thursday February 29, 2024 @10:03AM (#39550227)
Could the reason that 3 states in this case be more efficient than 2 states be that 3 is closer to 2.718... (Euler's number) than 2 is?
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by altruios on Thursday February 29, 2024 @10:30AM (#39550658)
Why not have some layers/nodes/systems be 2 states and have others be 3... couldn't you get arbitrarily close to Euler's number that way?