News for hackers. Stuff that matters.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by fgfm on Wednesday February 28, 2024 @01:12PM (#39541673)
It's funny how discoveries in NLP & computer vision complement each other. The replacement of multiplication by additions made me think about the AdderNet paper (https://arxiv.org/abs/1912.13200), which concluded as you had to suffer almost no performance drop.

Perhaps the accumulators in current hardware cannot leverage this to its full potential, but combined with such a strict quantization, this would open LLM to the wider ML community much earlier than expected (when consumer hardware allows you to train near SOTA LLMs from scratch on your machine).

There are no comments on this comment yet.