News for hackers. Stuff that matters.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by cs702 on Thursday February 29, 2024 @09:19AM (#39549574)
Thank you. Others on this thread have addressed the citation-trail issues you raise. I just want to tell you how helpful I find your comment about why ternary weights ought to work at all without degrading performance:

> My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.

Your guess sounds and feels right to me, even if currently there's no way to express it formally, with the rigor it deserves.

Thank you again for your comment!

There are no comments on this comment yet.