News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by mike_hearn on Wednesday May 01, 2024 @07:59AM (#40222046)
Is it possible that anti-scaling effect occurs because you are removing some middle layers to free up space for the extra output heads? I only scanned the paper quickly but what happens if you treat the technique as strictly additive and don't keep parameter sizes fixed?

There are no comments on this comment yet.