Re:Better and Faster Large Language Models via Multi-Token P...
by faabian on Wednesday May 01, 2024 @06:24AM
(#40221470)
Vectors can do what one-hot vectors cannot do -- no one said inputs need to be rows from an token_id -> vector embeddings map. Basically, we are doing this already by moving from one-hot vectors to n-tuples of one-hot vectors, increasing the effective vocabulary size from V to V^n.
[ Parent
| on HN ]
There are no comments on this comment yet.