News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by jerpint on Wednesday May 01, 2024 @06:08AM (#40221368)
My understanding is that tokenization is part of the bottleneck. When you break up a sentence to tokens, each token gets a vector representation. The dictionary of all tokens would be infinite if it was at the sentence level
Re:Better and Faster Large Language Models via Multi-Token P... by faabian on Wednesday May 01, 2024 @06:24AM (#40221470)
Vectors can do what one-hot vectors cannot do -- no one said inputs need to be rows from an token_id -> vector embeddings map. Basically, we are doing this already by moving from one-hot vectors to n-tuples of one-hot vectors, increasing the effective vocabulary size from V to V^n.