News for hackers. Stuff that matters.
Re:Teaching GPT-3 to reverse words by axiom92 on Monday May 16, 2022 @04:53PM (#31403014)
50k is not the number of unique words that GPT-3 supports, and perhaps you're referring to the BPE tokens. The input to GPT-3 is not tokenized by splitting on spaces, and is based on byte-pair encoding tokens. You can play with it here: https://beta.openai.com/tokenizer.

A rare word like blithe is tokenized into two BPE tokens: bl and ithe, whereas common words like the get their own token.

There are no comments on this comment yet.