News for hackers. Stuff that matters.
Re:Teaching GPT-3 to reverse words by rprenger on Monday May 16, 2022 @05:10PM (#31403175)
I don't think a larger vocab would help. All the individual letters are in the ~50k token vocab already, but the word "alphabet" will still not get tokenized to [a, l, p, h, a, b, e, t]. Using a larger vocab like PaLM's 256k vocab would have the same issue.

There are no comments on this comment yet.