Re:Why is GPT-3 15.77x more expensive for certain languages?
by minimaxir on Monday April 10, 2023 @02:39PM
(#35515756)
10% fewer tokens on average.
The vocab size itself is doubled. (~50k for GPT-2/3, ~100k for ChatGPT)
Re:Why is GPT-3 15.77x more expensive for certain languages?
by sillysaurusx on Monday April 10, 2023 @02:56PM
(#35515960)
Wow. Does that help to double the vocab size?
It certainly makes training more expensive. One clever trick to get some memory savings is to freeze the vocab embedding layer when fine tuning. It makes a noticeable improvement, both in speed and in mem required.
Surprised they went the larger vocab route. LLaMA is only 30k. I wonder what the reason is...
Thanks!
Re:Why is GPT-3 15.77x more expensive for certain languages?
by minimaxir on Monday April 10, 2023 @03:17PM
(#35516206)
A larger vocab takes longer to train but has no (practical) impact at inference time as an Embeddings index is just a key-value store, which is very helpful as GPT starts hitting scaling laws.
The tokenization speedups in that repo are very impressive. It was the most annoying part about processing 190,000 books. I think it took a few days on a server with 96 cores.
Surprisingly hard to figure out the vocab size from that repo.