Re:55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer wit...
by kuprel on Thursday February 22, 2024 @04:56PM
(#39473865)
This adds PyTorch/CUDA training support to Andrej Karpathy's minbpe. It takes 2min 28sec (148 seconds) on an RTX4090 to train the BasicTokenizer with a vocab_size of 512 on 307MB of Enron emails. The original code takes 2hrs 15min (8076 seconds) on an M2 Air with Python 3.11 to do this. That is a 55x speedup.
[ Parent
| on HN ]
Re:55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer wit...
by threesevenths on Friday February 23, 2024 @10:23AM
(#39481658)
Am I reading this right? A 55x improvement while also going from an M2 Air to an RTX 4090?
If so this doesn’t seem like a logical comparison and the 55x claim would likely not translate when using the same hardware.
Re:55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer wit...
by kuprel on Friday February 23, 2024 @12:06PM
(#39483206)
The M2 Air was actually much faster than whatever CPU was on the cloud RTX4090 machine I rented. I chose the stronger benchmark to compare to
Re:55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer wit...
by kuprel on Friday February 23, 2024 @03:16PM
(#39485562)
Using int16 and an H100 the speedup is actually 108x over the M2 air
Re:55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer wit...
by lostmsu on Saturday February 24, 2024 @01:22AM
(#39489569)
Why is it surprising? CPU-only M2 probably has under 1 teraops while RTX 4090 has 77. M2's GPU was not used, but even it only provides around 4 teraops, so would have been ~20x slower than 4090.