Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by Invictus0 on Tuesday May 28, 2024 @01:21PM (#40503074)

I'm not saying this to be rude, but I think you have a deep misunderstanding of how AI training works. You cannot just skip the matrix multiplications necessary to train the model, or get current hardware to do it faster.

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by xdavidliu on Tuesday May 28, 2024 @03:41PM (#40504748)

was the first sentence really necessary? The second sentence seems fine by itself.

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by nickpsecurity on Tuesday May 28, 2024 @07:16PM (#40506600)

There's work on replacing multiplication. Here's four examples:

https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_...

https://arxiv.org/abs/2012.03458

https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/E...

https://arxiv.org/pdf/2106.10860

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by benterix on Wednesday May 29, 2024 @06:17AM (#40510347)

No offence taken! As far as my (shallow!) understanding goes, the main challenge is the need for many GPUs with huge amounts of memory, and it still takes ages to train the model. So regarding the use of consumer GPUs, some work has been done already, and I've seen some setups where people combine of these and are successful. As for the the other aspects, maybe at some point we distill what is really needed to a smaller but excellent dataset that would give similar results in the final models.

[ Parent | on HN ]