News for hackers. Stuff that matters.

A comment on Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20. Parent comment

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by karpathy on Tuesday May 28, 2024 @04:31PM (#40505292)
That is 100% my intention and hope and I think we are very close to deleting all of that. Right now on master, I am already only using Python for the tokenization preprocessing. In principle the requirements for llm.c should be extremely minimal. I think this a few days of work that is high on my mind.

Biggest problem right now is finding a place that can host the 135GB of tokens for FineWeb100B. Will probably use S3 or something.

Related see: https://github.com/karpathy/llm.c/issues/482

[ Parent | on HN ]
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by metadat on Tuesday May 28, 2024 @07:26PM (#40506672)
Could this be a good case for a torrent?
[ Parent | on HN ]