News for hackers. Stuff that matters.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by ideashower on Thursday January 30, 2025 @10:39AM (#42878730)
Can anyone share what a training dataset would look like for something like this? What are some use cases?
Re:SmolGPT: A minimal PyTorch implementation for training a ... by efm on Thursday January 30, 2025 @12:44PM (#42880089)
Karpathy's nanoGPT has a full training pipeline using Shakespeare. [1]

The use case for this is learning in simple example.

[1] https://github.com/karpathy/nanoGPT

Re:SmolGPT: A minimal PyTorch implementation for training a ... by qrios on Thursday January 30, 2025 @05:55PM (#42883049)
It is downloading a dataset (TinyStories) from huggingface[1]. Here you can drill down deeper into the structure and content of the source data.

[1] https://huggingface.co/datasets/roneneldan/TinyStories