femtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer.

Everything is implemented from scratch, including the tensor processing logic along with training/inference code of a minimal GPT architecture.

Thanks for making this. Was searching for a pure rust GPT implementation just the other day!

> After hours of training on the Shakespeare database, on a 300k parameter model, this has been the output

I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?

I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).

So what are the expectations of this GPT model? Should it be bound by the same limitations?

Does this have potential for running on an SBC? Mine only has 2GB of RAM and I'm interested in running a toy language model on it.