Re:Pure Rust implementation of a minimal GPT language model
by mrkeen on Saturday June 03, 2023 @08:43AM
(#36176208)
> After hours of training on the Shakespeare database, on a 300k parameter model, this has been the output
I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?
I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).
So what are the expectations of this GPT model? Should it be bound by the same limitations?
I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?
I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).
So what are the expectations of this GPT model? Should it be bound by the same limitations?