Is the power of predictive text that crazy?
That's what I find so interesting about large language models: they consistently demonstrate abilities that were not predicted when they were first invented.
I don't think anyone working on them initially expected them to be able to write Lisp and Pascal, certainly not as well as they can already.
> It turns out it is!
Intelligence was in the data, we were just its vehicles.
The same happens with the huge volume of text this model has been trained off. We cannot even begin to grasp its size. It is more. More than that. Yes, even more.
Let’s say you train a predictive engine on the entirety of fiction and nonfiction works of the 19th century. You ask it to produce C code, and it does! That would be obviously crazy.
On the other extreme, imagine a predictive engine that has been trained on every bit of C code that has ever been written. You ask to produce C code, and it does! Maybe not so crazy this time.
It's effectively the same thing: given some words it predicts which word should come next.
But unlike the iPhone keyboard it's been trained for months on multiple TBs of text, and has the ability to use ~4,000 previous words as the input to its prediction.