People keep underestimating what next-token prediction can do, but they’re not wrong that it’s how LLM’s work.
It’s actually a good question: what will next-token prediction be able to do on new datasets? The error is thinking you can answer it, even in broad terms.
Haven’t we pretty much figured out these things are doing more than just predicting the next token at this point?
There’s probably a lot to be done with a “prediction machine”, birds aren’t all that smart but can catch bugs in midair.