News for hackers. Stuff that matters.

Is next token prediction the new "E = mc2"?

History has taught us that the most significant discoveries of formulas are often those that seem to be the simplest equations (even seem too simplistic to be true) but generalize the most.

Take Einstein's E = mc2 as an example.

My (maybe purely philosophical) hypothesis is now: Next token prediction might be for AI, what "E = mc2" has been for physics.

When Ilya Sutskever was asked in an interview what surprised in the most regarding the rapid advances in AI, his answer was, that "it" (meaning large language models and especially next token prediction) works at all.

And I agree. While it is easy to understand what next token prediction is doing, it is mindblowing what outputs can be achieved with such simplistic formula.

I'd like to discuss - do you think it's a valid statement to say "Next token prediction is the new E = mc2"?

( 7 comments | 4 points | on HN )
Re:Is next token prediction the new "E = mc2"? by Jensson on Thursday December 21, 2023 @04:19AM (#38718501)
Basic markov chain models are also next token prediction, so it isn't new, we have done that to make language models for over half a century at this point. It is one of the earliest things people did with computers.

Our modern LLMs are an evolution of that, but the next token prediction part is almost as old as computers.

Re:Is next token prediction the new "E = mc2"? by macilacilove on Thursday December 21, 2023 @03:29AM (#38718213)
The real invention is having a model that converges, does not lose the gradients, and having a framework to reliably train lots of parameters. Imo it is not historically accurate that next token prediction is the big invention. Some language models predict blanked tokens in the middle of sentences that are modern and interesting, and old language models like rnn-s are being used for quite some time, predicting next tokens one by one.
Re:Is next token prediction the new "E = mc2"? by dave4420 on Thursday December 21, 2023 @03:19AM (#38718170)
It took 40 years to get from E=mc^2 to the Trinity test, so ask again in the early 60s
Re:Is next token prediction the new "E = mc2"? by quickthrower2 on Thursday December 21, 2023 @04:49AM (#38718660)
That sequence of events had a lot of catalyst!
Re:Is next token prediction the new "E = mc2"? by dave4420 on Friday December 22, 2023 @09:45AM (#38734578)
There’s plenty of time for catalysts to show up before the 60s get here.

My point is really just that it’s too soon to tell.

Re:Is next token prediction the new "E = mc2"? by d--b on Thursday December 21, 2023 @07:30AM (#38719616)
It feels very strange to put "next token prediction" on the same level as "E = mc2".

"next token prediction" is an experiment that's puzzling, while E=mc2 is a theory (that Einstein developed to explain the puzzling experiments that were unexplained at the time).

It may or may not be that in the end there is a "simple formula" that explains why LLMs work, but we're not there yet.

Re:Is next token prediction the new "E = mc2"? by Tomte on Thursday December 21, 2023 @10:02AM (#38721272)
No. It's on the same level of categorical error as "A pumpkin is the new hedgehog."