It might have been said by somebody here but I believe these models are like information compilers. Sure they seem like “next token prediction” but AGAIN this undersells them. I once encoded leetcode solutions by replacing each keyword, variable and operation with another token, then rotated the lines. The model could tell exactly which problem the code solved. They can naturally find signal, ALWAYS!!