It is trained to predict tokens but to do that well it needs to develop some internal structure. That structure is mostly opaque but it appears to model the world increasingly well with scale.