The model is doing more than spitting out text, it's spitting out language. Language has much different statistical properties than random text, and we would expect that we would need a significant amount of complexity to avoid generating complete trash from what is effectively an inferred probability distribution over long passages of language-bearing text.