Why would you expect patterns to appear?

Everything is a different pattern. Noise isn't random - we just call 1 very specific unique pattern of it 'random'.

I was really into Convolutional Neural Networks back in the day and when you visualize the layers you’d get interesting patterns. I’m guessing the same would happen with LLMs but the patterns will be a lot more abstract but still some what recognizable.

Here’s an example of visualizing CNN layers; https://www.researchgate.net/figure/Visualization-of-example...