I was really into Convolutional Neural Networks back in the day and when you visualize the layers you’d get interesting patterns. I’m guessing the same would happen with LLMs but the patterns will be a lot more abstract but still some what recognizable.

Here’s an example of visualizing CNN layers; https://www.researchgate.net/figure/Visualization-of-example...