Every vision task needs edge/contrast/color detectors and these can be all mostly the same across ViTs.

The surprising thing is inter-modality shared variation. Not intra-modality which is quite obvious and expected.

I would like to see interpretability work into whether these high variance components are low level or high level abstractions.

It hints there may be common higher-level abstraction and compression processes in human consciousness.

The "human" part of that matters. This is all human-made data, collected from human technology, which was created to assist human thinking and experience.

So I wonder if this isn't so much about universals or Platonic ideals. More that we're starting to see the outlines of the shapes that define - perhaps constrict - our own minds.

Now I wonder how much this "Universal Subspace" corresponds to the same set of scraped Reddit posts and pirated books that apparently all the bigcorps used for model training. Is it 'universal' because it's universal, or because the same book-pirating torrents got reused all over?