How behaved they are visually depends on what drives variance and what you’re hoping to see. There are certainly some nice properties in some dimensionality reductions, but if you flatten a space of faces it’s less likely that you’ll get the property of “brown hair” as a query embedded in any visually interesting way than actually putting in a face as a query.

More clearly, symmetric retrieval is easier to visualize in a dimensionality reduced space than asymmetric retrieval.

I suspect that some form of multi vector document embedding would be more understandable in the reduced space than this single vector representation.