> A CLIP model consists of two sides, a ResNet vision model and a Transformer language model, trained to align pairs of images and text from the internet using a contrastive loss.

It isn't surprising that a model trained for alligning photos and corresponding text labels has neurons that have learned higher semantic concepts as that is exactly the goal of generalisation.