Note that approaches such as hyperdimensional computing somewhat undermine your argument.

With HDC, a network learns to encode features it sees in a (many-dimensional, hence the name) hypervector.

Once trained (without cats), show it a cat picture for the first time, it will encode it. Other cat pictures will be measurably close, hopefully (it may requires a few samples to understand the "cat" signature, though).

It reminds me a bit of "LORA" embeddings (is that the proper term?), or just changing the last (fully-connected) layer of a trained neural network.