You could maybe make a case for CNNs, but the fact that they're feed-forward rather than feedback means they're fundamentally representing a different object (CNN is a function, whereas the visual system is a feedback network).

Transformers, while not exactly functions, don't have a feedback mechanism similar to e.g. the cortical algorithm or any other neuronal structure I'm aware of. In general, the ML field is less concerned with replicating neural mechanisms than following the objective gradient.

Thanks for the considered answer. What is the cortical algorithm? (Yeah, it's been quite a few years since I did any bio psych...)

As far as I understand it, there's a standing hypothesis that cortical columns have a similar structure that is designed to learn arbitrary patterns via predictive coding, and that a lot of human plasticity arises from the interaction and flexibility of these columns.

Numenta has attempted to implement a system to this effect (see the wiki page https://en.wikipedia.org/wiki/Hierarchical_temporal_memory) for quite some time with not particularly much success.

Personally I think the kinds of minds we create in silico will end up being very different, because the advantages and disadvantages of the medium are just very different; for example, having a much stronger central processor and much weaker distributed memory, along with specialized precise circuits in addition to probabilistic ones.