The closest well-known example of something like this is deepmind's "perceiver" architecture.

https://www.deepmind.com/blog/building-architectures-that-ca...