Well, you did say any questions. ;)

Speaking of questions, it seems to me that in light of the universal approximation theorem, the question isn't "can a transformer represent a Kalman filter," but can it do so simply and somewhat efficiently. That seems to be what you're getting at here, yes?

Yes. Also the universal approximation theorem won't directly yield the kind of "strong" approximation we prove, which is uniform in time and holds for an arbitrary (possibly adversarial) sequence of disturbances.