We also assume that we only get a virus when we notice it.

The reality is probably more like: You're exposed to millions of different viral antigens over the course of your lifetime; Many of them don't have a mechanism to really hurt you, and nearly all of them are dealt with efficiently by your immune system.

Not only do they not cause symptoms, we don't even have a mechanism to identify them unless we already know what we're looking for.