> couldn’t protons in any reference frame be going quite fast relative to you?

By the laws of physics, yes. But most of the stuff in our universe, and definitely in our galaxy, tends to move roughly at the same speed.

See https://en.wikipedia.org/wiki/Comoving_and_proper_distances#...