Your intuition is out of proportion, because astronomical objects are big. Really big. A spider riding on the left side mirror of a 18-wheeler has a better chance of drastically changing its course.