Swiping down only refreshes when you are already at the top of the page, and it even gives some visual feedback to see that you have indeed reached the top and you are now initiating a refresh gesture.

I'm a quite energetic swiper and, to be honest, I have never found a situation where I wanted to return to the top and ended up refreshing due to too much swiping. But YMMV, I guess.