That's why it is called "tradeoff between exploration and exploitation". Exploration costs resources too, and has a lower chance of generating rewards in the short term. But without it agents can be stuck in a local minima without being able to "jump" to a better local minima.