In almost every battle with them, they had a vast superiority in numbers. Even after the catastrophic defeats they suffer when the Good Guys win, they seem to be able to quickly recover their numbers. It probably helps that they can thrive in environmentr where humans can't: high mountains, caves, the volcanic landscape of Mordor etc.