Dig a little deeper into the LOTR backstory. It was not so much a conflict between men but between supernatural forces, angels. It is a biblical metaphor, a story of higher powers settling bets by sending agents to earth to promote various agendas.

https://en.wikipedia.org/wiki/Wizards_(Middle-earth)