Having been a moderator of a large, 60k-ish person subreddit for a couple years, I can assure you that that's not the case: removing spam and other obvious rule breaking (sending users death threats, racial slurs, etc) took no more than a few minutes a day. Well over 90% of my fellow moderators' time was spent exceeding their authority: harassing people they just disliked, removing posts for personal biases, and so on.

It's a job that attracts that sort of person, people who just want things to function smoothly will get fed up with it quickly, the only people who stay at it will be the ones who want to mold the community to their vision (whether it wants to be molded or not).