I feel the same about it, but to be fair, this is one of those (many) rules that was essentially invented in the 18th century by prescriptive grammarians who took a guideline of style and turned it into a rule of grammar. “Less” has a long history of use with count nouns—originally as a counter, with the genitive, like “less of men” (cf. “shelves of books”).