Following this logic further, we might be best off if we reduced the population to a single person whose ideal, perfectly happy life is that of a hermit.

Philosophers have given this a lot of thought: the "repugnant conclusion" is what to Google for.