The problem with a utopian ideal is that utopias are, by definition, inhumane. Humans are not perfect, so you either have to make them perfect, or force them to conform to a perfect ideal which ignores and suppresses human nature. We war, we hunger, we lust, we hate, we fear... those aspects of our psyche aren't just going to go away because someone invents replicators and warp drives.
Humans that are like us struggling (and often failing, in their imperfection) to live up to an ideal in an imperfect universe are far more relatable and, ironically, more optimistic than simply having them be perfect to begin with and skipping over the messy details of how to get there from here.
Also, in practical storytelling terms, Gene Roddenberry's ideal humans evolved beyond a lot of the pathos that makes for compelling drama. Look at Wesley Crusher, whose character was basically Gene Rodenberry's Ubermensch, on one end of the spectrum , versus someone like Barclay on the other - which character would be more interesting to watch?
As for utopias, I've spent a lot of time thinking about them recently. I just finished taking a philosophy course on utopian and dystopian societies. The overall impression I got was that utopias are predicated on assumptions about human nature and they tend to marginalize or outright exclude anyone who doesn't fit in. This made for an interesting comparison with the dystopias.
I think Gene Roddenberry's argument is that we can develop a culture of excellence that overcomes the need for any sort of radical biological engineering to make people more agreeable. I don't know it's at all feasible but I find it interesting to imagine.
But then with the right acting and writing many different types of characters can become compelling - see Quark, for instance.
I don't think Discovery was darker than DS9 at all. I thought it was a fantastic attempt to explore how a highly progressive society deals with a culture who doesn't share their ideals.
They vast majority of the characters are huge believers in the highest ideals of the Federation. Throughout the series they struggle with sticking to those ideals, but at the end they manage to do it. I actually thought that the ending of Discovery was very similar thematically to Picard's decision with Hugh.
what is your take on Andromeda? I feel like in many respects it is really back to the original spirit.
We saw how deep this rabbit hole really went when they introduced Section 31 and even went so far as to do an episode where the Federation imposed martial law on earth, something I never would have imagined Roddenberry doing.