I've tried to get into Ian Banks. One feeling I get is that there's two kinds of optimist sci fi. Technology magically solves our problems - or technology can be both force for good and bad, let's use it for good. I have a feeling that Ian M Banks writes the former but I'm probably wrong here?
There can still be interesting conflict even if the bigger problems have been solved. Dispossessed is one good example I think. The conflict is partially about flaws inherent in the "better" society that they created.
And in Star Trek and to some degree in "The long way to a small angry planet" the conflict is between the larger society and the hold outs.
It remindsme a bit of how internet communities work.
I think it takes its premise much more seriously than star trek does. E.g. Star trek has free replicators for everyone and yet everyone is spending tons of money at Quark's for some reason. Which is fine, star trek has a lot of other good qualities, but i don't think it does a great job of exploring the implication of its technologies.
Some of it is, yeah. But I think part of what made her novels interesting was how she was (often) able to resolve conflict without violence. I think that's where the optimism comes from.
I was watching the recent tv adaptation of asimov's foundation series, and one of the most striking things was how much characters (including the good guys) resorted to violence. Where in the books the entire theme was that brain beat brawn and the main characters almost never solved their problems with violence.
Optimism is powerful, but it usually comes with impatience. We've shown an ability to decrease the prevalence of war globally, but the fact that improvements have taken longer than a generation and have not been monotonic or absolute makes it very hard to maintain a belief in the possibility of future improvements. The pace of change in the world can't keep up with the emotional rhythm of human optimism, disappointment, and disillusionment.
Also, many well-meaning people fight to tear down optimism because they identify it with complacency. Other well-meaning people fight to tear down optimism because they want people to focus on a different problem, so they want them to believe that there can be no progress on X without progress on Y.
As a result it is fashionable to deny that anything done in centuries of human struggle has made any positive difference in the world. How that is supposed to motivate anybody to add their own efforts to the struggle is left as an exercise for the reader I guess.
Consider classics like "The Cold Equations", where the institutional solution to space stowaways is to execute them, without so much as bothering to put up a warning sign first.
That said, Ursula K. LeGuin did author such a book, _Always Coming Home_, which is far more interesting and engaging than it has any right to be.
That said, my go-to reading for when I'm really depressed is her _The Lathe of Heaven_....
Its always weird to me when people use U Le Guin as an example of "optimistic" sci fi. Some of her writing is extremely bleak. Of course she is pretty prolific and has written a lot of different types of things.
Fundamentally i think the issue is that novels usually have to be about conflict. If everything is peachy you dont have a novel. I think it takes much more skill to write an interesting optimistic near future sci fi novel that is actually about the technical change instead of just using it as a setting.
Most of the optimistic technology novels i can think of that i liked were far future - Iain banks. Maybe greg egan although optimistic isnt exactly the right word. They show an interesting world, but its clearly not our world. Is it really an optimistic take on the future of technology if the technology in question is hundreds of years in the future?
Perhaps you'd like something like "Song for a new day" by Sarah Pinsker.
I dont really know if it fits as "optimism", but i'd like to mention "The Fortunate Fall" as a really interesting novel about technology.