It's also hard to make money when there's so much free software available, some of which is quite good. For example, it would take a huge amount of effort to improve on the functionality of Libre Office, Paint.net, Audacity, Wireshark, etc. (just to name a few of the free programs I use) but since people can get these for free, a competing product would have to be a whole lot better to convince people to pay for it.
I think there's still piles of room for innovation on the desktop, both for open source and commercial software. We've reached a lot of local optima, but things can still be a whole lot better.
For an example of a Photoshop and Paint.net competitor, check out http://www.pixelmator.com/mac/. They've been around for a few years and seem to be doing quite well. BitWig is a new competitor in the DAW space: http://www.bitwig.com/en/bitwig-studio.html.
This is a core element of my questions that other answers are unfortunately not addressing. Even if you make software that's less clunky than legacy-grade Adobe Kamatsu-tractor like offerings, how do you make make switching lucrative to the point of achieving self-sustainability?
Take Sketch for example (http://www.sketchapp.com). Everyone on our team used to use Adobe Illustrator, and now everyone uses Sketch instead, because for what we do, it is simpler and better.
Money? That's harder, unfortunately. It's hard to innovate when people want to cling on to what they already know. Make an image processing & digital painting utility, and people would only complain that it's not photoshop. Make a word processor, people complain that it's not MS Office.