Dark patterns ("bad UX") and broken discovery are a means to the end of "commoditizing the complements".

Bad UX in an Apple component is not a "dark pattern". It's just, well, bad UX.

Knowing Apple, knowing its history and people that work there and the things they care about, I think Apple is incapable of deliberately producing a bad UX. When it happens, it's caused by other factors rather than being an explicit decision. In the case of App Store discovery being broken, I think it's caused by the fact that the App Store seems to be run largely the way the Music Store is run (with the addition of app review), and nobody complains about search / discovery in the Music Store, but it just so happens that the infrastructure and design around the music store doesn't work quite so well when applied to apps.

Why do you think there is no visible investment/improvement in App Store UX, from the world's premier design proponent?

Because the people that care about design don't really care about the App Store (they care about products instead), and the people that run the App Store don't really have an incentive to try and make dramatic changes in something that, from their perspective, is wildly successful. Change might break things, especially when operating on the scale that Apple is (just think of what the load must be on the App Store).

Apple executive leadership is responsible for successful change in all departments, they could make this a priority if they wanted.

That's true. And if you care about this issue, you could try appealing to Apple executive leadership (emailing tim cook, or running a PR campaign designed to get Apple executive leadership to see your arguments).

Alternately, one could invest in startups that solve this problem.

I don't know how much has come from it yet, but here's some an example of some investment:

"Apple Acquired Search Startup Ottocat To Power The ‘Explore’ Tab In The App Store"

http://techcrunch.com/2015/04/06/ottocat-apple/