I think the biggest culprit here is the economy. Shareholders and investors don't like it when things just keep working like a well-oiled machine. It means their share price stays stagnant. They always want "more more more" and earnings, not stability and usability. So that pressures management to keep on finding ways of changing stuff for no real good reason even when they have already invented and reached the optimal UI. Search bars embedded in the window titlebar, start menus that change the order of things every time you click them, these all scream "we can't think of anything else, let's just change some stuff around".

A similar effect seems to happen even with e.g. Canonical and Ubuntu. Gnome2 had an awesome optimal UI and 90% of Linux users I know prefer that kind of no-BS UI but "management" wanted that Unity and Gnome3 nonsense just to make it look like they were making progress when they were really going leaps and bounds backwards in usability.