In practice, there have been a few games that have had patches released -- in almost all modern games a tiny fraction of the game is code, which can be easily replaced, leaving all the art and music which (hopefully) don't need as much patching.
That being said there are also many beautiful games that are amazingly fun, and the graphics do add to the experience, but they're more like the icing on the cake. That's not quite the metaphor I'm groping for, but you get the idea: the gameplay is a fundamental that has to be right, and then the graphics can really add something; without the gameplay the experience is always destined to be flat.
A lot of work goes into optimizing titles on consoles... it wouldn't make sense to skimp on such an important part of UX.
The modern need for patches and DLC makes that impossible now, in addition to the other reasons. But I think it would still be possible to greatly improve load times with a bit of optimisation - it's just not commercially important. Maybe if the console QA system imposed maximum load times it would be different - but then you'd see compromises made elsewhere, such as in level size.
(I play Cities:Skylines, where one of the more popular mods is simply called "Don't Crash", which lazy-loads other mod data to improve performance and reliability)