Microsoft's betting the farm on the idea that a consumer will leave the xbox on 24/7 in a low-power state and have it as THE hub for all things digital entertainment. Your TV goes through it for advanced features such as TV channel guides, extra content, etc. You play games on it. You chat, take video calls and the like on it via Skype. You steam music/movies to it as your media player. All of it is controlled via the Kinect gestures and voice recognition. It is also why they're planning to push updates/etc. through in idle times (e.g.: overnight) when the device is left in low power but able to do basic tasks.
Basically, MS wants to take the beachhead they established with the first two generations and expand it. They want the Xbox One to go against the Apple TV and XBMC media centres and the like to become the all-encompassing portal of your entertainment.
Unless, of course, the pass through works when the device is turned off... However, I haven't heard any mention yet that it does (does anyone know if it will support this?).
And plus, if you had any other multimedia devices (e.g. a Blu-ray player), wouldn't you still need to use the other inputs on the television anyways?