You generally have your phone with you, so you don't need to keep track of a separate device or devices. Unlike a TV remote, which is used near your TV, a house remote needs to be used anywhere. Everyone in your household can have their own remote (phone), so there's no passing the device around.

The types of controls are varied, and a color touch screen is generally the best option. Light switches, dimmer lights, colored lights, fan controls, AC controls, etc. While I can imagine creating a remote that handles all these different devices, it's going to be a challenge to create hardware that services all these different devices intuitively.