On a fundamental level, there's no reason you couldn't pair the device, by pointing your phone at the tv and having some sensor figure out what you want. Maybe it could fall back to sending a code to your TV via infrared, maybe it could show a QR code when it first boots, maybe it has some signal-strength based proximity sensor so that you can tap your phone to the TV to pair it.
Maybe these sorts of device pairings can be backed up in the cloud so that if you replace your phone, they can be transferred to your new one.
Personally, I wish I could control the TV volume with my phone because the TV remote always goes missing, but I always have my phone. In particular having a fingerprint-sensor phone has made using it as a remote less painful since I don't need to enter my PIN, but maybe we will get remotes that work from the lock screen and turn on when you pick them up.
At this point where we always have a communications device in our pocket, I don't see why remotes need to exist at all.
I've used Xbox Smartglass before which works pretty well as a controller replacement. I haven't tried using my phone as a remote. Do you find yourself having to look at your phone to control stuff?
Yes, but unlike a regular remote I don't find myself pressing the same button 100 times.
I no longer watch broadcast television, so there is no need to flip through channels to see what is on, I just select what I want to watch on my phone, and then tell my phone to play it on my roku/TV.
The interface could still use a lot of work, but remotes are very clearly stuck in the last century and we've just gotten very used to them.