Phone as remote control is great - OP might want to think about QR-code as a quick method of inputting the URL. I believe this is how YouTube remote control works.

A further improvement might be to use OPs internet-connected server to proxy the commands so the phone doesn't have to be connected to the same WiFi network, though I accept this is a very limited use-case and incurs some cost.

EDIT: Also - good work, OP!