How can I give guests control of my TV via their phones? As far as I know, that can’t be done automatically with iPhones, let alone people who don’t have iPhones.

They launch the Remote app on their phone, select the TV from the local network (if you don't banish them to a guest network), and pair with the code, same as you adding your phone.