you don't have to "worry" about wifi war driving per se - you just have to know that google already does it and likely sells that information to other companies. No need to worry except to know its likely already occurring.

And in terms of legality - you likely give them permission to do whatever they want in the terms and services you agree to on first boot, or install an app on your phone, etc. And you have no control of the TV's wifi - unless you hack the encryption or get direct access to the firmware you might never be able to tell whats going on. That's definitely beyond my capabilities.