I think it originated on Reddit and it's since been parroted here in the comments on basically every smart TV thread but I have yet to see actual evidence. It seems like a trivial theory to test - disconnect from your wifi or change its password, wait for ads to suddenly reappear on the TV (evidence it got a network connection from somewhere).

Similar FUD is being spread around HDMI's Ethernet channel; a way to carry network data over an HDMI cable. I have basically never seen it in the wild on any consumer device, but even if it were, it would still require the other device to cooperate and act as a switch/router to share its connection to the TV. Yet despite that every time smart TVs and privacy comes up someone mentions this.