They can't legally enforce it because the BBC guy has no special power to inspect your telly set and it's not technically a tax (as long as you don't consume BBC content). They also include YouTube/iPlayer consumption of BBC so it's unenforceable on a technical level too.

They've tried to scare people into compliance since the 50s: https://en.wikipedia.org/wiki/TV_detector_van