You do not have to defend your trademark against people who are not infringing it. You can only infringe a trademark when you use the mark in the same context it's registered in.

IANAL.