There's a fundamental difference between being spoken to and being spoken about. You have a right to prevent others from speaking to you but not about you (at least in the USA).