You don't have to say it, what do you mean by we?

Others may say it, but there's a difference between being annoyed that other people say something, and turning your comment in such a way that others saying it looks like you're being prevented from saying what you want.