It might seem pointlessly semantic, but I think it's important with issues such as sexual assault and rape. Especially considering the people on either side of this likely have their lives ruined.
"Eh, whatever" and "no, not without a condom" aren't the same thing.
If you don't communicate that you don't want to have sex it becomes a really murky topic, as you didn't give verbal nor non-verbal consent. But you also didn't give verbal nor nnon-verbal disconsent. This could be something you regret afterwards, but I don't think you can attribute that to malintent (or rape/sexual assault) on your sexual partner's side.
So removing the condom mid-act would be sexual assault, a broken condom wouldn't be. But the partner consented to sex with a condom.
Or in case a woman says she's on birthcontrol (or has a spiral) and it later turns out that was false, then you also didn't consent to that.
So basically my second point is that it takes a lot of language to define what is and what isn't sexual assault or rape. Condensing it down would be harmful to victims and people accused.