If you don't communicate that you don't want to have sex it becomes a really murky topic, as you didn't give verbal nor non-verbal consent. But you also didn't give verbal nor nnon-verbal disconsent. This could be something you regret afterwards, but I don't think you can attribute that to malintent (or rape/sexual assault) on your sexual partner's side.
So removing the condom mid-act would be sexual assault, a broken condom wouldn't be. But the partner consented to sex with a condom.
Or in case a woman says she's on birthcontrol (or has a spiral) and it later turns out that was false, then you also didn't consent to that.
So basically my second point is that it takes a lot of language to define what is and what isn't sexual assault or rape. Condensing it down would be harmful to victims and people accused.