It's not about privacy, it's about security. If there's a backdoor in a HSM or network interface, that backdoor can be used by others as well. That might start with foreign nation states, but might eventually leak to regular private persons or entities as well.

A backdoor is an extra attack vector with often very unfavorable properties that you as a user are unaware of.