That is true and a good point. But when you think about it, with a large enough budget you could create hardware backdoors in microprocessors in the manufacturing plant without the chip designer ever knowing about it.
Then you're in a weird world, where a flagship smartphone has backdoors created by Samsung, Google, three different governments and the weird guy that worked with the chip design in team #7.
There's so many ways to hide something within the software or hardware that it's incredibly hard to vet its security.
The US is currently in a cyber cold war with china. It's very probable that there are hardware backdoors within these products.