Yes you do. You get exactly what you pay for. If you pay for a dozen of German inspectors who think Chinese factory produces junk that is outside the spec, a dozen of German inspectors with that attitude are going to do inspections. If you do no want to pay those German inspectors, you are not going to get those inspections.
In most of the cases companies do not want to pay for it. Apple manufactures amazing iPhones in China. Apple iPhones easily last 5-6 years. Blu manufactures crappy Android phones. They barely last a year. Apple's iphones cost more to manufacture than Blu phones sell for.
But when you build ongoing relationships, you get benefits from not pushing for the bottom dollar at all costs.
A company with a reputation for quality can charge more, because you don't need observers and can just rely on them. And they want to deliver on the reputation, so that they can keep milking it.
JS Mill has some interesting observations going back a ways.
Auditing is still recommended.
Ignoring auditing for a moment:
The trustworthy reputation is quicker to build, if the quality of the product is easy to ascertain quickly.
But if you are a company in a sector where that's not possible, the reputation for quality becomes even more valuable.
And you can potentially ride it out for a long time in an exit scam. https://en.wikipedia.org/wiki/Exit_scam
But generally value derives from multiple independent credible verifiable audit mechanisms.