But if the team is a "black box" with their only contact with the rest of the firm being mediated by the Product Owner, then that team could be anywhere. I always saw Agile as a sly trick to make outsourcing easier by packaging the team into a self-contained unit.
Also, that black box team is by definition not "Agile" because the engineers are not face-to-face with the users, nor communicating with the users daily during development.
And if your "daily" communication with "users" involves specifications and talking to managers, it doesn't matter whether you're in-house or outsourced.
It looks different for modern consumer products, because your software is typically a small part of someone's life. They don't want to talk to you daily. Instead you have a product manager acting as proxy for the users. But if you have good instrumentation, you're still getting user feedback daily.
Even in the consumer context, you can talk to real users frequently. At my last startup, we had user testing every Tuesday afternoon. We'd book 4-5 sample users every week, and we'd always have some question we wanted to answer. And plenty of places have small beta tester groups that provide fast feedback on new features.
That all depends on a fast release cadence and a tight relationship between product management and developers. But once you have it, it's addictive for product managers. Oursourcing basically becomes impossible, because their turnaround time for answering a question drops from seconds to hours, and the turnaround time for shipping a small change goes from hours to days. This forces batch size back up, which is not only vastly less fun, but also reduces ability to explore the product space and find high-value business wins.
Every time I've been asked about outsourcing, I've asked, "Will you send your product managers there too?" 99% of the time the answer is no, because they can see how that would impair the information flow. The trick is getting people to extend that desire for connection and fast iteration past the managerial tier.
Agile helped. Many programmers appear to find unions distasteful, so they needed some other way to try to protect their jobs from outsourcing. Agile claims "face-to-face" is most efficient.