You're constructing a scenario to justify an abstract argument. The wall is the only salient feature in the walled garden metaphor. There's no terror lurking outside.

If you mean to imply that Apple's oversight has some advantages, such as security, that's a separate issue.

In the context of Apple's app store rules, the wall refers to Apple's rules. The absence of those rules does not reduce your freedom as a developer; you're can still abide by them without Apple requiring your to do so. But you have more choice without the rules.