A game is only a game if all the players have a shared set of rules and can trust each other to stay within them. As soon as someone breaks the rules the game is over, and the freedom provided is lost.
You're constructing a scenario to justify an abstract argument. The wall is the only salient feature in the walled garden metaphor. There's no terror lurking outside.
If you mean to imply that Apple's oversight has some advantages, such as security, that's a separate issue.
In the context of Apple's app store rules, the wall refers to Apple's rules. The absence of those rules does not reduce your freedom as a developer; you're can still abide by them without Apple requiring your to do so. But you have more choice without the rules.
For example, justice in a free society should not be arbitrary and capricious. It should be fair and equal, and applied with discretion. A society where minor rules are difficult to understand but are enforced to the maximum extent is a tyranny.