You can think of "acting on the world" as resolving questions about the world, like "what happens when I do X"?

You could observe the results of others acting, but it means that the questions you're getting answers to are outside of your control. So if you need to know the answer to a particular question, you either need to test it yourself, or hope that whoever you're watching will test it for you.

It's possible to learn from someone else acting and observing the results only if you understand what the observed person is doing and could map it to yourself. Which babies for example cannot do. They like you making crazy faces but when very young they don't learn how to do the same thing from that. I think for two reasons, first they don't know what you did to make that face (which in the "experimenting yourself" case they would have known even if it was a random action) and second they can't map what they see to actions with their own body. Both need to be learned first before you can learn from observation. While learning from your own (random) interactions with the world is possible from day 1.

A key part there is agentive intent. Pure passive observation lacks the information about which parts of the observed result were intended and expected (by the actor) and which are irrelevant noise or side-effects. In some sense the equivalent of looking at the data of a study and knowing which column represents the "intervention" and which column represents the outcome of intervention - as opposed to pure observation of those data points without knowing which is which.

We can learn well from a demonstration, where someone else is acting and we correctly understand the intent and expected results (perhaps they've explicitly communicated that, perhaps we just know implicitly through our previous shared experience) and observe what happens.

However, if we observe someone else - at an entirely different skill level - who is essentially testing a hypothesis (and we don't know that, we have no observation of the agent's internal) and getting useful data about where their world model diverges from reality, then it's not nearly as helpful for us to determine where our world model diverges from reality, that would require different actions and different data.