Robot, yes, or a sim. Simulators are like dynamic datasets. The "feedback loop" is also called the scientific method. Propose hypothesis, experiment, refine hypothesis.

People needed thousands of years to put together a few causal concepts about the world. AI would need its playtime too. It's not like a single person can come up with a causal model of the world by himself/herself. Just keep in mind that when comparing human intelligence with AI, so as not to ask of AI what no human can do.

There is one catch, if you have written (or have access to the internals of) a simulator that is faithful to what you want to simulate you already have the cause effect things worked out.

Not necessarily. For Go it is easy to simulate, hard to win. Many other things are easy to simulate at low level but have complex higher order behaviour (such as AutoML and ML graph compilation for GPU). A learning approach based on simulation is capable of finding super-human solutions, given enough compute.

Oh yes ! You are right. I was not thinking of artificial systems with well defined dynamics.

This is not true! There are techniques to determine causality passively. Here is one such example, which takes advantage of the behavior of noise to choose between "temperature causes altitude" and "altitude causes temperature": [PDF] http://papers.nips.cc/paper/3548-nonlinear-causal-discovery-...

I think that paper is using a heuristic that, for some problems, is a good guess which direction the causal direciton is.

Use your common sense though. Do you think we would have built the modern world (e.g. electricity) by passive observation of the world, or does really getting to the truth require controlled experiments?

All experiments require repeat measurements for confidence. None of this means that passive methods are impossible, which is what you were saying earlier.

That paper had no proof attached, because it's a heuristic and can easily be fooled by a bad setting. So maybe some causal relationship that are obvious can be learnt passively. But what I am saying is entirely consistent with what Pearl is saying in the article. To get to the next level of ai you need an experimenting agent. Passive methods in general are impossible for untangling causal relationship.