The point isn't just about causality. As an undergraduate, one of my philosophy professors (Bill Lycan) told a class "never attempt to define anything in terms of counterfactuals. No matter what it is, there will be decisive counterexamples."
To be super-clear, this is not a statement about Pearl, and it's not denying that counterfactuals are interesting, just a claim that you can't define much of anything in terms of them.
P.S. He actually said "analyze", but the way philosophers use that term, it's appropriate to make the substitution to avoid confusion.
> "This happened because I did that" seems to imply "Had I not done that, this would not have happened" -- a counterfactual.
The point is that constructing the counterfactual requires a more than just knowledge of a single causal link, but a broader set of causal knowledge about many causal links.
Language is messy and ambiguous and we do often colloquially use "X caused Y" to imply the truth of the counter factual "If X had not happened Y would not have happened". Interestingly, a different tense such as "X causes Y" generally does not have the same counter factual implication.
Couldn't you have a scenario where your action was the direct cause of an outcome that might have come about some other way without your involvement at all?
For instance, if a ball rolls down a hill because you pushed it, but a breeze blows a moment after you pushed the ball, if that breeze was strong enough to push the ball down the hill too, then you have a scenario where "that ball went down the hill because I pushed it" is true, but "had I not pushed the ball, it would not have rolled down this hill" is not true.