I think the confusion results from the fact that people use the phrase "pass by reference" without ever explicitly stating what the referenced thing is.

In the original sense of the phrase, I believe, that thing was the variable itself in the form of an alias, not the object/value that a pointer variable might be pointing to.