This would make sense if you're going with keywords, like "downward". But this logic only holds in language. If you compare it with human experience, then it no longer holds.

For example, did you notice that if you look at your own body, your legs are pointing "up"?