I don't really disagree, but I think you've set up a bit of a strawman - in the general case I think one is more often harder than the other. You can obviously look at extreme outliers for contradictory examples, but the general case is more interesting/common.

There are also some things that are harder or easier for people because we have innate hardware for certain things. Recognizing facial expressions (or really interpreting any sort of visual input) compared to doing complex math in your head. The Type 1 vs. Type 2 systems from thinking fast and slow - some things are generally easier for humans compared to other things.