I guess because plainly visible "tall masses" are not the only things contributing to the geoid? E.g. earth crust thickness might dominate.

In the context of this discussion, we're not so much interested in the absolute shape of the geoid, as in how much it might change because of melting ice sheets