This is contrasted with something like Imperial units, where every division of e.g. distance is supposed to be roughly based on some physical object. That's why you end up with 12 in/ft, 5280 ft/mi etc. Or alternatively you end up with metric-imperial hybrid units like the kilopound.
Not to mention, what units are convenient vary depending on what you do. For instance one of the SI alternative units for energy is the electron volt (eV), the work done to move an electron through a 1 volt potential. This is a tiny amount of energy on human scale- a common analogy is that 1 MeV (10^6 eV) is enough energy to make a single grain of sand twitch a little bit. But, if you're a nuclear physicist (or maybe a chemist) then eV are typically much more convenient than say Joules.
A certain amount of arbitrariness is inevitable at human scale.