For example, consider spelling a word. The briefest approach is to say the names of letters. But these are similar, often differing only in one phoneme. E.g., P and B differentiating sounds are both bilabial plosives, differing only in whether they are voiced. But people for whom clarity is important will use something like the NATO Phonetic Alphabet. There, you'd say "papa" and "bravo". [1] It is much less brief, but much more clear.
Plus, that's not how brains work. Using a word for different things that change depending on context is like reusing a variable, with a compiler that randomizes global and local scope. In aviation they watch out for "Regression to prior learning", in the kitchen I watch out for putting something in the microwave instead of the fridge because the action is similar.
If you ask me which way is left, I'll have to stop and think not only about what left and right even mean, probably looking at my hand to see what makes the L shape, but I'll have to wonder who's left you're referring to, in addition to performing a mental rotation, an extremely error prone operation that I basically avoid at all costs, if my heading is not the same as the ships.
Any small about of simplification or added reliability there is a good thing.
Especially in historical context. Assuming the port and starboard were not mostly symmetric and had features specific to docking and steering, you're actually referring to something specific, not an abstract direction.
These conventions are designed for the 95%, not the 5% or 75% or whatever who can instantly tell which way is left relative to any given orientation without any effort.
A familiar experience... I have to catch myself when I start to throw the spoon into the bin and the teabag into the dishwasher!