You hear a cry “man overboard on the left!”
Where is the overboard person? Are they to your left, the ships left, or the left of the person shouting?
If the cry is instead “man overboard on the port side!” there is no question.
This is about as worrisome as "you are the fire department, and you are told you need to rescue some people on the 14th floor. Quick, does that mean you go up 13 flights of stairs, or 11?" Realistically, you aren't getting it right on spur of the moment decision making.
I think starboard and port make more sense here because they only describe parts of the ship, so the meaning of the terms is completely unambiguous.
A crew (any kind of crew) requires exceedingly clear communication. "Ship left" and "ship right" could work, but what if the environment is noisy and you're not sure if the first word was "ship" or something else? Domain-specific, unique terms are fantastic tools for crews.
Speaking as somebody who sails occasionally, to me "left" and "port" are just very different concepts. It's sort of like driver's side and passenger's side of the car, but more intense. So I think the desire to use "left" mainly comes from people who don't spend much time on ships, an audience, not particularly important to sailors. Sort of like how non-programmers might struggle with some of our technical terms, but we'd never change.
I think it's just historical.