While this is true, we (at least in England) would normally say "leave the buoy to starboard" (though when racing we'd call it a 'mark' rather than a buoy). Perhaps it's habit, I'm not sure, but it's generally unambiguous.
In the context of a channel, the buoys (marks) are coded port and starboard, coloured red and green, and shaped like a can or a cone respectively.
When navigating a channel into harbour, the port marks are left to the port of the ship, and the starboard marks to starboard.
Except when weird species seize their feisty foreign neighbours.
In Region A (Europe, Greenland, Africa, most of Asia, most of Oceania) port marks are red and starboard marks are green.
In Region B (North and South America, Japan, South Korea, Philippines, Taiwan) port marks are green and starboard marks are red.