Not everyone will notice/remember, but some will, especially if it is logical - like 'the "heavier" end is down for more stability.'
Unfortunately it looks like the accepted orientation of HDMI connectors is a precarious "heavy side up."
Now "up" isn't always up, connectors on the side of a monitor for instance should often have the logo towards you since they are usually vertically mounted.
With a second of thought it mostly works as expected in my eyes. But there are exceptions.
HDMI isn't so bad.
S-Video connectors were terrible. Worse than USB in every way possible, if you can imagine that.
HDMI and Displayport are just hard to come in from an angle. DVI and VGA are much simpler in that regard.
Svideo was often quite simple, just find the whole and you are halfway done. Then carefully twist it and you can feel when it is correct. Even when you have no idea where the slot even is to start.
With HDMI and DP being so similar it is easy to mix them up.
You don't, for example, hear the same complaints about DVI, VGA, Serial, or Parallel ports. Why? Because you can feel the right orientation when you plug things in.
If you can't feel the right orientation, then the next best thing is to make the orientation not matter.
This is why S-Video sucks. Because you MUST see to work with it.
Of course, the main reason people have a problem with USB is because they are constantly interacting with it, plugging and unplugging things.