>I wonder how much the negatives of being short are caused by lower self esteem due to simply what people say and think about heights.

There are statistically proven negative points in being not very attractive in general. Beautiful people have (significantly) higher chance to become successful in their job and/or businesses.