This is one of those things you might not get unless you've been in the situation yourself- if the shoulder is small or dangerous (e.g. debris), taking the lane is the safer option. If you don't, you end up getting close-passed/buzzed as people trying to squeeze between you and the opposing lane. Some particularly bad humans will even get close on purpose (known as a "punish pass").