I also had the same experience, but not everyone has the same qualms about rejecting other's invitations. I know for a fact that I was considered rude for politely rejecting drinks when I decided against drinking, possibly cultural.