People with whole Bitcoin don't necessarily care about a $50 transaction fee. They still drive the transaction fees up by wasting space and thus cause people who decide to use segwit to pay more.