I'm not convinced that a vegetarian diet that includes factory farmed dairy involves much less cruelty than an omnivorous diet. Maybe more, if the omnivore were to avoid factory farmed meat and dairy.
That said, I'm not a vegetarian myself, because I don't believe it's in my rational self-interest.
If you believe that factory farmed dairy is just as cruel, then don't consume factory farmed dairy.
I don't believe its in my rational self-interest
Unless you count the environmental impact, it shouldn't be about your self-interest, but rather doing something that some see as cruel and unethical treatment of other living creatures.
(PS: I'm not vegetarian either, so I'm not attempting to change your opinion)
This might clarify my position a bit: https://en.m.wikipedia.org/wiki/Rational_egoism
Can I politely suggest you uncurl from the solipsistic ball you've tucked yourself into? It's nice out here...
Seriously, there is nothing about adopting a philosophy of rational self interest that implies solipsism. In fact, solipsism can clearly be shown not to be in one's own rational self interest :)