I'm not convinced that a vegetarian diet that includes factory farmed dairy involves much less cruelty than an omnivorous diet. Maybe more, if the omnivore were to avoid factory farmed meat and dairy.
That said, I'm not a vegetarian myself, because I don't believe it's in my rational self-interest.
If you believe that factory farmed dairy is just as cruel, then don't consume factory farmed dairy.
I don't believe its in my rational self-interest
Unless you count the environmental impact, it shouldn't be about your self-interest, but rather doing something that some see as cruel and unethical treatment of other living creatures.
(PS: I'm not vegetarian either, so I'm not attempting to change your opinion)
This might clarify my position a bit: https://en.m.wikipedia.org/wiki/Rational_egoism
Can I politely suggest you uncurl from the solipsistic ball you've tucked yourself into? It's nice out here...
Seriously, there is nothing about adopting a philosophy of rational self interest that implies solipsism. In fact, solipsism can clearly be shown not to be in one's own rational self interest :)
The question is then one of how you separate the edible part from the inedible part. Given that you can never do this precisely, you're going to end up either playing it too safe and leaving edible meat behind, or going too far and extracting inedible cowbits. Then the question is: is it a bigger sin to run the risk of harming humans, or to leave behind a fraction of a percent of the cow as a safety margin? Thought about in those terms, the edible meat left behind would be used, just not as food.