I think the most common goal of vegan or vegetarian diets is to reduce animal suffering, not to eliminate any part of your life that is even adjacent to animals. As you've pointed out, the latter would be impossible.