Maybe the difference is I'm not vegan or vegetarian for philosophical reasons, purely health reasons.

I don't really care what the label is to be honest, it's just how I eat.

Vegetarian is probably closer to being more accurate. But then again I will happily order a steak at a restaurant. So maybe I don't deserve any of these labels at all :)

A better way to put it is I prefer eating vegan rather than I am vegan

In a sensible world, vegetarian would mean plant based diet and vegan would still mean no use of animal products at all (extending beyond diet).

But you can be a vegetarian and eat fish apparently.

pescatarian (if we want to get deep into dieting labels)

Yes, I understand there is a label for a diet that includes fish and not other animal protein.

But my point is that people more or less ignore that there are specific labels with clear meanings and go for labels that have a vibe that they like, regardless of their accuracy.

Like 'vegan' has a philosophical overtone, it isn't just a label for a plant based approach to eating, it's a strict approach rooted in a rejection of (at least perceived) animal exploitation. So someone that eats plants most of the time and grabs the occasional steak as a treat isn't really anywhere close to vegan.