I really wouldn’t worry about “being viewed as a failure”. I think it’s admirable if you choose your passion over normality. You’ll be better at your work and at life, you’ll be happier and live longer. Job titles are, in my humble opinion, a trap.

So my advice is to set boundaries. If you don’t want to be sitting in meetings, then stand your ground or come to a compromise. If they don’t understand then quit! Find the path for you, life is too short to be unhappy and someone else’s pencil pusher.