I think a lot of people get annoyed (probably including myself) because the term sort of implies that there's great work-life balance which definitely isn't the case with a lot of small businesses.

Actually, what I was referring to is just called "company" or "business", or at least used to be called that.

I understood "lifestyle business" to be a company that tries to be adoptable to the lifestyle of the founder/owner, but maybe I understood it wrong or it changed.