I think this is too far narrow a definition of "lifestyle business". The better definition is simply a business that fits the founder's intent and lifestyle.
I know several founders that started from nothing, built their company to turn a sizable profit and resisted big VC money.
It's demeaning and inaccurate to call them "lifestyle businesses", they made a calculated decision. Roughly, they took a chance with self-financing, and maybe a ~40% chance bet on making a good living, versus taking the VC money and having 3% on becoming unbelievably wealthy.
Given a 40% chance of making several million and a 3% chance of making billions, it's pretty reasonable for folks to take the higher probability, even if the statistical expected value is better with the VCs. These aren't people who "gave up", often, they made the smarter decisions.
I find it interesting and even inspiring that some economies allow entrepreneurs to make this kind of choice. That is not my reality at all. In South-America, we grab every chance we get and don't let go. There is much less opportunity.
A company whose owners are working to grow the business consistently and organically is not a lifestyle business; it is also not pursuing "profit at all costs". Most privately-held businesses fall into this category.
As to "lifestyle business" being a slur, it is unfortunate that members of the growth-obsessed communities use it as such. They don't see a business with a fixed target size as serious. Incidentally, many also think the same about the organic growth case I mentioned. I ignore them, and most other things from VC startup culture.