Even a family business often involves several people investing. I don't think a two-founder business that didn't take outside investment is a red flag, but a one-founder business is.

Causality doesn't really matter in terms of putting of customers/investors/employees; if not taking outside investment is correlated with bad things, people will avoid you if you don't take outside investment, whichever direction the causal arrow runs.