I'm in a similar position, but don't avoid banks altogether. No debt, except a credit card balance which is paid off each month, and I only use that for the extra protection over a debit card.

For a short time after college I was in a bad spot with a low paying job and lots (relative to my income) of debt. In a way, I was fortunate to see what that's like early, with a relatively small amount of money (my CC limit was <$2000, but always maxed out because I was broke).

Not a situation I want to be in again.