Most people I know (including my parents) who've tried to do the whole "middle-class route to riches!" thing by becoming small-scale landlords have advised me to never, ever, ever try, for exactly the reasons you state: sooner or later you'll get fucked over so hard by a tenant that it wipes out years of income, and you've done it all for nothing.

Exception: one guy, well aware of these issues, had a hack for it—he only owned houses near medical or nursing schools. Nursing and medical students pay their rent and don't trash the place.