Are you really being "ostracized" for it? Lots of people I know semi retire on hobby farms. And you simply can't produce food as efficiently or cheaply as modern farming techniques, so you're in the privileged position (probably as the result of studying and working and earning) that you can afford to spend a large amount of time and money growing your on food.

This is a nice and fulfilling/entertaining lifestyle choice, that's great for you and you that you are in this position now. Maybe that advice you got about getting ahead wasn't so bad then?